Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-168

Problematic XA events in binlog after MySQL upgrade to 5.7

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Done
    • Affects Version/s: 0.3.4
    • Fix Version/s: 0.3.6, 0.4
    • Component/s: mysql-connector
    • Labels:
      None
    • Environment:

      KafkaConnect-0.10.0.0
      Dbz-0.3.4
      MySQL-5.7.16-10

      Description

      When we upgraded to MySQL 5.7.16-10 and started testing, following exception was thrown in kafka-connect logs and task process simply existed :

      [2016-12-19 14:51:49,049] ERROR Error parsing DDL statement and updating tables: XA START X'00000000000000000000ffffc0a811056a085e515857c20c0000b9f331',X'00000000000000000000ffffc0a811056a085e515857c20c0000b9f70000000400000000',131077 (io.debezium.connector.mysql.MySqlSchema:328)
      io.debezium.text.ParsingException: Expecting token type 128 at line 1, column 1 but found 'XA':  ===>> XA START X'000000000
      at io.debezium.text.TokenStream.consume(TokenStream.java:725)
      at io.debezium.relational.ddl.DdlParser.consumeStatement(DdlParser.java:567)
      at io.debezium.relational.ddl.DdlParser.parseUnknownStatement(DdlParser.java:375)
      at io.debezium.connector.mysql.MySqlDdlParser.parseNextStatement(MySqlDdlParser.java:158)
      at io.debezium.relational.ddl.DdlParser.parse(DdlParser.java:285)
      at io.debezium.relational.ddl.DdlParser.parse(DdlParser.java:266)
      at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:326)
      at io.debezium.connector.mysql.BinlogReader.handleQueryEvent(BinlogReader.java:390)
      at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:295)
      at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:877)
      at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:735)
      at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:451)
      at com.github.shyiko.mysql.binlog.BinaryLogClient$5.run(BinaryLogClient.java:632)
      at java.lang.Thread.run(Thread.java:745)
      [2016-12-19 14:51:49,051] ERROR Error parsing DDL statement and updating tables: XA END X'00000000000000000000ffffc0a811056a085e515857c20c0000b9f331',X'00000000000000000000ffffc0a811056a085e515857c20c0000b9f70000000400000000',131077 (io.debezium.connector.mysql.MySqlSchema:328)
      io.debezium.text.ParsingException: Expecting token type 128 at line 1, column 1 but found 'XA':  ===>> XA END X'00000000000
      at io.debezium.text.TokenStream.consume(TokenStream.java:725)
      at io.debezium.relational.ddl.DdlParser.consumeStatement(DdlParser.java:567)
      at io.debezium.relational.ddl.DdlParser.parseUnknownStatement(DdlParser.java:375)
      at io.debezium.connector.mysql.MySqlDdlParser.parseNextStatement(MySqlDdlParser.java:158)
      at io.debezium.relational.ddl.DdlParser.parse(DdlParser.java:285)
      at io.debezium.relational.ddl.DdlParser.parse(DdlParser.java:266)
      at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:326)
      at io.debezium.connector.mysql.BinlogReader.handleQueryEvent(BinlogReader.java:390)
      at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:295)
      at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:877)
      at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:735)
      at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:451)
      at com.github.shyiko.mysql.binlog.BinaryLogClient$5.run(BinaryLogClient.java:632)
      at java.lang.Thread.run(Thread.java:745)
      [2016-12-19 14:51:49,690] INFO Finished WorkerSourceTask{id=patient-data-connector-registry_trunk-20-0} commitOffsets successfully in 3 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:358)
      [2016-12-19 14:51:49,690] ERROR Task patient-data-connector-registry_trunk-20-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:142)
      org.apache.kafka.connect.errors.ConnectException: Unknown event type 123
      at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:157)
      at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:125)
      at io.debezium.connector.mysql.BinlogReader$ReaderThreadLifecycleListener.onEventDeserializationFailure(BinlogReader.java:548)
      at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:727)
      at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:451)
      at com.github.shyiko.mysql.binlog.BinaryLogClient$5.run(BinaryLogClient.java:632)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Unknown event type 123
      at com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.getEventType(EventHeaderV4Deserializer.java:45)
      at com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java:35)
      at com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java:27)
      at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:145)
      at io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:108)
      at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:716)
      ... 3 more
      [2016-12-19 14:51:49,691] ERROR Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:143)
      

      my.cnf properties :

      [mysqld]
      datadir=/var/lib/mysql
      socket=/var/lib/mysql/mysql.sock
      explicit_defaults_for_timestamp
      symbolic-links=0
      log-error=/var/log/mysqld.log
      pid-file=/var/run/mysqld/mysqld.pid
      default-storage-engine         = InnoDB
      max-allowed-packet             = 256M
      max-connect-errors             = 1000000
      character_set_server            = utf8
      collation_server                = utf8_general_ci
      max-heap-table-size             = 32M
      max-connections                 = 800
      thread-cache-size               = 50
      open-files-limit                = 65535
      innodb-flush-method             = O_DIRECT
      innodb-log-file-size           = 256M
      innodb-file-per-table          = 1
      innodb_buffer_pool_size         = 8G
      innodb_thread_concurrency       = 8
      innodb_page_size                = 32K
      innodb_lock_wait_timeout=300
      skip-name-resolve
      server_id=10
      log_bin=mysql_bin.log
      expire_logs_days=2
      max_binlog_size=100M
      binlog_format=ROW
      secure_file_priv = /tmp
      sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_ALL_TABLES
      innodb_file_format=Barracuda
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  rhauch Randall Hauch
                  Reporter:
                  anton_nazaruk Anton Nazaruk
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: