Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1476

MariaDB row level changes captured but not navigated to kafka

    XMLWordPrintable

Details

    Description

      Am using Debezium to stream data from MariaDB to Kafka.
      During the initial load the data is copied properly but the database changes are detected by Debezium but not sent to kafka.

      Below is error stack trace. Any leads highly appreciated.

      Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:19,859] ERROR Encountered change event 'Event{header=EventHeaderV4{timestamp=1568361089000, eventType=TABLE_MAP, serverId=1, headerLength=19, dataLength=89, nextPosition=4919121, flags=0}, data=TableMapEventData{tableId=2697, database='reports', table='TABLE_NAME', columnTypes=3, 3, -2, 3, 3, 3, 18, 5, 5, 5, 18, 18, 3, 5, 5, 5, 5, 5, 5, 5, 5, -2, 3, 5, 3, 3, 3, 3, 3, 18, columnMetadata=0, 0, 65072, 0, 0, 0, 0, 8, 8, 8, 0, 0, 0, 8, 8, 8, 8, 8, 8, 8, 8, 65120, 0, 8, 0, 0, 0, 0, 0, 0, columnNullability={}, eventMetadata=null}}' at offset {ts_sec=1568361089, file=mariadb-bin.000209, pos=4918332, server_id=1, event=2} for table reports.TABLE_NAME whose schema isn't known to this connector. One possible cause is an incomplete database history topic. Take a new snapshot in this case.
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: Use the mysqlbinlog tool to view the problematic event: mysqlbinlog --start-position=4919013 --stop-position=4919121 --verbose mariadb-bin.000209 (io.debezium.connector.mysql.BinlogReader:747)
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:19,859] ERROR Error during binlog processing. Last offset stored = null, binlog reader near position = mariadb-bin.000209/4919013 (io.debezium.connector.mysql.BinlogReader:1054)
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:19,859] ERROR Failed due to error: Error processing binlog event (io.debezium.connector.mysql.BinlogReader:209)
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: org.apache.kafka.connect.errors.ConnectException: Encountered change event for table reports.TABLE_NAMEwhose schema isn't known to this connector
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:208)
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:508)
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1130)
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:978)
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:581)
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:860)
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at java.lang.Thread.run(Thread.java:748)
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: Caused by: org.apache.kafka.connect.errors.ConnectException: Encountered change event for table reports.TABLE_NAMEwhose schema isn't known to this connector
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.informAboutUnknownTableIfRequired(BinlogReader.java:758)
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.handleUpdateTableMetadata(BinlogReader.java:733)
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:492)
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: #011... 5 more
Sep 13 08:08:19 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:19,859] INFO Error processing binlog event, and propagating to Kafka Connect so it stops this connector. Future binlog events read before connector is shutdown will be ignored. (io.debezium.connector.mysql.BinlogReader:513)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:20,134] INFO WorkerSourceTask{id=mysql_live_2-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:398)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:20,134] INFO WorkerSourceTask{id=mysql_live_2-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:415)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:20,135] INFO WorkerSourceTask{id=mysql_live_2-0} Finished commitOffsets successfully in 1 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:497)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:20,135] ERROR WorkerSourceTask{id=mysql_live_2-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:179)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: org.apache.kafka.connect.errors.ConnectException: Encountered change event for table reports.TABLE_NAMEwhose schema isn't known to this connector
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:208)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:508)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1130)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:978)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:581)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:860)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at java.lang.Thread.run(Thread.java:748)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: Caused by: org.apache.kafka.connect.errors.ConnectException: Encountered change event for table reports.TABLE_NAMEwhose schema isn't known to this connector
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.informAboutUnknownTableIfRequired(BinlogReader.java:758)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.handleUpdateTableMetadata(BinlogReader.java:733)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:492)
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: #011... 5 more
Sep 13 08:08:20 ip-172-31-4-198 connect-distributed[4062]: [2019-09-13 08:08:20,136] ERROR WorkerSourceTask{id=mysql_live_2-0} Task is being killed

      Attachments

        Issue Links

          is related to

          Feature Request - Feature requests from customers and/or users DBZ-1145 Clarification on MySQL vs MariaDb Usage

          • Minor - To be worked after urgent, high, and medium priorities are resolved. This term is chosen when the severity of the issue is low, or the complexity or effort to correct it may be higher, relatively speaking. For low priority issues, known workarounds exist or are not needed due to the trivial effort needed to address the issue.
          • Closed
          relates to

          Feature Request - Feature requests from customers and/or users DBZ-2913 Provide first class support for MariaDB

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              rupsn rups n (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: