Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-138

Deleting Debezium connector in KC causes NPEs

    Details

      Description

      While we were running experiments for DBZ-129, we noticed an odd behavior. During one of our steps, we did a curl -X DELETE http://kafka-connect:8083/connectors/debezium-test. This caused Kafka Connect (KC) to start to shut down the connector. During the shutdown, we started getting a ton of these messages:

      Oct 18, 2016 5:18:01 PM com.github.shyiko.mysql.binlog.BinaryLogClient notifyEventListeners
      WARNING: io.debezium.connector.mysql.BinlogReader$$Lambda$102/704926729@e9c75dc choked on Event{header=EventHeaderV4{timestamp=1476811081000, eventType=QUERY, serverId=798028720, headerLength=19, dataLength=45, nextPosition=223258, flags=8}, data=QueryEventData{threadId=196595, executionTime=0, errorCode=0, database='', sql='COMMIT'}}
      java.lang.NullPointerException
      	at io.debezium.relational.history.KafkaDatabaseHistory.storeRecord(KafkaDatabaseHistory.java:157)
      	at io.debezium.relational.history.AbstractDatabaseHistory.record(AbstractDatabaseHistory.java:44)
      	at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:359)
      	at io.debezium.connector.mysql.BinlogReader.handleQueryEvent(BinlogReader.java:377)
      	at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:280)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:877)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:734)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:451)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient$5.run(BinaryLogClient.java:632)
      	at java.lang.Thread.run(Thread.java:745)
      
      Oct 18, 2016 5:18:02 PM com.github.shyiko.mysql.binlog.BinaryLogClient notifyEventListeners
      WARNING: io.debezium.connector.mysql.BinlogReader$$Lambda$102/704926729@e9c75dc choked on Event{header=EventHeaderV4{timestamp=1476811082000, eventType=QUERY, serverId=798028720, headerLength=19, dataLength=45, nextPosition=223450, flags=8}, data=QueryEventData{threadId=196595, executionTime=0, errorCode=0, database='', sql='COMMIT'}}
      java.lang.NullPointerException
      	at io.debezium.relational.history.KafkaDatabaseHistory.storeRecord(KafkaDatabaseHistory.java:157)
      	at io.debezium.relational.history.AbstractDatabaseHistory.record(AbstractDatabaseHistory.java:44)
      	at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:359)
      	at io.debezium.connector.mysql.BinlogReader.handleQueryEvent(BinlogReader.java:377)
      	at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:280)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:877)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:734)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:451)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient$5.run(BinaryLogClient.java:632)
      	at java.lang.Thread.run(Thread.java:745)
      
      Oct 18, 2016 5:18:03 PM com.github.shyiko.mysql.binlog.BinaryLogClient notifyEventListeners
      WARNING: io.debezium.connector.mysql.BinlogReader$$Lambda$102/704926729@e9c75dc choked on Event{header=EventHeaderV4{timestamp=1476811083000, eventType=QUERY, serverId=798028720, headerLength=19, dataLength=45, nextPosition=223642, flags=8}, data=QueryEventData{threadId=196595, executionTime=0, errorCode=0, database='', sql='COMMIT'}}
      java.lang.NullPointerException
      	at io.debezium.relational.history.KafkaDatabaseHistory.storeRecord(KafkaDatabaseHistory.java:157)
      	at io.debezium.relational.history.AbstractDatabaseHistory.record(AbstractDatabaseHistory.java:44)
      	at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:359)
      	at io.debezium.connector.mysql.BinlogReader.handleQueryEvent(BinlogReader.java:377)
      	at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:280)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:877)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:734)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:451)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient$5.run(BinaryLogClient.java:632)
      	at java.lang.Thread.run(Thread.java:745)
      

      The NPEs just kept getting spewed until we kill -9'd the KC worker. We haven't been able to reproduce this, and haven't seen it again.

      NOTE: this was running on the 0.3.3 branch with the DBZ-129 PR applied to it.

      cc Akshath Patkar.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  rhauch Randall Hauch
                  Reporter:
                  criccomini Chris Riccomini
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: