Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-588

Debezium skipping binlog events silently

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • 0.7.3
    • 0.6.2
    • mysql-connector
    • None

    Description

      When Debezium encounters an insert/update/delete binlog event that it cannot process, it appears to skip the event entirely with only a Debug level log message. This causes not only data loss, but lack of visibility that it encountered an issue.

      Data Loss: Since the event is skipped, the reader continues to read the binlog beyond this point and commit future offsets. Therefore, it may be impossible to recover the skipped events which may have been valid indeed.

      Visibility The skipped events are only logged at Debug level, so they are not visible in production logs. When we encountered this on production environment, it took 4 weeks to notice something was wrong.

      For use cases that require no data loss, we need the reader to stop completely and throw a hard exception if it encounters this scenario. This gives clear notice and a chance to figure out what went wrong and attempt to recover.

      Example: For UPDATE event, this is the only log message we currently get. Note this is DEBUG level only, nothing on INFO level. 
       
      [2018-01-18 16:52:03,920] DEBUG Skipping update table metadata event: Event{header=EventHeaderV4{timestamp=1516312323000, eventType=TABLE_MAP, serverId=223344, headerLength=19, dataLength=100, nextPosition=6456, flags=0}, data=TableMapEventData{tableId=444, database='mydb', table='mytable', columnTypes=-2, 15, 15, 15, 15, 15, -2, -2, 10, 1, 1, -2, -2, -2, 15, 15, -2, 17, -2, 17, -2, 17, columnMetadata=65056, 50, 50, 50, 255, 100, 63233, 65056, 0, 0, 0, 63233, 63233, 65026, 32, 32, 65056, 0, 65056, 0, 65056, 0, columnNullability={5, 6, 8, 11, 12}}} (io.debezium.connector.mysql.BinlogReader:633)
      ]}} (io.debezium.connector.mysql.BinlogReader:730)
      

      Attachments

        Activity

          People

            jpechane Jiri Pechanec
            pgoransson@cainc.com Peter Goransson (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: