Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1033

Mysql binlog reader lost data if restart task when last binlog event is QUERY event.

    XMLWordPrintable

    Details

    • Steps to Reproduce:
      Hide

      Reproduce procedure:

      1. create table `test` (id int, val varchar(20));
      2. start dbz connector to sync for table test.
      3. alter table `test` comment 'test1'/'test2' for N times.
      4. restart the task.
      5. insert into `test` VALUES(1, 'test')

      The binlog event for insert will be skipped.

      Show
      Reproduce procedure: 1. create table `test` (id int, val varchar(20)); 2. start dbz connector to sync for table test. 3. alter table `test` comment 'test1'/'test2' for N times. 4. restart the task. 5. insert into `test` VALUES(1, 'test') The binlog event for insert will be skipped.

      Description

      Binlog reader increased the count for restartEventsToSkip each time after it processed binlog event. That count would be reset when there is a 'BEGIN' sql or a 'COMMIT' sql. However, if the task was restarted after processing a DDL event, it would increase the restartEventsToSkip count as well as record the latest binlog position in offset, and the offset will be updated to kafka offset topic by the DDL record. And then, if the task was restarted, it will skip the first N events after restart which should not be skipped.

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ooooorz Hanlin Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: