Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1128

Advancing LSN in the first iteration - possible data loss

    XMLWordPrintable

    Details

      Description

      These are (commitLsn, changeLsn) pairs from Kafka:

      0x000443D80073D0DA0005 0x000443D80073D037005E
      0x000443D80073D0DA0005 0x000443D80073D0370060
      0x000443D80073D0DA0005 0x000443D80073D0370062
      0x000443D80073D0DA0005 0x000443D80073D0370064
      0x000443D80073D0DA0005 0x000443D80073D0370066
      0x000443D80073D0DA0005 0x000443D80073D0370068
      0x000443D80073D0DA0005 0x000443D80073D037006A
      0x000443D80073D0DA0005 0x000443D80073D037006C
      0x000443D80073D0DA0005 0x000443D80073D037006E
      0x000443D80073D0DA0005 0x000443D80073D037006F
      0x000443D80073D0DA0005 0x000443D80073D0370075
      0x000443D80073D0DA0005 0x000443D80073D0370077
      0x000443D80073D0DA0005 0x000443D80073D0370079
      0x000443D80073D0DA0005 0x000443D80073D037007A
      0x000443D80073D0DA0005 0x000443D80073D0370080
      0x000443D80073D0DA0005 0x000443D80073D0370082
      0x000443D80073D0DA0005 0x000443D80073D0370084
      0x000443D80073D0DA0005 0x000443D80073D0370086
      0x000443D80073D0DA0005 0x000443D80073D0370088
      0x000443D80073D0DA0005 0x000443D80073D037008A
      0x000443D80073D0DA0005 0x000443D80073D037008C
      0x000443D80073D0DA0005 0x000443D80073D037008E
      0x000443D80073D0DA0005 0x000443D80073D0370090
      0x000443D80073D0DA0005 0x000443D80073D0370092
      0x000443D80073D0DA0005 0x000443D80073D0370094
      0x000443D80073D0DA0005 0x000443D80073D0370096
      0x000443D80073D0DA0005 0x000443D80073D0370098
      0x000443D80073D0DA0005 0x000443D80073D037009A
      0x000443D80073D0DA0005 0x000443D80073D037009C
      0x000443D80073D0DA0005 0x000443D80073D037009E
      0x000443D80073D0DA0005 0x000443D80073D03700A0
      0x000443D80073D0DA0005 0x000443D80073D03700A2
      0x000443D80073D0DA0005 0x000443D80073D03700A4
      0x000443D80073D0DA0005 0x000443D80073D03700A6
      0x000443D80073D0DA0005 0x000443D80073D03700A8
      0x000443D80073D1F30001 0x000443D80073D0EC0016
      0x000443D80073D1F30001 0x000443D80073D0EC001C
      0x000443D80073D1F30001 0x000443D80073D0EC001E
      

      These are (commitLsn, changeLsn) pairs from capture instance:

      0x000443D80073D0DA0005 0x000443D80073D037005E
      0x000443D80073D0DA0005 0x000443D80073D0370060
      0x000443D80073D0DA0005 0x000443D80073D0370062
      0x000443D80073D0DA0005 0x000443D80073D0370064
      0x000443D80073D0DA0005 0x000443D80073D0370066
      0x000443D80073D0DA0005 0x000443D80073D0370068
      0x000443D80073D0DA0005 0x000443D80073D037006A
      0x000443D80073D0DA0005 0x000443D80073D037006C
      0x000443D80073D0DA0005 0x000443D80073D037006E
      0x000443D80073D0DA0005 0x000443D80073D037006F
      0x000443D80073D0DA0005 0x000443D80073D0370075
      0x000443D80073D0DA0005 0x000443D80073D0370077
      0x000443D80073D0DA0005 0x000443D80073D0370079
      0x000443D80073D0DA0005 0x000443D80073D037007A
      0x000443D80073D0DA0005 0x000443D80073D0370080
      0x000443D80073D0DA0005 0x000443D80073D0370082
      0x000443D80073D0DA0005 0x000443D80073D0370084
      0x000443D80073D0DA0005 0x000443D80073D0370086
      0x000443D80073D0DA0005 0x000443D80073D0370088
      0x000443D80073D0DA0005 0x000443D80073D037008A
      0x000443D80073D0DA0005 0x000443D80073D037008C
      0x000443D80073D0DA0005 0x000443D80073D037008E
      0x000443D80073D0DA0005 0x000443D80073D0370090
      0x000443D80073D0DA0005 0x000443D80073D0370092
      0x000443D80073D0DA0005 0x000443D80073D0370094
      0x000443D80073D0DA0005 0x000443D80073D0370096
      0x000443D80073D0DA0005 0x000443D80073D0370098
      0x000443D80073D0DA0005 0x000443D80073D037009A
      0x000443D80073D0DA0005 0x000443D80073D037009C
      0x000443D80073D0DA0005 0x000443D80073D037009E
      0x000443D80073D0DA0005 0x000443D80073D03700A0
      0x000443D80073D0DA0005 0x000443D80073D03700A2
      0x000443D80073D0DA0005 0x000443D80073D03700A4
      0x000443D80073D0DA0005 0x000443D80073D03700A6
      0x000443D80073D0DA0005 0x000443D80073D03700A8 <--- The last record in Kafka with commitLsn = 0x000443D80073D0DA0005
      0x000443D80073D0DA0005 0x000443D80073D03700AA
      0x000443D80073D0DA0005 0x000443D80073D03700AC
      0x000443D80073D0DA0005 0x000443D80073D03700AE
      0x000443D80073D0DA0005 0x000443D80073D03700B0
      0x000443D80073D0DA0005 0x000443D80073D03700B2
      0x000443D80073D0DA0005 0x000443D80073D03700B4
      0x000443D80073D0DA0005 0x000443D80073D03700B5
      0x000443D80073D0DA0005 0x000443D80073D03700BB
      0x000443D80073D0DA0005 0x000443D80073D03700BD
      0x000443D80073D0DA0005 0x000443D80073D03700BF
      0x000443D80073D0DA0005 0x000443D80073D03700C1
      0x000443D80073D0DA0005 0x000443D80073D03700C3
      0x000443D80073D0DA0005 0x000443D80073D03700C5
      0x000443D80073D0DA0005 0x000443D80073D03700C7
      0x000443D80073D0DA0005 0x000443D80073D03700C9
      0x000443D80073D0DA0005 0x000443D80073D03700CB
      0x000443D80073D0DA0005 0x000443D80073D03700CD
      0x000443D80073D0DA0005 0x000443D80073D03700CF
      0x000443D80073D0DA0005 0x000443D80073D03700D0
      0x000443D80073D0DA0005 0x000443D80073D03700D6
      0x000443D80073D0DA0005 0x000443D80073D03700D8
      0x000443D80073D0DA0005 0x000443D80073D03700D9
      0x000443D80073D1F30001 0x000443D80073D0EC0016
      0x000443D80073D1F30001 0x000443D80073D0EC001C
      0x000443D80073D1F30001 0x000443D80073D0EC001E
      

      This is the last offset saved before KConnect restart (note it is exactly the same as commitLsn of the last record saved in Kafka):

      {"commit_lsn":"000443d8:0073d0da:0005","change_lsn":"000443d8:0073d037:00a8"}
      
      2019-01-31 11:48:57,445 INFO Found previous offset SqlServerOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.sqlserver.Source:STRUCT}, sourceInfo=SourceInfo [serverName=XXX, changeLsn=000443d8:0073d037:00a8, commitLsn=000443d8:0073d0da:0005, snapshot=false, sourceTime=null], partition={server=XXX}, snapshotCompleted=false] (io.debezium.connector.sqlserver.SqlServerConnectorTask:142)
      2019-01-31 11:49:00,613 INFO Last position recorded in offsets is 000443d8:0073d0da:0005(000443d8:0073d037:00a8) (io.debezium.connector.sqlserver.SqlServerStreamingChangeEventSource:72)
      

      At first glance it seems, that the remaining, unprocessed part with commitLsn=0x000443D80073D0DA0005 was skipped because in the very first iteration commitLsn is increased by one:

                      // Reading interval is inclusive so we need to move LSN forward
                      final Lsn fromLsn = lastProcessedPosition.getCommitLsn().isAvailable() ? connection.incrementLsn(lastProcessedPosition.getCommitLsn())
                              : lastProcessedPosition.getCommitLsn();
      

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                jpechanec Jiri Pechanec
                Reporter:
                grzegorzk Grzegorz KoĊ‚akowski
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: