Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-521

Possibility of commit LSN before record is consumed/notified

    Details

      Description

      Version: 0.7.1
      Modules: postgres connector, EmbeddedEngine
      Postgres Version: AWS RDS 9.6.5

      There may be chance that LSN committed back to Postgres can be later than what written to offset store. task.commit() on postgres connector (commitRecord() is not implemented) seems to use latest lsn received (e.g. record it puts into queue). Offset (LSN) that is written to the store is the latest record consumed by the consumer which may not be the latest one from the queue.

      I believe postgres may truncate / archive WAL block once LSN is committed?

      To reproduce:
      1) Make consumer (notify callback) slow so that changes are queue up
      2) Start EmbeddedEngine
      3) Make DB changes continuously until commit happens
      4) Check offset store for LSN and postgres committed LSN

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                jpechanec Jiri Pechanec
                Reporter:
                thon Thon Mekathikom
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: