Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-521

Possibility of commit LSN before record is consumed/notified

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 0.7.2
    • 0.7.1
    • postgresql-connector
    • None

    Description

      Version: 0.7.1
      Modules: postgres connector, EmbeddedEngine
      Postgres Version: AWS RDS 9.6.5

      There may be chance that LSN committed back to Postgres can be later than what written to offset store. task.commit() on postgres connector (commitRecord() is not implemented) seems to use latest lsn received (e.g. record it puts into queue). Offset (LSN) that is written to the store is the latest record consumed by the consumer which may not be the latest one from the queue.

      I believe postgres may truncate / archive WAL block once LSN is committed?

      To reproduce:
      1) Make consumer (notify callback) slow so that changes are queue up
      2) Start EmbeddedEngine
      3) Make DB changes continuously until commit happens
      4) Check offset store for LSN and postgres committed LSN

      Attachments

        Activity

          People

            jpechane Jiri Pechanec
            thon_jira Thon Mekathikom (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: