Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7483

Allow configuring Debezium to resume replication at slot position instead of at stored LSN

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Unresolved
    • Icon: Major Major
    • 2.6-backlog
    • None
    • postgresql-connector
    • None
    • False
    • None
    • False
    • 0
    • 0% 0%

      Which use case/requirement will be addressed by the proposed feature?

      AWS Aurora upgrades reset the LSN sequence, resulting in the unhappy state that the only way to fix replication is to reset offsets manually using a tool such as kafkacat.  This remains an issue even if we drop the replication slot or advance it.  It essentially ignores the slot position.

      There are other scenarios in which we could end up with metadata specifying an LSN resume point that is impossible given available logical replication slots.  The proposal is to allow us to "reset" that metadata based on slot position.

      Implementation ideas

      One possibility is to add a configuration option such as resume.at.slot.position = true.  The idea is that we can force debezium to take the slot position as the resume point, instead of the stored LSN position.

      jpechane suggested attempting to do this first using Field.createInternal.  See discussion.

      In native logical replication postgres -> postgres, that is basically how it works if you drop a slot, or advance a slot. The subscriber is going to connect and resume based on the position of the slot.

      But upon facing a troublesome situation like post-upgrade LSNs are reset, for us to update the connector config with an option that allows it to resume the LSN at the slot position would much more easily allow us to get out of this situation than resetting kafka offsets.  Once things are working again, that is, the slot is actively replicating and has advanced, the connector could be restarted with the option removed.

            Unassigned Unassigned
            jfinzel Jeremy Finzel
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: