Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2397

Transaction data loss when process restarted

XMLWordPrintable

      After upgrading from postgresql 9.6, decoderbufs/debezium 0.7.5 to postgresql 12.3, decoderbufs/debezium 1.1.2.Final we began to notice data loss when the debezium process was restarted while a transaction was open. For example:

      1. Start debezium process
      2. Begin transaction T
      3. Insert record A inside transaction T
      4. Insert record X outside transaction T
      5. Stop debezium process
      6. Insert record B inside transaction T
      7. Insert record Y outside transaction T
      8. Start debezium process
      9. Insert record C inside transaction T
      10. Insert record Z outside transaction T
      11. Commit transaction T
      12. Observe that record A has been dropped from the CDC stream.

      NOTE: `shouldMessageBeSkipped` method in `io.debezium.connector.postgresql.connection.AbstractMessageDecoder` will log and return true when it sees the message for record A.

       

      NOTE: We’ve also been able to confirm that the bug was introduced sometime on or before decoderbufs/debezium 0.9.5.Final

        1. diff
          2 kB
          Stephen Webber

            jpechane Jiri Pechanec
            atongen@gmail.com Andrew Tongen (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: