Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7698

Provide option to encode ordering metadata in each record

    XMLWordPrintable

Details

    • False
    • None
    • False
    • 0
    • 0% 0%

    Description

      Feature request or enhancement

      This came up in Zulip discussion here. The motivation is that when kafka re-partition or other event happens where keys lose their ordering properties (i.e., ordering only guaranteed per partition), then getting consistent & correct data is both manual and error prone.

      The possible solution is to have each message contain a collection of fields on the change event's ordering metadata. With these fields any system can establish for an arbitrary pair of changes for the same primary key which is more recent.

      Which use case/requirement will be addressed by the proposed feature?

      Allow for downstream systems to automatically handle kafka re-partition events.

      Implementation ideas (optional)

      In order to help make the discussion concrete, we created a PR in the debezium-vitess-connector repo that demonstrates how this will look for this connector. However, it can be generalized to all connectors (as long as they provide an implementation of the respective interfaces).

      Attachments

        Activity

          People

            Unassigned Unassigned
            tthorn Thomas Thornton
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: