Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-306

add support for different mongodb _id types in key struct

    Details

    • Type: Enhancement
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 0.5.1
    • Fix Version/s: 0.6
    • Component/s: mongodb-connector
    • Labels:
      None
    • Environment:

      The suggested improvement is not environment specific.

      Description

      This addresses a potential design flaw with regard to the handling of the _id field of MongoDB CDC events in the key struct of SourceRecords. Currently, the problem is that _id fields are always
      represented as flat strings and do not expose type specific information to the consumers (e.g. String vs. ObjectId, String vs Integer, Full Document,...)

      Rationale:

      At the moment, one cannot even distinguish simple cases e.g. a numeric _id field holding the integer 1234 vs. a string _id field holding "1234". While for CREATE/READ events consumers can re-create the correct (original) _id type based on the "after" field found in the value struct, it is not possible to use this workaround for idempotent change or delete events respectively. For these we need to rely on a correct _id type in order to be able to refer to the corresponding data records and apply the changes or deletetions at the consuming sink.

      There are different solutions to tackle this enhancement. Some of which would not work with AVRO but only with JSON serialization of Kafka Connect records. There's already a pull request discussion dealing with this, see https://github.com/debezium/debezium/pull/258

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                hpgrahsl H.P. Grahsl
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: