Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-971

Convey original operation type when using flattening SMTs

    XMLWordPrintable

    Details

      Description

      The problem

      Currently there's no way to inform whether the message operation is, this information is lost along the way through the `UnwrapFromMongoDbEnvelope`, we can only distinguish between deletions or not.

      The scenario

      I want to be able to tell whether the message is an update or insert thus easing logic within the Sink Connector.
      The rationale is about optimizing the Sink in a way it doesn't have to look for an item before knowing whether it should insert or update, this can potentially save a great number of queries in the Sink side.
      Basically avoid `saveOrUpdate` situations by knowing exactly which operation to run.

      Past proposed solution (rejected)

      Currently the value schema has a metadata name which informs the server, db and collection, I want to add on it the operation as in `io.debezium.data.Envelope.Operation`
      The current value looks like: `serverX.dbA.c1`
      Proposed value would be:

      • `serverX.dbA.c1.r` for initial read
      • `serverX.dbA.c1.c` for creation
      • `serverX.dbA.c1.u` for updates

      This is very similar to the initialSync key in the non cdc way.

      Proposed solution

      Add a new header into the kafka record, finally being available in the message itself, this header will be optional via a transformer flag (MONGO_OPERATION_HEADER) and will respect the `io.debezium.data.Envelope.Operation` enum.
      The proposed header name is `application/debezium-operation`

      Implementation details

      This can be an optional feature within the transformer thus not breaking BC

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                renatomefi Renato Mefi
                Reporter:
                renatomefi Renato Mefi
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: