Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1297

Expose outbox event structure at level of Kafka Connect messages

    Details

    • Type: Task
    • Status: Pull Request Sent (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 1.0-backlog
    • Component/s: core-library
    • Labels:
      None

      Description

      Currently, the payload structure of outbox events is fully opaque at the Kafka message level, as it's a string-ified JSON field (or, base64-encoded Avro as one user reported to do).

      It'd be useful for consumers, if there was more explicit schema support for that payload structure. I.e. the payload schema should be part of the event schema itself (by "expanding" the single payload field into a properly typed Kafka Connect data structure). Depending on the chosen serializer, the outbox event payload schema then would be part of the JSON schema sent with each message or it would be registered with the schema registry when using Avro.

      An open question is how to convey the schema information from the application writing to the outbox table to the Debezium outbox router (which would do this "expansion" before sending messages to Kafka). Similar to the existing serialization options in Kafka Connect, one could think of passing the JSON schema in a separate column (when using JSON as payload) or a schema id resolvable against the registry (when using Avro).

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  gunnar.morling Gunnar Morling
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated: