Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3698

PoC for adding transformations / SMT steps to the Debezium UI

    XMLWordPrintable

Details

    Description

      Create a PoC that adds another Debezium UI wizard page/step (or multiple steps/pages) to configure Kafka Connect transformations, aka SMTs - Single Message Transforms.
      SMTs provide capabilities to apply transformations like adding, dropping or aggregating field data, renaming fields, masking data, filter out messages that should not go to Kafka, etc.

      Here's a writeup by Gunnar about SMTs on his blog: https://www.morling.dev/blog/single-message-transforms-swiss-army-knife-of-kafka-connect/

      Confluent gives a good overview of the most common SMTs that come shipped with every Kafka distribution:
      https://docs.confluent.io/platform/current/connect/transforms/overview.html

      Some advanced SMTs by Debezium in the docs:
      https://debezium.io/documentation/reference/configuration/event-flattening.html
      https://debezium.io/documentation/reference/configuration/filtering.html
      https://debezium.io/documentation/reference/configuration/topic-routing.html
      https://debezium.io/documentation/reference/configuration/content-based-routing.html

      Please decide on 2-3 transformations (choose basic ones like filtering, replace field, etc) that should be implemented as a PoC, sketch a UI for it and let's discuss that with UX and finally implement a PoC to add transformations to the Debezium UI. It should be possible to use a single transformation or multiple and it should be possible to order them (which one gets called first, which next, which last).
      The UI does not necessarily have to return the exact Kafka Connect config format. An abstract format that can be converted to the Kafka Connect config is sufficient (as you see fit).

      Attachments

        Issue Links

          Activity

            People

              ishukla Indra Shukla
              rk3rn3r René Kerner
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: