Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-29

Handle large messages

    XMLWordPrintable

Details

    Description

      Most Kafka clusters run with a max message size of 1MB - 2MB. I'm not sure if/how Debezium/Kafka Connect handle this now, but there should be a relief valve for messages that are larger than the limit. Common pitfalls to this are tables that have large TEXT or BLOB columns.

      Some ideas:

      1. Add the ability to project out certain columns on a per-table basis. Users could then filter out the offending columns.
      2. Add the ability to drop the messages if they're too large.
      3. Add the ability to truncate text/blob columns.

      I'm partial to have options both (1) and (3). (2) is not something that I would use.

      Attachments

        Activity

          People

            rhauch Randall Hauch (Inactive)
            criccomini Chris Riccomini (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: