Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1217

Postgres connector with wal2json streaming fails to process messages that are missing meta data

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • None
    • 0.9.4.Final
    • postgresql-connector
    • None

    Description

      I have introduced a bug in StreamingWal2JsonMessageDecoder with my change to make Debezium process WAL2JSON messages with empty changes. Line 180 used to have a check to only run when "if (currentChunk != null)." This needed to be removed to make my changes work, but there seems to be an edgecase where changes can come in without a txid or any other meta data. In this case placeholder values are set in outOfOrderChunk like this:

      txId = UNDEFINED_LONG;
      timestamp = UNDEFINED_STRING;
      commitTime = null;
      messageInProgress = true;
      currentChunk = null;

      The problem is that the Wal2JsonReplicationMessage can't handle these values, and it would not make sense to try and process a fake txid etc. I can make a PR to fix it, but there are a few ways fix can be made:

      1. Only process if txId != UNDEFINED_LONG.
      2. Try to determine the txid somehow ( I think there is a way to query the PG connection to determine that) and fill in the other commitTime with current time.
      3. Some other better idea that I can't think of.

      Going the first route would be very easy and would be similar to how it used to work, but I'm just not sure if there would be some issue by not processing the message.

      Attachments

        Activity

          People

            Unassigned Unassigned
            trolison Taylor Rolison (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: