Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1217

Postgres connector with wal2json streaming fails to process messages that are missing meta data

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.9.4.Final
    • Fix Version/s: None
    • Component/s: postgresql-connector
    • Labels:
      None

      Description

      I have introduced a bug in StreamingWal2JsonMessageDecoder with my change to make Debezium process WAL2JSON messages with empty changes. Line 180 used to have a check to only run when "if (currentChunk != null)." This needed to be removed to make my changes work, but there seems to be an edgecase where changes can come in without a txid or any other meta data. In this case placeholder values are set in outOfOrderChunk like this:

      txId = UNDEFINED_LONG;
      timestamp = UNDEFINED_STRING;
      commitTime = null;
      messageInProgress = true;
      currentChunk = null;

      The problem is that the Wal2JsonReplicationMessage can't handle these values, and it would not make sense to try and process a fake txid etc. I can make a PR to fix it, but there are a few ways fix can be made:

      1. Only process if txId != UNDEFINED_LONG.
      2. Try to determine the txid somehow ( I think there is a way to query the PG connection to determine that) and fill in the other commitTime with current time.
      3. Some other better idea that I can't think of.

      Going the first route would be very easy and would be similar to how it used to work, but I'm just not sure if there would be some issue by not processing the message.

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                trolison Taylor Rolison
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: