Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-454

Postgres wal2json drops unknown types during updates

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 0.7.0
    • 0.7.0
    • postgresql-connector
    • None

    Description

      debezium using wal2json returns different records from the initial sync and a subsequent update for the same record when the type is a custom/non-standard one (Postgis geometry in this case).

      https://gist.github.com/rcoup/2ed1601577bb42d47db5f595f8d2ff46#file-update-json-L246 shows geom=null, whereas in the initial sync it has a (hex) value.

      • initial.json is what you get for the row during --from-beginning
      • update.json is what you get when you do UPDATE nz_titles SET id=id WHERE id=976948;
      • log.txt is the associated kafka-connect/debezium log.

      One thing that I noticed were some log messages about geom:

      DEBUG  Postgres|pgwal_test|records-stream-producer  - field 'geom' (STRING) from column geom geometry(2147483647,0)   [io.debezium.relational.TableSchemaBuilder]
      
      DEBUG  Postgres|pgwal_test|records-stream-producer  Mapped columns for table 'public.nz_titles' to schema: {"name" : "pgwal_test.public.nz_titles.Value", "type" : "STRUCT", "optional" : "true", "fields" : [..., {"name" : "geom", "index" : "11", "schema" : {"name" : "io.debezium.data.Json", "type" : "STRING", "optional" : "true", "version" : "1"}}]}   [io.debezium.relational.TableSchemaBuilder]
      

      The type is a geometry(MultiPolygon,2193) to be specific, so I'd imagine it'll just fall through into a string as it's unknown (as the first line implies). But the second line says io.debezium.data.Json. A chunk-o-hex won't validate as JSON, so maybe that's how it ends up null?

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              robert.coup Robert Coup (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: