Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-681

PostgreSQL connector doesn't handle TIME(p) columns correctly with wal2json

    XMLWordPrintable

    Details

    • Steps to Reproduce:
      Hide

      1. Have a PostgeSQL instance with wal2Json plugin (I used docker debezium/postgres)
      2. Have a PostgreSQL connector stood up, my config:

      { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "plugin.name": "wal2json", "tasks.max": "1", "database.hostname": "postgres", "database.port": "5432", "database.user": "postgres", "database.password": "***", "database.dbname": "postgres", "database.server.name": "elasticsearch", "table.whitelist": "public.dbo.testmssqltable", "decimal.handling.mode": "double" }

      3. create matching table:
      CREATE TABLE "dbo.testmssqltable" (
      "id" int not NULL,
      "int" int,
      "numeric" numeric,
      "time" time,
      PRIMARY KEY("id"));
      4. Insert example data, with time with precision:
      insert into "dbo.testmssqltable" ("id","int","numeric", "time") VALUES (1, 1, 111.234567, '12:12:12.1');
      5. Error is thrown and connectors stops to work, and I am actually unable to continue to work with this connector as long as there is such time example in DB, as it will fail each time when restarted.

      Show
      1. Have a PostgeSQL instance with wal2Json plugin (I used docker debezium/postgres) 2. Have a PostgreSQL connector stood up, my config: { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "plugin.name": "wal2json", "tasks.max": "1", "database.hostname": "postgres", "database.port": "5432", "database.user": "postgres", "database.password": "***", "database.dbname": "postgres", "database.server.name": "elasticsearch", "table.whitelist": "public.dbo.testmssqltable", "decimal.handling.mode": "double" } 3. create matching table: CREATE TABLE "dbo.testmssqltable" ( "id" int not NULL, "int" int, "numeric" numeric, "time" time, PRIMARY KEY("id")); 4. Insert example data, with time with precision: insert into "dbo.testmssqltable" ("id","int","numeric", "time") VALUES (1, 1, 111.234567, '12:12:12.1'); 5. Error is thrown and connectors stops to work, and I am actually unable to continue to work with this connector as long as there is such time example in DB, as it will fail each time when restarted.

      Description

      When using PostgresSQL source connector on a table with a column of type Time, connector will fail (and stop working) when there is example data present with precision.

      Stack trace:

      org.apache.kafka.connect.errors.ConnectException: An exception ocurred in the change event producer. This connector will be stopped.
      	at io.debezium.connector.base.ChangeEventQueue.throwProducerFailureIfPresent(ChangeEventQueue.java:168)
      	at io.debezium.connector.base.ChangeEventQueue.poll(ChangeEventQueue.java:149)
      	at io.debezium.connector.postgresql.PostgresConnectorTask.poll(PostgresConnectorTask.java:151)
      	at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:179)
      	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
      	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.kafka.connect.errors.ConnectException: java.time.format.DateTimeParseException: Text '12:12:12.1' could not be parsed, unparsed text found at index 8
      	at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.format(DateTimeFormat.java:117)
      	at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.time(DateTimeFormat.java:92)
      	at io.debezium.connector.postgresql.connection.wal2json.Wal2JsonReplicationMessage.getValue(Wal2JsonReplicationMessage.java:250)
      	at io.debezium.connector.postgresql.connection.wal2json.Wal2JsonReplicationMessage$1.getValue(Wal2JsonReplicationMessage.java:135)
      	at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$columnValues$2(RecordsStreamProducer.java:430)
      	at java.util.ArrayList.forEach(ArrayList.java:1249)
      	at io.debezium.connector.postgresql.RecordsStreamProducer.columnValues(RecordsStreamProducer.java:425)
      	at io.debezium.connector.postgresql.RecordsStreamProducer.process(RecordsStreamProducer.java:239)
      	at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$streamChanges$1(RecordsStreamProducer.java:119)
      	at io.debezium.connector.postgresql.connection.wal2json.Wal2JsonMessageDecoder.processMessage(Wal2JsonMessageDecoder.java:61)
      	at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.deserializeMessages(PostgresReplicationConnection.java:209)
      	at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.read(PostgresReplicationConnection.java:194)
      	at io.debezium.connector.postgresql.RecordsStreamProducer.streamChanges(RecordsStreamProducer.java:119)
      	at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$start$0(RecordsStreamProducer.java:105)
      	... 5 more
      Caused by: java.time.format.DateTimeParseException: Text '12:12:12.1' could not be parsed, unparsed text found at index 8
      	at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1952)
      	at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851)
      	at java.time.LocalTime.parse(LocalTime.java:441)
      	at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.lambda$time$2(DateTimeFormat.java:92)
      	at io.debez
      

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                jpechanec Jiri Pechanec
                Reporter:
                przemekak Przemek Pawlowski
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: