Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3469

"Found null value for non-optional schema" error when issuing TRUNCATE from Postgres on a table with a PK

    XMLWordPrintable

Details

    • False
    • False
    • Undefined
    • Hide

      0. Start debezium container:

           KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
     VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter

      1 Configure Debezium:

      { "name": "psql-connector",
 "config":
{ "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "postgres", "database.port": "5432", "database.user": "debezium", "database.password": "debezium", "database.dbname" : "postgres", "database.server.name": "postgres", "plugin.name": "pgoutput", "slot.name" : "tester", "database.history.kafka.bootstrap.servers": "kafka:9092", "database.history.kafka.topic": "schema-changes.history", "truncate.handling.mode": "include", "provide.transaction.metadata": "true" }
}

      2. Run in Postgres:

      CREATE TABLE truncate_table (f1 INTEGER PRIMARY KEY);
INSERT INTO truncate_table VALUES (1),(2),(3);
TRUNCATE TABLE truncate_table;\{code}
      Show
      0. Start debezium container: KEY_CONVERTER: io.confluent.connect.avro.AvroConverter VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter 1 Configure Debezium: { "name" : "psql-connector" , "config" : { "connector.class" : "io.debezium.connector.postgresql.PostgresConnector" , "database.hostname" : "postgres" , "database.port" : "5432" , "database.user" : "debezium" , "database.password" : "debezium" , "database.dbname" : "postgres" , "database.server.name" : "postgres" , "plugin.name" : "pgoutput" , "slot.name" : "tester" , "database.history.kafka.bootstrap.servers" : "kafka:9092" , "database.history.kafka.topic" : "schema-changes.history" , "truncate.handling.mode" : "include" , "provide.transaction.metadata" : " true " } } 2. Run in Postgres: CREATE TABLE truncate_table (f1 INTEGER PRIMARY KEY); INSERT INTO truncate_table VALUES (1),(2),(3); TRUNCATE TABLE truncate_table;\{code}

    Description

      Postgres version: 11.4
      Debezium version: debezium/connect:1.5, from container

      Attempting to run TRUNCATE in Postgres results in the following error in Debezium:

      2021-04-26 11:28:56,958 ERROR  ||  WorkerSourceTask{id=psql-connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted   [org.apache.kafka.connect.runtime.WorkerTask]
org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler
	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:206)
	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:132)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.convertTransformedRecord(WorkerSourceTask.java:311)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:340)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:264)
	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)
	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.kafka.connect.errors.DataException: Found null value for non-optional schema
	at io.confluent.connect.avro.AvroData.validateSchemaValue(AvroData.java:1177)
	at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:381)
	at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:344)
	at io.confluent.connect.avro.AvroConverter.fromConnectData(AvroConverter.java:89)
	at org.apache.kafka.connect.storage.Converter.fromConnectData(Converter.java:63)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.lambda$convertTransformedRecord$1(WorkerSourceTask.java:311)
	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:156)
	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:190)
	... 11 more
2021-04-26 11:28:56,959 INFO   ||  Stopping down connector   [io.debezium.connector.common.BaseSourceTask]

      Attachments

        Activity

          People

            jpechane Jiri Pechanec
            philip_stoev Philip Stoev (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: