Details
-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
0.9.4.Final
-
None
Description
When the Postgres connector comes across a timestamp value, it parses the value using the format yyyy-MM-dd HH:mm:ss[.S]. Unfortunately, there are dates in our database that do not fit his format because the years are larger than 4 digits. When the connector tries to parse them, it throws a DateTimeParseException and dies.
Is there a recommended way of handling this without making any code changes? For example, I will probably have to modify the connector and try to manually parse a value that falls outside of the yyyy-MM-dd HH:mm:ss[.S] format.
Stack trace available here is a gist: https://gist.github.com/jaydeesimon/e754b2b25327c9df0577df1f586bbf89
or see below
[2019-04-29 23:27:08,923] INFO WorkerSourceTask{id=ghr-connector3-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:199)
[2019-04-29 23:27:09,094] ERROR Cannot parse time/date value '29398-03-04 01:49:01.698797', expected format 'yyyy-MM-dd HH:mm:ss[.S]' (io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat:140)
[2019-04-29 23:27:09,095] ERROR unexpected exception while streaming logical changes (io.debezium.connector.postgresql.RecordsStreamProducer:146)
org.apache.kafka.connect.errors.ConnectException: java.time.format.DateTimeParseException: Text '29398-03-04 01:49:01.698797' could not be parsed at index 0
at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.format(DateTimeFormat.java:141)
at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.timestamp(DateTimeFormat.java:101)
at io.debezium.connector.postgresql.connection.wal2json.Wal2JsonReplicationMessage.getValue(Wal2JsonReplicationMessage.java:268)
at io.debezium.connector.postgresql.connection.wal2json.Wal2JsonReplicationMessage$1.getValue(Wal2JsonReplicationMessage.java:141)
at io.debezium.connector.postgresql.RecordsStreamProducer.columnValues(RecordsStreamProducer.java:476)
at io.debezium.connector.postgresql.RecordsStreamProducer.process(RecordsStreamProducer.java:268)
at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$streamChanges$1(RecordsStreamProducer.java:134)
at io.debezium.connector.postgresql.connection.wal2json.NonStreamingWal2JsonMessageDecoder.processMessage(NonStreamingWal2JsonMessageDecoder.java:70)
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.deserializeMessages(PostgresReplicationConnection.java:269)
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.read(PostgresReplicationConnection.java:254)
at io.debezium.connector.postgresql.RecordsStreamProducer.streamChanges(RecordsStreamProducer.java:134)
at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$start$0(RecordsStreamProducer.java:120)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.time.format.DateTimeParseException: Text '29398-03-04 01:49:01.698797' could not be parsed at index 0
at java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2046)
at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1948)
at java.base/java.time.LocalDateTime.parse(LocalDateTime.java:492)
at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.lambda$timestamp$0(DateTimeFormat.java:101)
at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.format(DateTimeFormat.java:138)
... 16 more
[2019-04-29 23:27:09,612] INFO Cluster ID: f8EHzNZpSJSrVpxMaaR1EA (org.apache.kafka.clients.Metadata:285)
[2019-04-29 23:27:09,649] INFO WorkerSourceTask{id=ghr-connector3-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:397)
[2019-04-29 23:27:09,649] INFO WorkerSourceTask{id=ghr-connector3-0} flushing 5 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:414)
[2019-04-29 23:27:09,658] INFO WorkerSourceTask{id=ghr-connector3-0} Finished commitOffsets successfully in 9 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:496)
[2019-04-29 23:27:09,659] ERROR WorkerSourceTask{id=ghr-connector3-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:177)
org.apache.kafka.connect.errors.ConnectException: An exception ocurred in the change event producer. This connector will be stopped.
at io.debezium.connector.base.ChangeEventQueue.throwProducerFailureIfPresent(ChangeEventQueue.java:170)
at io.debezium.connector.base.ChangeEventQueue.poll(ChangeEventQueue.java:151)
at io.debezium.connector.postgresql.PostgresConnectorTask.poll(PostgresConnectorTask.java:159)
at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:244)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:220)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.kafka.connect.errors.ConnectException: java.time.format.DateTimeParseException: Text '29398-03-04 01:49:01.698797' could not be parsed at index 0
at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.format(DateTimeFormat.java:141)
at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.timestamp(DateTimeFormat.java:101)
at io.debezium.connector.postgresql.connection.wal2json.Wal2JsonReplicationMessage.getValue(Wal2JsonReplicationMessage.java:268)
at io.debezium.connector.postgresql.connection.wal2json.Wal2JsonReplicationMessage$1.getValue(Wal2JsonReplicationMessage.java:141)
at io.debezium.connector.postgresql.RecordsStreamProducer.columnValues(RecordsStreamProducer.java:476)
at io.debezium.connector.postgresql.RecordsStreamProducer.process(RecordsStreamProducer.java:268)
at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$streamChanges$1(RecordsStreamProducer.java:134)
at io.debezium.connector.postgresql.connection.wal2json.NonStreamingWal2JsonMessageDecoder.processMessage(NonStreamingWal2JsonMessageDecoder.java:70)
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.deserializeMessages(PostgresReplicationConnection.java:269)
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.read(PostgresReplicationConnection.java:254)
at io.debezium.connector.postgresql.RecordsStreamProducer.streamChanges(RecordsStreamProducer.java:134)
at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$start$0(RecordsStreamProducer.java:120)
... 5 more
Caused by: java.time.format.DateTimeParseException: Text '29398-03-04 01:49:01.698797' could not be parsed at index 0
at java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2046)
at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1948)
at java.base/java.time.LocalDateTime.parse(LocalDateTime.java:492)
at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.lambda$timestamp$0(DateTimeFormat.java:101)
at io.debezium.connector.postgresql.connection.wal2json.DateTimeFormat$ISODateTimeFormat.format(DateTimeFormat.java:138)
... 16 more
[2019-04-29 23:27:09,660] ERROR WorkerSourceTask{id=ghr-connector3-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:178)
Attachments
Issue Links
- duplicates
-
-
- Closed
-
