Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-746

Document rules for "slot.name" property of the Postgres connector

    XMLWordPrintable

Details

    Description

      The inclusion of "slot.name" does not work with Postgres and leaves things in a non-obvious state. After creating a new connector with "slot.name" specified:

      • curl returns 201 Created, so caller thinks it worked

        HTTP/1.1 201 Created
        Date: Tue, 19 Jun 2018 20:11:19 GMT
        Location: http://localhost:8083/connectors/connect-xyz
        Content-Type: application/json
        Content-Length: 619
        Server: Jetty(9.2.24.v20180105)

      • Kafka Connect logs show error

        [2018-06-19 11:01:09,589] INFO 127.0.0.1 - - [19/Jun/2018:17:01:08 +0000] "POST /connectors/ HTTP/1.1" 201 619 1285 (org.apache.kafka.connect.runtime.rest.RestServer:60)
        [2018-06-19 11:01:11,267] INFO user 'test' connected to database 'test' on PostgreSQL 10.4 on x86_64-apple-darwin17.5.0, compiled by Apple LLVM version 9.1.0 (clang-902.0.39.1), 64-bit with roles:
        role 'pg_read_all_settings' [superuser: false, replication: false, inherit: true, create role: false, create db: false, can log in: false]
        role 'pg_stat_scan_tables' [superuser: false, replication: false, inherit: true, create role: false, create db: false, can log in: false]
        role 'pg_monitor' [superuser: false, replication: false, inherit: true, create role: false, create db: false, can log in: false]
        role 'pg_read_all_stats' [superuser: false, replication: false, inherit: true, create role: false, create db: false, can log in: false]
        role 'pg_signal_backend' [superuser: false, replication: false, inherit: true, create role: false, create db: false, can log in: false]
        role 'test' [superuser: true, replication: false, inherit: true, create role: true, create db: true, can log in: true] (io.debezium.connector.postgresql.PostgresConnectorTask:74)
        [2018-06-19 11:01:11,269] INFO Found previous offset source_info[server='localhost', lsn=1A/57E94158, txId=9087, useconds=1529426992355000, snapshot=false] (io.debezium.connector.postgresql.PostgresConnectorTask:88)
        [2018-06-19 11:01:11,269] INFO Previous snapshot has completed successfully, streaming logical changes from last known position (io.debezium.connector.postgresql.PostgresConnectorTask:103)
        [2018-06-19 11:01:11,271] INFO Requested thread factory for connector PostgresConnector, id = localhost named = records-stream-producer (io.debezium.util.Threads:231)
        [2018-06-19 11:01:12,053] DEBUG No replication slot 'xyz-test' is present for plugin 'decoderbufs' and database 'test' (io.debezium.connector.postgresql.connection.PostgresConnection:141)
        [2018-06-19 11:01:12,054] DEBUG Creating new replication slot 'xyz-test' for plugin 'DECODERBUFS' (io.debezium.connector.postgresql.connection.PostgresReplicationConnection:95)
        [2018-06-19 11:01:12,065] INFO WorkerSourceTask
        Unknown macro: {id=connect-xyz-test-0}
        Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:328)
        [2018-06-19 11:01:12,066] INFO WorkerSourceTask

        flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:345)
        [2018-06-19 11:01:12,066] ERROR WorkerSourceTask
        Unknown macro: {id=connect-xyz-test-0}
        Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:172)
        io.debezium.jdbc.JdbcConnectionException: ERROR: syntax error
        at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.initReplicationSlot(PostgresReplicationConnection.java:136)
        at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.<init>(PostgresReplicationConnection.java:79)
        at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.<init>(PostgresReplicationConnection.java:38)
        at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$ReplicationConnectionBuilder.build(PostgresReplicationConnection.java:349)
        at io.debezium.connector.postgresql.PostgresTaskContext.createReplicationConnection(PostgresTaskContext.java:63)
        at io.debezium.connector.postgresql.RecordsStreamProducer.<init>(RecordsStreamProducer.java:75)
        at io.debezium.connector.postgresql.PostgresConnectorTask.start(PostgresConnectorTask.java:105)
        at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:45)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:164)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        Caused by: org.postgresql.util.PSQLException: ERROR: syntax error
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428)
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354)
        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301)
        at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287)
        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264)
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:260)
        at org.postgresql.replication.fluent.logical.LogicalCreateSlotBuilder.make(LogicalCreateSlotBuilder.java:48)
        at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.initReplicationSlot(PostgresReplicationConnection.java:102)
        ... 15 more
        [2018-06-19 11:01:12,068] ERROR WorkerSourceTask

        Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:173)

      • Postgres logs show error: "ERROR: syntax error"

        2018-06-19 11:01:12.053 MDT [17813] LOG: execute <unnamed>: select * from pg_replication_slots where slot_name = $1 and database = $2 and plugin = $3
        2018-06-19 11:01:12.053 MDT [17813] DETAIL: parameters: $1 = 'xyz-test', $2 = 'test', $3 = 'decoderbufs'
        2018-06-19 11:01:12.054 MDT [17813] LOG: disconnection: session time: 0:00:00.776 user=test database=test host=127.0.0.1 port=56011
        2018-06-19 11:01:12.057 MDT [17814] LOG: connection received: host=127.0.0.1 port=56012
        2018-06-19 11:01:12.058 MDT [17814] LOG: replication connection authorized: user=test
        2018-06-19 11:01:12.063 MDT [17814] ERROR: syntax error

      • Postgres shows no rows in pg_replication_slots

        # select * FROM pg_replication_slots;
        slot_name | plugin | slot_type | datoid | database | temporary | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn
        ----------+---------------------------------------------------------------------------------------------------------------
        (0 rows)

      once in this state, the following gets it working again:

      versions in use:

      • Debezium 0.7.5
      • Postgres 10.4
      • Kafka 2.11
      • Mac OS X 10.13.5

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              kerdener Kaan Erdener (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: