Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-800

PostgreSQL LSNs are not committed when receiving events for filtered-out tables

    XMLWordPrintable

Details

    Description

      If a PostgreSQL table is ignored via the table.whitelist or table.blacklist connector option, Debezium ignores WAL events on that table. However, it does not commit the LSNs. This causes WAL files to build up on the database until write activity happens on the whitelisted tables. This may cause problems for tables with infrequent writes.

      From the code, it seems that LSN commits are triggered whenever the Kafka offsets are committed:
      https://github.com/debezium/debezium/blob/v0.8.0.Final/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/PostgresConnectorTask.java#L136

      However, table filtering bypasses that code:
      https://github.com/debezium/debezium/blob/v0.8.0.Final/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/RecordsStreamProducer.java#L229

      This bug might also be occurring when the table operation is unknown/ignored:
      https://github.com/debezium/debezium/blob/v0.8.0.Final/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/RecordsStreamProducer.java#L254

      Attachments

        Activity

          People

            gunnar.morling Gunnar Morling
            csang1 Christopher Sang (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: