Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7719

More logging when skipping events

XMLWordPrintable

    • False
    • None
    • False

      Which use case/requirement will be addressed by the proposed feature?

      When using Postgres connector and Debezium has to process a larger WAL, it could happen that the application looks like stuck because it skips a lot of messages, and so it doesn't call the handleBatch method and also doesn't print any log message.
      For example, if the WAL is sorted and the events from the watched tables are at the end.
      In this situation, it would be useful to see INFO or WARN messages that the application is running, but skipping a lot of events.

      Implementation ideas (optional)

      Actually, there is code that is very close to this: https://github.com/debezium/debezium/blob/main/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/PostgresStreamingChangeEventSource.java#L401
      Its concern is WAL size, but it looks like it doesn't count with all the event types - in our case, it reported 10000 skipped events after 50 minutes. When processing events (not skipping), the machine was able to process 20k events per second, so this doesn't look like an performance issue.
      So if this code could be improved to count with all events types, and GROWING_WAL_WARNING_LOG_INTERVAL constant to be configurable, it would be perfect.

      What is similar is the logStatistics method in BaseSourceTask but it is IMHO too high-level and doesn't know about skipped events.

            vjuranek@redhat.com Vojtech Juranek
            michal.augustyn Michal Augustýn
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: