This ticket requests the ability to monitor the progress for each table within an incremental snapshot request. For context, we are looking to move away from pausing long-running connectors and creating a snapshot-only connector when we determine a table within our Read-Only MySQL database needs to be re-snapshotted. 

    Currently, a benefit of this method is that we can carefully monitor the snapshot-only connector from start to finish, ensuring the process executes as expected via the SnapshotCompleted  metric [1].

    When considering moving to Incremental Snapshotting, using a Kafka Signal Topic to start a snapshot for a set of tables, we have realized that the actual process of Incremental Snapshotting is a black box as far as tracking the snapshot progress of each table programmatically. From the point of producing the signal topic, we don’t know what is happening internally in Debezium with each table's snapshot.

We would like to request that the following information is exposed:

• An incremental snapshot request was received/processed (perhaps via a Kafka topic or Debezium metric?)
• The status of each table within an incremental snapshot request
• Ideally, this information would persist beyond the life of a connector, maybe in a Kafka topic

    Chris Cranford mentioned that similar information could be “deduced indirectly” using existing Debezium metrics (RemainingTableCount and TotalTableCount) [2], but we have found that to adopt Incremental Snapshotting safely, these might not be sufficient in making sure a snapshot has started/finished at the table level granularity.

[1]: https://debezium.io/documentation/reference/1.9/connectors/mysql.html#connectors-snaps-metric-snapshotcompleted_mysql

[2]:

https://groups.google.com/g/debezium/c/LbU18ejfRHk/m/HD2gTcMRAAAJ?utm_medium=email&utm_source=footer

[3]: https://debezium.io/documentation/reference/1.9/configuration/signalling.html#debezium-signaling-description-of-required-structure-of-a-signaling-data-collection