Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1628

Schema recovery should be atomic operation

    XMLWordPrintable

Details

    • Feature Request
    • Resolution: Unresolved
    • Major
    • None
    • 1.0.0.Beta3
    • mysql-connector
    • None
    • 0
    • 0% 0%

    Description

      Unfortunate timing can cause corrupted schema history topic during schema recovery operation. That corruption can be minor and only detected hours later when suddenly debezium encounters data change event for table not found in recovered schema.

      To make it more robust could schema recovery detect the last attempt was not properly finished and either fail task or automatically start the recovery again from beginning.

      Steps to reproduce:
      1. have running connector, have large amount schemas in db.
      2. update job config, change the history topic to some new topic, change snapshot.mode to schema_only_recovery
      3. recovery operation begins, let it complete ~50%.
      4. restart the docker container
      5. container stats, detects that history topic exists, offsets exists = all good
      6. wait for connector to complete starting
      7. make mysql data change in some table where recovery snapshot didn't yet cover
      8. witness task crash :/

      Attachments

        Activity

          People

            Unassigned Unassigned
            pimpelsang Eero Koplimets (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: