Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-998

MongoDB initial sync process should be more robust

    XMLWordPrintable

Details

    • Enhancement
    • Resolution: Obsolete
    • Major
    • None
    • 1.7.0.CR1
    • mongodb-connector
    • None

    Description

      Based on experience and looking into the source code, Debezium's MongoDB Connector's behavior is to restart the initial sync process whenever there is any interruption in the process - e.g. Kafka Connect service restart, task rebalancing, exception, ... This is also documented in previous Issues:
      https://issues.jboss.org/projects/DBZ/issues/DBZ-182?filter=allissues
      https://issues.jboss.org/browse/DBZ-712
      https://issues.jboss.org/browse/DBZ-838

      This poses a problem in cases where the initial sync process takes a long time to finish (even several days). The chance of some interruption event occuring is growing with the length of initial sync operation, making it very difficult and that much more time-consuming to get to the oplog-reading state.

      We propose a modification to the behavior that will allow Debezium to continue and finish already started initial sync process when recovering from any interruption event (given all other constraints remain valid, such oplog not rolling over in the meantime, data consistency).

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              dalibor.frivaldsky.memsource Dalibor Frívaldský (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: