Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1238

Add option to skip table locks when snapshotting

    XMLWordPrintable

Details

    • Feature Request
    • Resolution: Done
    • Major
    • 0.10.0.Beta3
    • 0.9.4.Final
    • postgresql-connector
    • None
    • 0
    • 0% 0%

    Description

      Currently, the snapshotting takes a 'share update exclusive' lock on every table it is snapshotting. In the code, this seems to be done to ensure that the schema doesn't change, but since this is done inside of a serializable transaction, it seems like it is likely redundant, as within this transaction the schema should remain the same for all tables, with any schemas changes being applied but not seen in this transaction (at least from what I understand of PG isolation levels).

      I think this might be done to ensure that no changes can happen until we are ready to consume from the slot so that we know the schema is consistent when we start the stream consumer.

      However, because those locks block vacuuming as well as schema changes and for large DBs the process to snapshot make take many hours, that may be undesirable to block changes or vacuums for such a long time. While this may put the user at risk of some schema inconsistency, in some cases, not having a guarantee of schema consistency is probably an okay trade off for more flexibility in running vacuums and making schema changes.

      Assuming that all I said above is correct and this lock isn't required for correctness of data, then it makes sense to add an option to opt out of those locks. This option could be `snapshot.skip.locks` and it would default to `false` so that the current behavior stays the same, but if turned on, no locks would be taken, which makes snapshotting lighter weight but at some potential schema inconsistency (which should eventually be fixed with other schema refreshes).

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              addisonj Addison Higham (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: