Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7608

Poor snapshot performance during schema snapshot DDL processing

    XMLWordPrintable

Details

    • Important

    Description

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      This issue has been found on 2.5.2.Final and 2.6.0.Alpha2. We have not tested previous versions to these but we know that 2.2.1.Final does NOT have this bug as that is what we run in production.

      What is the connector configuration?

      We have a MYSQL Debezium connector.

      What is the captured database version and mode of depoyment?

      (E.g. on-premises, with a specific cloud provider, etc.)

      MySQL Debezium connector with Percona Server

      What behaviour do you expect?

      We expect this phase of the schema snapshot to be very quick as it was in the past. We have roughly 500 databases on this MYSQL instance and 700,000 tables. On 2.2.1.Final it takes about 5-10 minutes for all the DROP TABLE DDL to be captured. This is what we expect as a standard. In addition, the CREATE TABLE DDL also seems significantly slower.

      What behaviour do you see?

      Processing snapshot DDL 'DROP TABLE IF EXISTS `db1`.`table1`' for database 'db1'
      - this occurs for 700,000 tables
      - on 2.2.1.Final it takes around 5 minutes and moves on to CREATE DDL
      - on 2.6.0.Alpha2/2.5.2.Final it takes over 4 hours then moves on to CREATE DDL
       
      Processing snapshot DDL 'CREATE TABLE'...
      - this occurs for 700,000 tables
      - on 2.2.1.Final it takes roughly an hour
      - on 2.6.0.Alpha2/2.5.2.Final it takes over 4 hours

      Building the internal schema history...
      - on 2.2.1.Final it takes roughly an hour
      - on 2.6.0.Alpha2/2.5.2.Final it takes roughly 3 hours
       
      Final results,
      2.6.0.Alpha2/2.5.2.Final: "Writes to MySQL tables prevented for a total of 11:27:43.904" (it took 11 hours)
      2.2.1.Final: Didnt capture that same log ^, but it was streaming in rougly 2-2.5 hours

      Do you see the same behaviour using the latest relesead Debezium version?

      (Ideally, also verify with latest Alpha/Beta/CR version)

      yes this has been tested on 2.6.0.Alpha2 and 2.5.2.Final

      Do you have the connector logs, ideally from start till finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      I simply can say that in DEBUG level, we see this log

      Processing snapshot DDL 'DROP TABLE IF EXISTS `db1`.`table1`' for database 'db1'

      for hours, getting each table in a way that is much slower than 2.2.1.Final

      How to reproduce the issue using our tutorial deployment?

      You would most likely need a large amount of DB's and a large amount of tables to notice this performance issue.

       

      This may be related to this

      https://github.com/debezium/debezium/pull/4824#discussion_r1378810435

      Attachments

        1. logs.txt.
          13 kB
        2. snapshot_logs.txt
          145 kB

        Issue Links

          Activity

            People

              Unassigned Unassigned
              drew-vz Drew von Zweck
              Mario Fiore Vitale
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: