Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3168

Forever stuck with new binlog parser (1.3 and later) when processing big JSON column data

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • 1.4.2.Final
    • 1.3.0.Alpha1, 1.4.1.Final
    • mysql-connector
    • None
    • False
    • False
    • Undefined
    • Hide

      create table into mysql database

      CREATE TABLE `events_blackhole` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `topic` varchar(100) NOT NULL,
  `message_key` varchar(255) DEFAULT NULL,
  `message_value` json NOT NULL,
  `message_headers` json DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8;

      2. connect debezium
      3. run attached file as insert statement

      Watch debezium container CPU go 100% and no future events nor connector fail with error.

      Show
      create table into mysql database CREATE TABLE `events_blackhole` ( `id` int (11) unsigned NOT NULL AUTO_INCREMENT, `topic` varchar(100) NOT NULL, `message_key` varchar(255) DEFAULT NULL, `message_value` json NOT NULL, `message_headers` json DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8; 2. connect debezium 3. run attached file as insert statement Watch debezium container CPU go 100% and no future events nor connector fail with error.

    Description

      After upgrading debezium from 1.2 to 1.4 in production we noticed that in some cases debezium got infinitely stuck processing particular event. CPU at 100%... Restart or downgrade to 1.3 didn't solve it. After downloading 1.2 it continue as if normal.

      After discussing on gitter chat we extracted debug level logs of it and also thread dump.  From there it was clearly issue with thead that handles JSON parsing in binlog.

      So afterwards did a mysql dump and found the exact event that caused it. Also reproduce it locally.

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. DBZ-3106 Forever stuck with new binlog parser (1.3 and later) when processing big JSON column data

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed

          Activity

            People

              jpechane Jiri Pechanec
              pimpelsang Eero Koplimets (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: