Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-142

NVARCHAR columns are not supported

    Details

    • Steps to Reproduce:
      Hide

      1. Run debezium configured for a MySQL database.
      2. Create a table with a NVARCHAR column:

      CREATE TABLE test2 (id INT PRIMARY KEY, someColumn NVARCHAR(100) NOT NULL);
      

      3. Check the debezium log output

      Show
      1. Run debezium configured for a MySQL database. 2. Create a table with a NVARCHAR column: CREATE TABLE test2 (id INT PRIMARY KEY , someColumn NVARCHAR(100) NOT NULL ); 3. Check the debezium log output

      Description

      Debezium seems to have problems when columns with type NVARCHAR appears in the binlog. NVARCHAR is equivalent to a VARCHAR CHARACTER SET utf8 (http://dev.mysql.com/doc/refman/5.7/en/charset-national.html).

      It works fine when the table exists when starting Debezium, then for the initial snapshot it is parsed as a regular VARCHAR and everything is OK:

      CREATE TABLE test (id INT PRIMARY KEY, someColumn NVARCHAR(100) NOT NULL);
      
      2016-10-26 08:36:08,757] INFO 	CREATE DATABASE test (io.debezium.connector.mysql.SnapshotReader:569)
      [2016-10-26 08:36:08,799] INFO 	USE test (io.debezium.connector.mysql.SnapshotReader:569)
      [2016-10-26 08:36:08,836] INFO 	CREATE TABLE `test` (
        `id` int(11) NOT NULL,
        `someColumn` varchar(100) CHARACTER SET utf8 NOT NULL,
        PRIMARY KEY (`id`)
      ) ENGINE=InnoDB DEFAULT CHARSET=latin1 (io.debezium.connector.mysql.SnapshotReader:569)
      [2016-10-26 08:36:08,876] INFO Step 7: releasing global read lock to enable MySQL writes (io.debezium.connector.mysql.SnapshotReader:325)
      

      But, if I create a new table with Debezium running with a NVARCHAR, so that we get the CREATE TABLE command in the binlog stream, then I get this error:

      CREATE TABLE test2 (id INT PRIMARY KEY, someColumn NVARCHAR(100) NOT NULL);
      
      [2016-10-26 08:36:50,373] ERROR Error parsing DDL statement and updating tables: CREATE TABLE test2 (id INT PRIMARY KEY, someColumn NVARCHAR(100) NOT NULL) (io.debezium.connector.mysql.MySqlSchema:328)
      io.debezium.text.ParsingException: Expecting ')' at line 1, column 60 but found '(':  someColumn NVARCHAR ===>> (100) NOT NULL)
      	at io.debezium.text.TokenStream.consume(TokenStream.java:694)
      	at io.debezium.connector.mysql.MySqlDdlParser.parseCreateDefinitionList(MySqlDdlParser.java:548)
      	at io.debezium.connector.mysql.MySqlDdlParser.parseCreateTable(MySqlDdlParser.java:365)
      	at io.debezium.connector.mysql.MySqlDdlParser.parseCreate(MySqlDdlParser.java:271)
      	at io.debezium.connector.mysql.MySqlDdlParser.parseNextStatement(MySqlDdlParser.java:146)
      	at io.debezium.relational.ddl.DdlParser.parse(DdlParser.java:285)
      	at io.debezium.relational.ddl.DdlParser.parse(DdlParser.java:266)
      	at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:326)
      	at io.debezium.connector.mysql.BinlogReader.handleQueryEvent(BinlogReader.java:390)
      	at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:295)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:877)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:735)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:451)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient$5.run(BinaryLogClient.java:632)
      	at java.lang.Thread.run(Thread.java:745)
      

      I assume this applies to the other synonyms for NVARCHAR listed here as well: http://dev.mysql.com/doc/refman/5.7/en/charset-national.html

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                depe Dennis Persson
                Reporter:
                depe Dennis Persson
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: