Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-541

Provide an option to only store DDL statements referring to captured tables in DB history topic

    XMLWordPrintable

    Details

    • Type: Feature Request
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.7.2
    • Component/s: mysql-connector
    • Labels:
      None
    • Environment:

      production

      Description

      After upgrade to version 0.7 history topic is 1.1 GB which is huge
      No matter how much we will increase the
      database.history.kafka.recovery.attempts
      database.history.kafka.recovery.poll.interval.ms
      we get this error
      The database history couldn't be recovered. Consider to increase the value for database.history.kafka.recovery.poll.interval.ms

      We are getting this error below.
      I have to mention i am using only 33 table and my history is 2 weeks old and my topic is huge .1.1 GB
      I think it will be issue for everyone.
      I think we should add the history only the white-list (33) not all.
      This is not a enhancement rather a blocker bug since the only bypass u have is to delete the topic and start from blank each time it's happen.

      the full error:

      "2018-01-14 13:54:51,249] ERROR WorkerSourceTask{id=ks-connector4032-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:172)
      org.apache.kafka.connect.errors.ConnectException: java.lang.IllegalStateException: The database history couldn't be recovered. Consider to increase the value for database.history.kafka.recovery.poll.interval.ms
      	at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:194)
      	at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:157)
      	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
      	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.IllegalStateException: The database history couldn't be recovered. Consider to increase the value for database.history.kafka.recovery.poll.interval.ms
      	at io.debezium.relational.history.KafkaDatabaseHistory.recoverRecords(KafkaDatabaseHistory.java:196)
      	at io.debezium.relational.history.AbstractDatabaseHistory.recover(AbstractDatabaseHistory.java:61)
      	at io.debezium.connector.mysql.MySqlSchema.loadHistory(MySqlSchema.java:249)
      	at io.debezium.connector.mysql.MySqlTaskContext.loadHistory(MySqlTaskContext.java:122)
      	at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:80)
      	... 8 more
      "
      

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                jpechanec Jiri Pechanec
                Reporter:
                shlomihassan shlomi hassan
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: