Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1771

Create example connector

XMLWordPrintable

      There should be a very basic connector implementation, based on the new framework (as used by SQL Server, Postgres, etc.) that shows external contributors how to implement a CDC connector based on Debezium infrastructure.

      The basic idea is to have a "database" represented by single file. Changes to that file will be tracked using Java's WatchService. The format of the file is like so:

      EventType;Id;FirstName;LastName;Salary;Active
      S|1|Alice|Smith|3000|true
      S|2|Bob|Miller|2000|false
      I|3|Bruce|Masters|4000|false
      U|1|Sarah|Smith|3000|false
      D|3|Bruce|Masters|4000|false
      EventType;Id;FirstName;LastName;Salary;Active;Birthday
      U|1|Sarah|Smith|3000|false|1980-12-31
      ...
      

      Event type would be one of (S)napshot, (I)nsert, (U)date, (D)elete. I.e. this file is like a log, not like a table.

      The snapshot change event source would read all the S events and then stop. The streaming source would read all the other rows and emit corresponding events (updates won't have the old value for sake of simplicity). Whenever the watch service detects a change of the file, the change event source will continue reading the file from the last processed offset (= line number) until the end.

      The implementation should show:

      • snapshot source
      • streaming source
      • hand-over from the former to the latter, using the line number as offset
      • offset handling
      • value type conversion, based on the column types
      • Evolution of schema
      • Handling of a configuration option (the path and name of the database file)

      Besides the implementation (which should really be as simple as possible), there should be accompanying documentation of the relevant pieces of the implementation.

            ccranfor@redhat.com Chris Cranford
            gunnar.morling Gunnar Morling
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: