Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1054

Column blacklisting in SQL Server connector

    XMLWordPrintable

Details

    • Enhancement
    • Resolution: Obsolete
    • Major
    • None
    • None
    • sqlserver-connector
    • None

    Description

      This feature has been already implemented in MySQL connector.

      Motivation:
      1) Some columns might contain sensitive data which shouldn't be captured

      2) SQL Sever CDC mechanism has a limitation on column size

      According to docs:

      By default, the maximum size that can be added to a captured column in a single INSERT, UPDATE, WRITETEXT, or UPDATETEXT statement is 65,536 bytes or 64 KB. To increase this size to support larger LOB data, use the Configure the max text repl size Server Configuration Option to specify a larger maximum size. For more information, see Configure the max text repl size Server Configuration Option.

      We might want to skip huge columns if they are not needed.

      CDC mechanism already supports columns white-listing:

      [ @captured_column_list = ] 'captured_column_list'
      Identifies the source table columns that are to be included in the change table. captured_column_list is nvarchar(max) and can be NULL. If NULL, all columns are included in the change table.

      Column names must be valid columns in the source table. Columns defined in a primary key index, or columns defined in an index referenced by index_name must be included.

      captured_column_list is a comma-separated list of column names. Individual column names within the list can be optionally quoted by using either double quotation marks ("") or square brackets ([]). If a column name contains an embedded comma, the column name must be quoted.

      captured_column_list cannot contain the following reserved column names: __$start_lsn, __$end_lsn, __$seqval, __$operation, and __$update_mask.

      Attachments

        Issue Links

          Activity

            People

              grzegorz.kolakowski Grzegorz Kołakowski (Inactive)
              grzegorz.kolakowski Grzegorz Kołakowski (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: