Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1071

Support Postgresql arbitrary-dimensional array

    Details

    • Type: Feature Request
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.9.0.Beta2
    • Fix Version/s: None
    • Component/s: postgresql-connector
    • Labels:
      None
    • Environment:
      • Debezium 0.9.0Beta2
      • wal2json commit d4c0e814696695bbf853c48b38b7479e0f83f6c7
      • Postgresql 11.1
      • confluent platform 5.1
      • zookeeper 3.4.13

      Description

      Current behavior

      Debezium Postgresql connector does not support arbitrary-dimensional array. For example,

      In case of two-dimensional array of INT
      Postgresql connector warns it and the result value is null.
      Avro schema

      {
        "name":"c_arr_int",
        "type":[
          "null",
          {
            "type":"array",
            "items":[
              "null",
              "int"
            ],
            "connect.parameters":{
              "__debezium.source.column.type":"_INT4",
              "__debezium.source.column.length":"10"
            }
          }
        ],
        "default":null
      }
      

      the warning message

      [stgresValueConverter] Unexpected value for JDBC type 4 and column c_arr_int-element int4(10, 0) DEFAULT VALUE NULL: class=Integer[]
      

      In case of two-dimensional array of TEXT
      Postgresql connector does not warn as follow and the result value seems to be casted string.
      Avro schema

      {
        "name":"c_arr_text",
        "type":[
          "null",
          {
            "type":"array",
            "items":[
              "null",
              "string"
            ],
            "connect.parameters":{
              "__debezium.source.column.type":"_TEXT",
              "__debezium.source.column.length":"2147483647"
            }
          }
        ],
        "default":null
      }
      

      result value

      ['[Ljava.lang.String;@3ce73358']
      

      Request

      1. Warning should be logged.
        At least, as long as Debezium does not support arbitrary-dimensional array, warning should be logged when it finds a two or more dimensional array of any data type. Moreover, if "include.unknown.datatypes" is enabled, raw binary representation of a column should be set as the result.
      2. Support arbitrary-dimensional array
        I guess the reason why Debezium does not support it is that both avro and protobuf can not define an arbitrary-dimensional array schema.
        My suggestion is to define such array schema as a linear(flatten) array with the dimension infomation.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  takaomag Takao Magoori
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  1 Start watching this issue

                  Dates

                  • Created:
                    Updated: