Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-512

For old connector OID should be used to detect schema change

    Details

    • Steps to Reproduce:
      Hide

      When the postgis extension is created, PG sets up the `spatial_ref_sys` table with 5500 rows.

      For tests I want to ignore it, so https://github.com/debezium/debezium/pull/374/files#diff-062e7af96d2f14c29aff466562a13e9aR122 waits for <20s for those records to load (into `postgis.spatial_ref_sys`, so dropped by the filter), before continuing on with testing actual values in `public.postgis_table`.

      This is now taking 5+ minutes (Eclipse debug).

      Appears to happen with both wal2json and decoderbufs. The above timings are from wal2json though.

      Show
      When the postgis extension is created, PG sets up the `spatial_ref_sys` table with 5500 rows. For tests I want to ignore it, so https://github.com/debezium/debezium/pull/374/files#diff-062e7af96d2f14c29aff466562a13e9aR122 waits for <20s for those records to load (into `postgis.spatial_ref_sys`, so dropped by the filter), before continuing on with testing actual values in `public.postgis_table`. This is now taking 5+ minutes (Eclipse debug). Appears to happen with both wal2json and decoderbufs. The above timings are from wal2json though.

      Description

      I think the changes for DBZ-379 is doing some bad things. Like querying the database every received message for updated type information...

      2017-12-15 13:03:33,240 DEBUG  Postgres|test_server|records-stream-producer  received new message at position 0/EBB2ACF
      io.debezium.connector.postgresql.connection.wal2json.Wal2JsonReplicationMessage@1e753d69   [io.debezium.connector.postgresql.RecordsStreamProducer]
      2017-12-15 13:03:33,240 DEBUG  Postgres|test_server|records-stream-producer  detected new type for column 'srid', old type was '4', new type is '23'; refreshing table schema   [io.debezium.connector.postgresql.RecordsStreamProducer]
      2017-12-15 13:03:33,267 DEBUG  Postgres|test_server|records-stream-producer  Connected to jdbc:postgresql://localhost:5432/postgres with {sslmode=disable, user=postgres, password=postgres, server.name=test_server, assumeMinServerVersion=9.4}   [io.debezium.jdbc.JdbcConnection]
      2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  refreshing DB schema for table 'postgis.spatial_ref_sys'   [io.debezium.connector.postgresql.PostgresSchema]
      2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  Mapping table 'postgis.spatial_ref_sys' to schemas under 'test_server.postgis.spatial_ref_sys'   [io.debezium.relational.TableSchemaBuilder]
      2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  - field 'srid' (INT32) from column srid int4(10,0) NOT NULL   [io.debezium.relational.TableSchemaBuilder]
      2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  - field 'srid' (INT32) from column srid int4(10,0) NOT NULL   [io.debezium.relational.TableSchemaBuilder]
      2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  - field 'auth_name' (STRING) from column auth_name varchar(256,0)   [io.debezium.relational.TableSchemaBuilder]
      2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  - field 'auth_srid' (INT32) from column auth_srid int4(10,0)   [io.debezium.relational.TableSchemaBuilder]
      2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  - field 'srtext' (STRING) from column srtext varchar(2048,0)   [io.debezium.relational.TableSchemaBuilder]
      2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  - field 'proj4text' (STRING) from column proj4text varchar(2048,0)   [io.debezium.relational.TableSchemaBuilder]
      2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  Mapped primary key for table 'postgis.spatial_ref_sys' to schema: {"name" : "test_server.postgis.spatial_ref_sys.Key", "type" : "STRUCT", "optional" : "false", "fields" : [{"name" : "srid", "index" : "0", "schema" : {"type" : "INT32", "optional" : "false"}}]}   [io.debezium.relational.TableSchemaBuilder]
      2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  Mapped columns for table 'postgis.spatial_ref_sys' to schema: {"name" : "test_server.postgis.spatial_ref_sys.Value", "type" : "STRUCT", "optional" : "true", "fields" : [{"name" : "srid", "index" : "0", "schema" : {"type" : "INT32", "optional" : "false"}}, {"name" : "auth_name", "index" : "1", "schema" : {"type" : "STRING", "optional" : "true"}}, {"name" : "auth_srid", "index" : "2", "schema" : {"type" : "INT32", "optional" : "true"}}, {"name" : "srtext", "index" : "3", "schema" : {"type" : "STRING", "optional" : "true"}}, {"name" : "proj4text", "index" : "4", "schema" : {"type" : "STRING", "optional" : "true"}}]}   [io.debezium.relational.TableSchemaBuilder]
      2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  sending create event 'SourceRecord{sourcePartition={server=test_server}, sourceOffset={lsn=247147215, txId=1152, ts_usec=1513342952883563000}} ConnectRecord{topic='test_server.postgis.spatial_ref_sys', kafkaPartition=null, key=Struct{srid=2113}, value=Struct{after=Struct{srid=2113,auth_name=EPSG,auth_srid=2113,srtext=PROJCS["NZGD2000...
      

      I guess the key bit is

      detected new type for column 'srid', old type was '4', new type is '23'; refreshing table schema  
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  robert.coup Robert Coup
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: