Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7488

Poor snapshot performance with new reselect SMT

    XMLWordPrintable

Details

    Description

      What Debezium connector do you use and what version?

      2.6.0.Alpha2

      What is the connector configuration?

      { "connector.class": "io.debezium.connector.oracle.OracleConnector", "key.converter": "io.confluent.connect.avro.AvroConverter", "key.converter.schema.registry.url": "http://kafka:8081", "value.converter": "io.confluent.connect.avro.AvroConverter", "value.converter.schema.registry.url": "http://kafka:8081", "schema.name.adjustment.mode": "avro", "database.hostname": "dbhost", "database.port": "1521", "database.user": "user", "database.password": "pw", "database.dbname": "dbname", "database.pdb.name": "pdbname", "topic.prefix": "ASE3", "table.include.list": "ASEDBUSR.FDOSSDECKPARAMATTR", "snapshot.select.statement.overrides": "ASEDBUSR.FDOSSDECKPARAMATTR", "snapshot.select.statement.overrides.ASEDBUSR.FDOSSDECKPARAMATTR": "SELECT * FROM ASEDBUSR.FDOSSDECKPARAMATTR ORDER BY CREATED ASC, PKEY ASC", "snapshot.fetch.size": "10000", "snapshot.max.threads": 2, "event.processing.failure.handling.mode": "fail", "tombstones.on.delete": "true", "message.key.columns": "ASEDBUSR.FDOSSDECKPARAMATTR:BOID", "snapshot.mode": "initial", "log.mining.strategy": "online_catalog", "log.mining.archive.destination.name": "LOG_ARCHIVE_DEST_10", "lob.enabled": "true", "log.mining.transaction.retention.hours": "24", "internal.log.mining.log.query.max.retries": "10", "time.precision.mode": "connect", "tasks.max": "1", "schema.history.internal.store.only.captured.tables.ddl": "true", "schema.history.internal.kafka.bootstrap.servers": "kafka:9092", "schema.history.internal.kafka.topic": "ase.schema-changes.asedbusr", "schema.history.internal.consumer.security.protocol": "SASL_SSL", "schema.history.internal.producer.security.protocol": "SASL_SSL", "schema.history.internal.consumer.sasl.mechanism": "SCRAM-SHA-512", "schema.history.internal.producer.sasl.mechanism": "SCRAM-SHA-512", "schema.history.internal.consumer.sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"user\" password=\"pw\";", "schema.history.internal.producer.sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"user\" password=\"pw\";", "notification.enabled.channels": "sink", "notification.sink.topic.name": "ASE3.SNAPSHOT.STATUS", "transforms": "changes,moveHeadersToValue,unwrap", "transforms.changes.type": "io.debezium.transforms.ExtractChangedRecordState", "transforms.changes.header.changed.name": "Changed", "transforms.moveHeadersToValue.type": "io.debezium.transforms.HeaderToValue", "transforms.moveHeadersToValue.headers": "Changed", "transforms.moveHeadersToValue.fields": "changes", "transforms.moveHeadersToValue.operation": "move", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.add.fields": "source.table:META_SRC_TABLENAME,source.ts_ms:META_SRC_TS,op:META_SRC_OP,source.scn:META_SRC_SCN,source.snapshot:META_SRC_SNAPSHOT,source.user_name:META_SRC_USER,changes:META_SRC_CHANGES", "transforms.unwrap.delete.handling.mode": "rewrite", "transforms.unwrap.drop.tombstones": "false", "post.processors": "reselector",    "reselector.type": "io.debezium.processors.reselect.ReselectColumnsPostProcessor",    "reselector.reselect.columns.include.list": "ASEDBUSR.FDOSSDECKPARAMATTR:TEXTVALUE,ASEDBUSR.FDOSSDECKPARAMATTR:BIGTEXTVALUE",    "reselector.reselect.unavailable.values": "true",    "reselector.reselect.null.values": "true" }

      What is the captured database version and mode of depoyment?

      Oracle 19c CDB on prem

      What behaviour do you expect?

      The same snapshot performance we had before (20'000 events / sec)

      What behaviour do you see?

      Poor snapshot performance (200 events / sec).

      Looks like the SMT is also active during the snapshot?!

      Attachments

        Issue Links

          Activity

            People

              ccranfor@redhat.com Chris Cranford
              renerh René Rütter
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: