-
Bug
-
Resolution: Unresolved
-
Major
-
2.4.1.Final, 2.5.0.Beta1
-
None
-
False
-
None
-
False
In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.
Bug report
For bug reports, provide this information, please:
What Debezium connector do you use and what version?
SQL-Server connector 2.4.1 (https://repo1.maven.org/maven2/io/debezium/debezium-connector-sqlserver/2.4.1.Final/debezium-connector-sqlserver-2.4.1.Final-plugin.tar.gz)
What is the connector configuration?
KafkaConnect:
apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnect metadata: name: debezium-connect-cluster namespace: kafka annotations: strimzi.io/use-connector-resources: "true" spec: version: 3.6.0 replicas: 1 bootstrapServers: debezium-cluster-kafka-bootstrap:9092 logging: type: inline loggers: connect.root.logger.level: DEBUG #connect.root.logger.level: DEBUG config: producer.max.request.size: 5000000 max.request.size: 5000000 config.providers: secrets config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider group.id: connect-cluster offset.storage.topic: connect-cluster-offsets config.storage.topic: connect-cluster-configs status.storage.topic: connect-cluster-status # -1 means it will use the default replication factor configured in the broker config.storage.replication.factor: -1 offset.storage.replication.factor: -1 status.storage.replication.factor: -1 build: output: type: docker image: XXXXXXXXXX.dkr.ecr.eu-central-1.amazonaws.com/debezium-connect-sqlserver:latest pushSecret: debezium-ecr-secret plugins: - name: debezium-sqlserver-connector artifacts: - type: tgz url: https://repo1.maven.org/maven2/io/debezium/debezium-connector-sqlserver/2.4.1.Final/debezium-connector-sqlserver-2.4.1.Final-plugin.tar.gz template: pod: imagePullSecrets: - name: awsecr-cred
KafkaConnector:
apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnector metadata: name: debezium-connector-sqlserver namespace: kafka labels: strimzi.io/cluster: debezium-connect-cluster spec: class: io.debezium.connector.sqlserver.SqlServerConnector tasksMax: 1 config: log4j.rootLogger: DEBUG, stdout tasks.max: 1 database.hostname: 192.168.3.152 database.port: 1433 database.user: "sa" database.password: "Passw0rd" database.names: "debeziumdb,debeziumdb2" database.server.name: dec2000 database.encrypt: false table.include.list: "dbo.debezium_signal,dbo.randomtable,dbo.randomtable2" signal.data.collection: "debeziumdb.dbo.debezium_signal" schema.history.internal.kafka.bootstrap.servers: debezium-cluster-kafka-bootstrap:9092 schema.history.internal.kafka.topic: dbhistory.dec2000 decimal.handling.mode: string topic.prefix: dec2000
What is the captured database version and mode of depoyment?
SQL Server 2016 Enterprise on Windows as well as SQL Server 2022 on Linux via Kubernetes.
Here is a T-SQL Script to create my test environment:
use master GO --alter database debeziumdb set single_user with rollback immediate --alter database debeziumdb2 set single_user with rollback immediate DROP DATABASE IF EXISTS debeziumdb; DROP DATABASE IF EXISTS debeziumdb2; GO GO CREATE DATABASE debeziumdb2; CREATE DATABASE debeziumdb; GO USE debeziumdb; GO DROP TABLE IF EXISTS randomtable; CREATE TABLE randomtable ( ID INT IDENTITY(1,1) PRIMARY KEY, randomtext NVARCHAR(500) ); GO DROP TABLE IF EXISTS debezium_signal CREATE TABLE [dbo].[debezium_signal]( [id] [varchar](42) NOT NULL, [type] [varchar](32) NOT NULL, [data] [varchar](2048) NULL, PRIMARY KEY CLUSTERED ( [id] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] GO DECLARE @Counter INT = 1; WHILE @Counter <= 50 BEGIN DECLARE @RandomText NVARCHAR(500); SET @RandomText = ( SELECT TOP 1 LEFT(CONVERT(NVARCHAR(MAX), NEWID()), 500) FROM sys.all_objects ); INSERT INTO randomtable (randomtext) VALUES (@RandomText); SET @Counter = @Counter + 1; END; GO EXEC sys.sp_cdc_enable_db GO EXEC sys.sp_cdc_enable_table @source_schema = N'dbo', @source_name = N'randomtable', @role_name = NULL, @supports_net_changes = 0 GO EXEC sys.sp_cdc_enable_table @source_schema = N'dbo', @source_name = N'debezium_signal', @role_name = NULL, @supports_net_changes = 0 GO USE debeziumdb2; GO DROP TABLE IF EXISTS randomtable2; CREATE TABLE randomtable2 ( ID INT IDENTITY(1,1) PRIMARY KEY, randomtext NVARCHAR(500) ); GO DECLARE @Counter INT = 1; WHILE @Counter <= 50 BEGIN DECLARE @RandomText NVARCHAR(500); SET @RandomText = ( SELECT TOP 1 LEFT(CONVERT(NVARCHAR(MAX), NEWID()), 500) FROM sys.all_objects ); INSERT INTO randomtable2 (randomtext) VALUES (@RandomText); SET @Counter = @Counter + 1; END; GO GO EXEC sys.sp_cdc_enable_db GO EXEC sys.sp_cdc_enable_table @source_schema = N'dbo', @source_name = N'randomtable2', @role_name = NULL, @supports_net_changes = 0 GO
What behaviour do you expect?
The connector should register multiple signals, including "execute-snapshot":
2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'schema-changes' using class 'io.debezium.pipeline.signal.actions.SchemaChanges' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0] 2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'snapshot-window-open' using class 'io.debezium.pipeline.signal.actions.snapshotting.OpenIncrementalSnapshotWindow' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0] 2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'execute-snapshot' using class 'io.debezium.pipeline.signal.actions.snapshotting.ExecuteSnapshot' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0] 2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'snapshot-window-close' using class 'io.debezium.pipeline.signal.actions.snapshotting.CloseIncrementalSnapshotWindow' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0] 2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'log' using class 'io.debezium.pipeline.signal.actions.Log' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0] 2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'stop-snapshot' using class 'io.debezium.pipeline.signal.actions.snapshotting.StopSnapshot' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0] 2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'resume-snapshot' using class 'io.debezium.pipeline.signal.actions.snapshotting.ResumeIncrementalSnapshot' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0] 2023-12-11 08:44:02,306 DEBUG [debezium-connector-sqlserver|task-0] Registering signal 'pause-snapshot' using class 'io.debezium.pipeline.signal.actions.snapshotting.PauseIncrementalSnapshot' (io.debezium.pipeline.signal.SignalProcessor) [task-thread-debezium-connector-sqlserver-0]
The connector should create a snapshot after creating a signal:
INSERT INTO debezium_signal (id, type, data) VALUES (NEWID(), 'execute-snapshot', '{"data-collections": ["debeziumdb.dbo.randomtable"]}')
What behaviour do you see?
The log output above mentioning the registration of multiple signals is not produced. Also, after creating the "execute-snapshot" signal, the following log is produced:
2023-12-06 13:21:13,165 WARN [debezium-connector-sqlserver|task-0] Signal '0B91F382-4AE5-40EF-BF7E-94A1EDF9B54B' has been received but the type 'execute-snapshot' is not recognized (io.debezium.pipeline.signal.SignalProcessor)
However, if only one database is configured, the behavior is as expected:
apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnector metadata: name: debezium-connector-sqlserver namespace: kafka labels: strimzi.io/cluster: debezium-connect-cluster spec: class: io.debezium.connector.sqlserver.SqlServerConnector tasksMax: 1 config: log4j.rootLogger: DEBUG, stdout tasks.max: 1 database.hostname: 192.168.3.152 database.port: 1433 database.user: "sa" database.password: "Passw0rd" database.names: "debeziumdb" database.server.name: dec2000 database.encrypt: false table.include.list: "dbo.debezium_signal,dbo.randomtable" signal.data.collection: "debeziumdb.dbo.debezium_signal" schema.history.internal.kafka.bootstrap.servers: debezium-cluster-kafka-bootstrap:9092 schema.history.internal.kafka.topic: dbhistory.dec2000 decimal.handling.mode: string topic.prefix: dec2000
Do you see the same behaviour using the latest relesead Debezium version?
Yes, as well with the current 2.5.0 Beta Version.
Do you have the connector logs, ideally from start till finish?
Here are the logs with two databases configured and no signals being registered: https://gist.github.com/guyincognito-io/d28e9b47045f5aaaec87b8fbf6da908b
Here are logs with one database configured and all signals being registered, as expected:
https://gist.github.com/guyincognito-io/a60c4d458639c58ac7587aef3f0a0901
How to reproduce the issue using our tutorial deployment?
I am not sure, I can look into this if required.
- is duplicated by
-
DBZ-7583 SQL Server signal actions not recognized with more than one database
- Closed