Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
1.0.0.Beta2
-
None
Description
Hi,
when I use avro consumer to look to the content of a topic I can see huge number of message with the same info:
null {"before":null,"after":{"zza_tst_01.zza.lccs_users.Value":{"lcc_id":{"int":106},"user_id":{"int":5674}}},"source":{"version":"0.10.0.Beta2","connector":"mysql","name":"zza-tst-01","ts_ms":0,"snapshot":{"string":"true"},"db":"zza","table":{"string":"lccs_users"},"server_id":0,"gtid":null,"file":"mysql-bin.003575","pos":49315918,"row":0,"thread":null,"query":null},"op":"r","ts_ms":{"long":1571753854384}}
null {"before":null,"after":{"zza_tst_01.zza.lccs_users.Value":{"lcc_id":{"int":106},"user_id":{"int":5674}}},"source":{"version":"0.10.0.Beta2","connector":"mysql","name":"zza-tst-01","ts_ms":0,"snapshot":{"string":"true"},"db":"zza","table":{"string":"lccs_users"},"server_id":0,"gtid":null,"file":"mysql-bin.003575","pos":49315918,"row":0,"thread":null,"query":null},"op":"r","ts_ms":{"long":1571753854384}}
null {"before":null,"after":{"zza_tst_01.zza.lccs_users.Value":{"lcc_id":{"int":106},"user_id":{"int":5674}}},"source":{"version":"0.10.0.Beta2","connector":"mysql","name":"zza-tst-01","ts_ms":0,"snapshot":{"string":"true"},"db":"zza","table":{"string":"lccs_users"},"server_id":0,"gtid":null,"file":"mysql-bin.003575","pos":49315918,"row":0,"thread":null,"query":null},"op":"r","ts_ms":{"long":1571753854384}}
null {"before":null,"after":{"zza_tst_01.zza.lccs_users.Value":{"lcc_id":{"int":106},"user_id":{"int":5674}}},"source":{"version":"0.10.0.Beta2","connector":"mysql","name":"zza-tst-01","ts_ms":0,"snapshot":{"string":"true"},"db":"zza","table":{"string":"lccs_users"},"server_id":0,"gtid":null,"file":"mysql-bin.003575","pos":49315918,"row":0,"thread":null,"query":null},"op":"r","ts_ms":{"long":1571753854384}}
null {"before":null,"after":{"zza_tst_01.zza.lccs_users.Value":{"lcc_id":{"int":106},"user_id":{"int":5674}}},"source":{"version":"0.10.0.Beta2","connector":"mysql","name":"zza-tst-01","ts_ms":0,"snapshot":{"string":"true"},"db":"zza","table":{"string":"lccs_users"},"server_id":0,"gtid":null,"file":"mysql-bin.003575","pos":49315918,"row":0,"thread":null,"query":null},"op":"r","ts_ms":{"long":1571753854384}}
null {"before":null,"after":{"zza_tst_01.zza.lccs_users.Value":{"lcc_id":{"int":106},"user_id":{"int":5674}}},"source":{"version":"0.10.0.Beta2","connector":"mysql","name":"zza-tst-01","ts_ms":0,"snapshot":{"string":"true"},"db":"zza","table":{"string":"lccs_users"},"server_id":0,"gtid":null,"file":"mysql-bin.003575","pos":49315918,"row":0,"thread":null,"query":null},"op":"r","ts_ms":{"long":1571753854384}}
null {"before":null,"after":{"zza_tst_01.zza.lccs_users.Value":{"lcc_id":{"int":106},"user_id":{"int":5674}}},"source":{"version":"0.10.0.Beta2","connector":"mysql","name":"zza-tst-01","ts_ms":0,"snapshot":{"string":"true"},"db":"zza","table":{"string":"lccs_users"},"server_id":0,"gtid":null,"file":"mysql-bin.003575","pos":49315918,"row":0,"thread":null,"query":null},"op":"r","ts_ms":{"long":1571753854384}}
null {"before":null,"after":{"zza_tst_01.zza.lccs_users.Value":{"lcc_id":{"int":106},"user_id":{"int":5674}}},"source":{"version":"0.10.0.Beta2","connector":"mysql","name":"zza-tst-01","ts_ms":0,"snapshot":{"string":"true"},"db":"zza","table":{"string":"lccs_users"},"server_id":0,"gtid":null,"file":"mysql-bin.003575","pos":49315918,"row":0,"thread":null,"query":null},"op":"r","ts_ms":{"long":1571753854384}}
null {"before":null,"after":{"zza_tst_01.zza.lccs_users.Value":{"lcc_id":{"int":106},"user_id":{"int":5674}}},"source":{"version":"0.10.0.Beta2","connector":"mysql","name":"zza-tst-01","ts_ms":0,"snapshot":{"string":"true"},"db":"zza","table":{"string":"lccs_users"},"server_id":0,"gtid":null,"file":"mysql-bin.003575","pos":49315918,"row":0,"thread":null,"query":null},"op":"r","ts_ms":{"long":1571753854384}}
null {"before":null,"after":{"zza_tst_01.zza.lccs_users.Value":{"lcc_id":{"int":106},"user_id":{"int":5674}}},"source":{"version":"0.10.0.Beta2","connector":"mysql","name":"zza-tst-01","ts_ms":0,"snapshot":{"string":"true"},"db":"zza","table":{"string":"lccs_users"},"server_id":0,"gtid":null,"file":"mysql-bin.003575","pos":49315918,"row":0,"thread":null,"query":null},"op":"r","ts_ms":{"long":1571753854384}}
This table: lccs_users was added after the connector was created into the connector config parameter: table.whitelist.
We have use this parameter: "snapshot.new.tables": "parallel" to tell Debezium to take into account this new listed table.
If I counf the record on the MySQL source table:
+----------+ | count(*) | +----------+ | 1.526.914 | +----------+
and number of message in topics : 12089131
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID zza-tst-01.zza.lccs_users 0 12089131 12089131 0 - - -
Questions:
- is it normal to have all these message with same record with no difference (ops=r)?
- is it normal to have so much messages in topics regarding the soruce of MySQL table ?
Regards,
Eric
