Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-865

Error when MongoDB collection contains characters not compatible with kafka topic naming

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 0.9.0.Alpha2
    • Component/s: mongodb-connector
    • Labels:
      None
    • Environment:

      Kafka Connect Version : 1.0.1-L0

    • Steps to Reproduce:
      Hide

      1. Create a mongo collection with name "my@collection", for instance, and add a record.
      2. Start mongo source connector that reads from this mongo table

      Observed result:

      • data is not written to kafka.
      • Endless logs:

        [2018-08-17 09:20:39,685] WARN [Producer clientId=producer-46] Error while fetching metadata with correlation id 308131 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246)
        [2018-08-17 09:20:39,686] WARN [Producer clientId=producer-52] Error while fetching metadata with correlation id 307868 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246)
        [2018-08-17 09:20:39,686] WARN [Producer clientId=producer-43] Error while fetching metadata with correlation id 329832 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246)
        [2018-08-17 09:20:39,712] WARN [Producer clientId=producer-49] Error while fetching metadata with correlation id 307934 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246)
        

      Show
      1. Create a mongo collection with name "my@collection", for instance, and add a record. 2. Start mongo source connector that reads from this mongo table Observed result: data is not written to kafka. Endless logs: [2018-08-17 09:20:39,685] WARN [Producer clientId=producer-46] Error while fetching metadata with correlation id 308131 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246) [2018-08-17 09:20:39,686] WARN [Producer clientId=producer-52] Error while fetching metadata with correlation id 307868 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246) [2018-08-17 09:20:39,686] WARN [Producer clientId=producer-43] Error while fetching metadata with correlation id 329832 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246) [2018-08-17 09:20:39,712] WARN [Producer clientId=producer-49] Error while fetching metadata with correlation id 307934 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246)

      Description

      If a mongo collection contains characters that are not compatible with the Kafka topic naming conventions (it contains a character other than ASCII alphanumerics, '.', '_' and '-'), the connector will fail when trying to create and write the data down to a topic.
      Moreover, the connecter is unable to stop throwing the error, continuing retrying and ending up filling up the logs.

      Suggested fix:
      The source connector should be able to identify invalid characters for kafka topics on the mongo collection and skip them when creating the topic.

      Extra notes:
      Transforms did not seem to do as work around. Topic with original collection name is still tried to be created.

      transforms=Reroute
      transforms.Reroute.type=io.debezium.transforms.ByLogicalTableRouter
      transforms.Reroute.topic.regex=(.*)my@collection(.*)
      transforms.Reroute.topic.replacement=$1my_collection
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  gunnar.morling Gunnar Morling
                  Reporter:
                  ruileal Rui Leal
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: