Mongo debezium connector has worked on our dev and qa env, however once we moved to perf which has Mongo Atlas, tasks are not getting created when we start the debezium connector.
FYI:
Worker IP whitelisted on Mongo Atas
Monga Atlas version 3.4, Mongo java driver 3.4
User has Atlas admin permission
Config:
{
"connector.class": "io.debezium.connector.mongodb.MongoDbConnector",
"schema.registry.url": "http://testing-schemareg01.cloud.operative.com:8081,http://testing-schemareg02.cloud.operative.com:8081",
"zookeeper.connect": "testing-zookeeper01.cloud.operative.com:2181,testing-zookeeper02.cloud.operative.com:2181,testing-zookeeper03.cloud.operative.com:2181",
"tasks.max": "1",
"name": "mongo_debezium_testing",
"bootstrap.servers": "testing-kafka01.cloud.operative.com:9091,testing-kafka02.cloud.operative.com:9091,testing-kafka03.cloud.operative.com:9091",
"mongodb.name": "perftest",
"mongodb.hosts": "testing-shard-0/testing-shard-00-00-mxmsa.mongodb.net:27017,testing-shard-0/testing-shard-00-01-mxmsa.mongodb.net:27017,testing-shard-0/testing-shard-00-02-mxmsa.mongodb.net:27017",
"mongo.ssl.enabled":"true",
"mongodb.password": "testing",
"mongodb.user": "testuser",
"mongodb.members.auto.discover": "true",
"collection.whitelist": "testdb.testcollection"
}
Below is the exception we are seeing,
2019-02-27 06:22:43 [DistributedHerder] INFO o.a.k.c.r.d.DistributedHerder - Finished starting connectors and tasks
2019-02-27 06:23:03 [Thread-14] ERROR i.d.c.mongodb.ReplicaSetDiscovery - Error while reading the 'shards' collection in the 'config' database: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=testing-shard-00-02-mxmsa.mongodb.net:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketReadException: Prematurely reached end of stream}}, {address=testing-shard-00-00-mxmsa.mongodb.net:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketReadException: Prematurely reached end of stream}}, {address=testing-shard-00-01-mxmsa.mongodb.net:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketReadException: Prematurely reached end of stream}}]
com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=testing-shard-00-02-mxmsa.mongodb.net:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketReadException: Prematurely reached end of stream}}, {address=testing-shard-00-00-mxmsa.mongodb.net:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketReadException: Prematurely reached end of stream}}, {address=testing-shard-00-01-mxmsa.mongodb.net:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketReadException: Prematurely reached end of stream}}]
at com.mongodb.connection.BaseCluster.getDescription(BaseCluster.java:167)
at com.mongodb.Mongo.getConnectedClusterDescription(Mongo.java:885)
at com.mongodb.Mongo.createClientSession(Mongo.java:877)
at com.mongodb.Mongo$3.getClientSession(Mongo.java:866)
at com.mongodb.Mongo$3.execute(Mongo.java:823)
at com.mongodb.MongoIterableImpl.execute(MongoIterableImpl.java:130)
at com.mongodb.MongoIterableImpl.iterator(MongoIterableImpl.java:77)
at com.mongodb.MappingIterable.iterator(MappingIterable.java:40)
at io.debezium.connector.mongodb.MongoUtil.contains(MongoUtil.java:181)
at io.debezium.connector.mongodb.MongoUtil.contains(MongoUtil.java:170)
at io.debezium.connector.mongodb.MongoUtil.onDatabase(MongoUtil.java:114)
at io.debezium.connector.mongodb.MongoUtil.onCollection(MongoUtil.java:129)
at io.debezium.connector.mongodb.MongoUtil.onCollectionDocuments(MongoUtil.java:148)
at io.debezium.connector.mongodb.ReplicaSetDiscovery.getReplicaSets(ReplicaSetDiscovery.java:66)
at io.debezium.connector.mongodb.ReplicaSetMonitorThread.run(ReplicaSetMonitorThread.java:62)
2019-02-27 06:23:03 [Thread-14] INFO org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out