Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-502

Add example for sending CDC events to Elasticsearch

    XMLWordPrintable

Details

    • Task
    • Resolution: Done
    • Major
    • 0.7.2
    • None
    • examples
    • None

    Description

      The unwrap-smt example should be expanded to cover Elasticsearch as a sink, too. Also a blog post could fall out of this.

      These are the JARs needed for the sink connector in the local Docker ifle:

      # Deploy Confluent Elasticsearch sink connector
      ENV KAFKA_CONNECT_ES_DIR=$KAFKA_CONNECT_PLUGINS_DIR/kafka-connect-elasticsearch
      RUN mkdir $KAFKA_CONNECT_ES_DIR && cd $KAFKA_CONNECT_ES_DIR &&\
              curl -sO http://packages.confluent.io/maven/io/confluent/kafka-connect-jdbc/3.3.0/kafka-connect-jdbc-3.3.0.jar &&\
              curl -sO http://packages.confluent.io/maven/io/confluent/kafka-connect-elasticsearch/3.3.0/kafka-connect-elasticsearch-3.3.0.jar && \
              curl -sO http://central.maven.org/maven2/io/searchbox/jest/2.0.0/jest-2.0.0.jar && \
              curl -sO http://central.maven.org/maven2/org/apache/httpcomponents/httpcore-nio/4.4.4/httpcore-nio-4.4.4.jar && \
              curl -sO http://central.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5.1/httpclient-4.5.1.jar && \
              curl -sO http://central.maven.org/maven2/org/apache/httpcomponents/httpasyncclient/4.1.1/httpasyncclient-4.1.1.jar && \
              curl -sO http://central.maven.org/maven2/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.jar && \
              curl -sO http://central.maven.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar && \
              curl -sO http://central.maven.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar && \
              curl -sO http://central.maven.org/maven2/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.jar && \
              curl -sO http://central.maven.org/maven2/io/searchbox/jest-common/2.0.0/jest-common-2.0.0.jar && \
              curl -sO http://central.maven.org/maven2/com/google/code/gson/gson/2.4/gson-2.4.jar
      

      And that's the connector config:

      {
          "name": "es-sink",
          "config": {
              "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
              "tasks.max": "1",
              "topics": "customers",
              "topic.index.map" : "logs:logs_index",
              "connection.url" : "http://localhost:9200",
              "type.name" : "log",
              "key.ignore" : "true",
              "schema.ignore" : "true",
              "transforms": "unwrap",
              "transforms.unwrap.type": "io.debezium.transforms.UnwrapFromEnvelope"
          }
      }
      

      Attachments

        Activity

          People

            jpechane Jiri Pechanec
            gunnar.morling Gunnar Morling
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: