Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1065

Connector consumes huge amount of memory

    XMLWordPrintable

    Details

      Description

      When SQL Server connector is run on a "hot" table it consumes huge amount of memory. By hot I mean 2-3k events per second.

      I've created a heap dump and analyzed it using Eclipse MAT. It suggests that vast majority of memory is consumed by TDSPacket objects:

      One instance of "com.microsoft.sqlserver.jdbc.TDSPacket" loaded by "org.apache.kafka.connect.runtime.isolation.PluginClassLoader @ 0x8199da18" occupies 25,106,882,424 (98.90%) bytes.

      Some background:
      The connector was down for a few hours. When I started it, the amount of memory used was increasing quickly until it reached Xmx limit. Then time spend on GC started increasing up to a point when KConnect stopped being responsive and finally died because of OutOfMemoryError. When I increased Xmx from 24GB to 30GB and started KConnect again, memory usage quickly reached 26GB. After 20 minutes I have restarted KConnect and again memory usage quickly reached 26GB again (see attachments).

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                grzegorzk Grzegorz Kołakowski
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: