Connector consumes huge amount of memory

When SQL Server connector is run on a "hot" table it consumes huge amount of memory. By hot I mean 2-3k events per second.

I've created a heap dump and analyzed it using Eclipse MAT. It suggests that vast majority of memory is consumed by TDSPacket objects:

One instance of "com.microsoft.sqlserver.jdbc.TDSPacket" loaded by "org.apache.kafka.connect.runtime.isolation.PluginClassLoader @ 0x8199da18" occupies 25,106,882,424 (98.90%) bytes.

Some background:
The connector was down for a few hours. When I started it, the amount of memory used was increasing quickly until it reached Xmx limit. Then time spend on GC started increasing up to a point when KConnect stopped being responsive and finally died because of OutOfMemoryError. When I increased Xmx from 24GB to 30GB and started KConnect again, memory usage quickly reached 26GB. After 20 minutes I have restarted KConnect and again memory usage quickly reached 26GB again (see attachments).