In my project, I'm using the Kryo serialization library to serialize/deserialize objects to/from byte[]. Some of these byte arrays appear to be getting corrupted during transmission.

The problem happens very infrequently and only under load. I have to send tens of thousands of messages from hundreds of threads just to get two or three errors. The exception occurs when Kryo tries to deserialize the byte[] on the receiving node. So far, all of the errors I've seen happen when Kryo tries to find a class to deserialize an object, and the stack traces end with something like:

Caused by: java.lang.ClassNotFoundException: java.util.Date[SOH][GS]

Note that the [SOH] and [GS] are not literally those strings, they're unprintable ASCII 0x1 and 0x1D.

I'm confident that Kryo itself is not the source of the problem. I added some code to my project to deserialize the message on the sending node, immediately after serializing it, and log any exception thrown from the deserialization. The sending node did not log an exception, but the receiving node still did. My code passes the byte[] to the Message constructor without doing anything to it.

It also appears that I am not the first person to encounter this issue. See https://code.google.com/p/kryo/issues/detail?id=102