Over time I have observed the following exceptions:

2016-10-06 15:40:26.261 [level=ERROR] - java.io.IOException: Unsupported protocol version 163
org.infinispan.persistence.spi.PersistenceException: java.io.IOException: Unsupported protocol version 163
at org.infinispan.marshall.core.MarshalledEntryImpl.unmarshall(MarshalledEntryImpl.java:116) ~[org.infinispan-infinispan-core-8.2.2.Final.jar:8.2.2.Final]
at org.infinispan.marshall.core.MarshalledEntryImpl.getMetadata(MarshalledEntryImpl.java:72) ~[org.infinispan-infinispan-core-8.2.2.Final.jar:8.2.2.Final]
at org.infinispan.persistence.PersistenceUtil.loadAndCheckExpiration(PersistenceUtil.java:120) ~[org.infinispan-infinispan-core-8.2.2.Final.jar:8.2.2.Final]
at org.infinispan.persistence.PersistenceUtil.lambda$loadAndStoreInDataContainer$0(PersistenceUtil.java:98) ~[org.infinispan-infinispan-core-8.2.2.Final.jar:8.2.2.Final]
at org.infinispan.container.DefaultDataContainer.lambda$compute$3(DefaultDataContainer.java:325) ~[org.infinispan-infinispan-core-8.2.2.Final.jar:8.2.2.Final]
at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.compute(BoundedEquivalentConcurrentHashMapV8.java:3600) ~[org.infinispan-infinispan-commons-8.2.2.Final.jar:8.2.2.Final]
at org.infinispan.container.DefaultDataContainer.compute(DefaultDataContainer.java:324) ~[org.infinispan-infinispan-core-8.2.2.Final.jar:8.2.2.Final]
at org.infinispan.persistence.PersistenceUtil.loadAndStoreInDataContainer(PersistenceUtil.java:91) ~[org.infinispan-infinispan-core-8.2.2.Final.jar:8.2.2.Final]
at org.infinispan.interceptors.CacheLoaderInterceptor.loadInContext(CacheLoaderInterceptor.java:371) ~[org.infinispan-infinispan-core-8.2.2.Final.jar:8.2.2.Final]
at org.infinispan.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:366) ~[org.infinispan-infinispan-core-8.2.2.Final.jar:8.2.2.Final]
at org.infinispan.interceptors.CacheLoaderInterceptor.visitDataCommand(CacheLoaderInterceptor.java:187) ~[org.infinispan-infinispan-core-8.2.2.Final.jar:8.2.2.Final]
at org.infinispan.interceptors.CacheLoaderInterceptor.visitGetKeyValueCommand(CacheLoaderInterceptor.java:141) ~[org.infinispan-infinispan-core-8.2.2.Final.jar:8.2.2.Final]
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43) ~[org.infinispan-infinispan-core-8.2.2.Final.jar:8.2.2.Final]

This seems to happen fairly regularly on a number of different hosts.

The invalid version values seem to change.

I'm wondering if there is a race condition in how the data is stored, and in particular the metadata.