Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-15887

Rolling upgrade to 14.0.26 not possible

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • 14.0.26.Final
    • None
    • None
      • Have an application use embedded Infinispan on version 14.0.19
      • Update the application to use embedded Infinispan on version 14.0.26
      • Trigger a rolling deployment

      We are using embedded Infinispan in an application that runs on a standard Kubernetes deployment.

      When we updated from 14.0.19 to 14.0.26, we observed that the new pod would not successfully start with this exception:

      java.io.IOException: Unknown type: 28
      	at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:720)
      	at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357)
      	at org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32)
      	at org.infinispan.topology.CacheTopology$Externalizer.doReadObject(CacheTopology.java:269)
      	at org.infinispan.topology.CacheTopology$Externalizer.doReadObject(CacheTopology.java:250)
      	at org.infinispan.commons.marshall.InstanceReusingAdvancedExternalizer.readObject(InstanceReusingAdvancedExternalizer.java:102)
      	at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727)
      	at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708)
      	at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357)
      	at org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32)
      	at org.infinispan.topology.CacheStatusResponse$Externalizer.readObject(CacheStatusResponse.java:98)
      	at org.infinispan.topology.CacheStatusResponse$Externalizer.readObject(CacheStatusResponse.java:85)
      	at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727)
      	at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708)
      	at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357)
      	at org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32)
      	at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.readObject(SuccessfulResponse.java:71)
      	at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.readObject(SuccessfulResponse.java:63)
      	at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727)
      	at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708)
      	at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357)
      	at org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:191)
      	at org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:220)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1562)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1471)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1663)
      	at org.jgroups.JChannel.up(JChannel.java:734)
      	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:936)
      	at org.jgroups.protocols.FRAG2.up(FRAG2.java:139)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:245)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:245)
      	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:846)
      	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:226)
      	at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1086)
      	at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:825)
      	at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:807)
      	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:456)
      	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:680)
      	at org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:105)
      	at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:180)
      	at org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:189)
      	at org.jgroups.protocols.MERGE3.up(MERGE3.java:274)
      	at org.jgroups.protocols.Discovery.up(Discovery.java:294)
      	at org.jgroups.stack.Protocol.up(Protocol.java:340)
      	at org.jgroups.protocols.TP.passMessageUp(TP.java:1184)
      	at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:107)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      	at java.base/java.lang.Thread.run(Thread.java:833) 

      After some investigation on our end, I think the reason for this behaviour is the change to the CacheTopology wire format in commit e30f657cdd1f1a684d3b01cd57779728e7939d41, resulting in the new pod not being able to read messages produced by the old version.

            Unassigned Unassigned
            tobias.bahls@getflip.com Tobias Bahls
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: