Uploaded image for project: 'JBoss Enterprise Application Platform 6'
  1. JBoss Enterprise Application Platform 6
  2. JBPAPP6-1676

NullPointerException: problem invoking Commit/RollbackCommand after bringing up a failed node

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Migrated to another ITS
    • Affects Version/s: EAP 6.0.1
    • Fix Version/s: None
    • Component/s: Clustering
    • Labels:
      None
    • Docs QE Status:
      NEW

      Description

      Failover test [1] revealed these exceptions:

      10:05:17,363 WARN  [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-19,null) Problems unmarshalling remote command from byte buffer: java.io.StreamCorruptedException: Unexpected byte found when reading an object: 0
      	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:750)
      	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
      	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
      	at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119)
      	at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:162)
      	at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:66)
      	at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:395)
      	at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:224)
      	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
      	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
      	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
      	at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163)
      	at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:115)
      	at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:104)
      	at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:200)
      	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:459)
      	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:366)
      	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)
      	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:602)
      	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
      	at org.jgroups.JChannel.up(JChannel.java:716)
      	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
      	at org.jgroups.protocols.RSVP.up(RSVP.java:188)
      	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
      	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889)
      	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
      	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
      	at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:708)
      	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:566)
      	at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
      	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
      	at org.jgroups.protocols.FD.up(FD.java:273)
      	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
      	at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
      	at org.jgroups.protocols.Discovery.up(Discovery.java:359)
      	at org.jgroups.stack.Protocol.up(Protocol.java:363)
      	at org.jgroups.protocols.TP.passMessageUp(TP.java:1185)
      	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1733)
      	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1715)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
      	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
      
      10:05:18,070 INFO  [javax.enterprise.resource.webcontainer.jsf.config] (MSC service thread 1-7) Initializing Mojarra 2.1.13-jbossorg-1 20121010-0442 for context '/clusterbench'
      10:05:18,361 WARN  [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-19,null) Problems invoking command CommitCommand {gtx=null, cacheName='default-host/clusterbench'}: java.lang.NullPointerException
      	at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.markTransactionAsRemote(AbstractTransactionBoundaryCommand.java:88)
      	at org.infinispan.commands.CommandsFactoryImpl.initializeReplicableCommand(CommandsFactoryImpl.java:351)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:226)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:203)
      	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:459)
      	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:366)
      	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)
      	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:602)
      	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
      	at org.jgroups.JChannel.up(JChannel.java:716)
      	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
      	at org.jgroups.protocols.RSVP.up(RSVP.java:188)
      	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
      	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889)
      	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
      	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
      	at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:708)
      	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:566)
      	at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
      	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
      	at org.jgroups.protocols.FD.up(FD.java:273)
      	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
      	at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
      	at org.jgroups.protocols.Discovery.up(Discovery.java:359)
      	at org.jgroups.stack.Protocol.up(Protocol.java:363)
      	at org.jgroups.protocols.TP.passMessageUp(TP.java:1185)
      	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1733)
      	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1715)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
      	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
      
      ...
      several more "Problems invoking command CommitCommand" here
      ...
      
      10:05:18,422 WARN  [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-19,null) Problems invoking command RollbackCommand {gtx=null, cacheName='default-host/clusterbench'}: java.lang.NullPointerException
      	at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.markTransactionAsRemote(AbstractTransactionBoundaryCommand.java:88)
      	at org.infinispan.commands.CommandsFactoryImpl.initializeReplicableCommand(CommandsFactoryImpl.java:356)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:226)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:203)
      	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:459)
      	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:366)
      	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)
      	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:602)
      	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
      	at org.jgroups.JChannel.up(JChannel.java:716)
      	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
      	at org.jgroups.protocols.RSVP.up(RSVP.java:188)
      	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
      	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889)
      	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
      	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)
      	at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:708)
      	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:566)
      	at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
      	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
      	at org.jgroups.protocols.FD.up(FD.java:273)
      	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
      	at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
      	at org.jgroups.protocols.Discovery.up(Discovery.java:359)
      	at org.jgroups.stack.Protocol.up(Protocol.java:363)
      	at org.jgroups.protocols.TP.passMessageUp(TP.java:1185)
      	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1733)
      	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1715)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
      	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
      

      You will note that the first exception is StreamCorruptedException: Unexpected byte found when reading an object: 0, which points to JBPAPP6-1288. I'm not sure if this is a duplicate or not, but I can't find a single mention of the other exceptions in JIRA.

      This happened after last node of the cluster (perf21, see [2]) was shut down regularly and then started again, just after it joined the cluster.

      See:
      [1] https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Failover/job/eap-6x-failover-http-session-shutdown-repl-sync/77/
      [2] https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Failover/job/eap-6x-failover-http-session-shutdown-repl-sync/77/console-perf21/

      Note that this was actually 6.0.1.ER4.1, but we don't have that version number in JIRA, so I set 6.0.1.ER4 in Affects Version/s.

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                pferraro Paul Ferraro
                Reporter:
                lthon Ladislav Thon
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: