Uploaded image for project: 'JBoss Enterprise Application Platform 4 and 5'
  1. JBoss Enterprise Application Platform 4 and 5
  2. JBPAPP-9952

JGroups: Concurrency issues during soak test

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • EAP_EWP 5.2.0 ER4
    • EAP_EWP 5.2.0 ER2
    • Clustering
    • None
    • RHEL6 x64_84, IBM JDK 1.7

    • Not Required
    • NEW

    Description

      During our soak test run in the mentioned environment, we observed the following exceptions/error. We saw a handful of them during the whole test.

      The same soak scenario was run on the same servers with Oracle JDK1.6 and we did not see these issues.

      [JBoss] 08:05:32,377 ERROR [org.jgroups] (Incoming-152,10.16.88.182:55200) uncaught exception in Thread[Incoming-152,10.16.88.182:55200,10,Thread Pools] (thread group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )
      [JBoss] java.util.ConcurrentModificationException
      [JBoss] 	at java.util.Hashtable$HashIterator.next(Hashtable.java:197)
      [JBoss] 	at java.util.Hashtable$HashEnumIterator.next(Hashtable.java:711)
      [JBoss] 	at org.jgroups.protocols.VERIFY_SUSPECT.adjustSuspectedMembers(VERIFY_SUSPECT.java:300)
      [JBoss] 	at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:99)
      [JBoss] 	at org.jgroups.protocols.BARRIER.down(BARRIER.java:107)
      [JBoss] 	at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:675)
      [JBoss] 	at org.jgroups.protocols.UNICAST.down(UNICAST.java:455)
      [JBoss] 	at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:299)
      [JBoss] 	at org.jgroups.protocols.VIEW_SYNC.down(VIEW_SYNC.java:204)
      [JBoss] 	at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:542)
      [JBoss] 	at org.jgroups.protocols.pbcast.CoordGmsImpl.handleViewChange(CoordGmsImpl.java:463)
      [JBoss] 	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:749)
      [JBoss] 	at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:192)
      [JBoss] 	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
      [JBoss] 	at org.jgroups.protocols.UNICAST.up(UNICAST.java:328)
      [JBoss] 	at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:895)
      [JBoss] 	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:708)
      [JBoss] 	at org.jgroups.protocols.BARRIER.up(BARRIER.java:136)
      [JBoss] 	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:174)
      [JBoss] 	at org.jgroups.protocols.FD.up(FD.java:284)
      [JBoss] 	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:328)
      [JBoss] 	at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
      [JBoss] 	at org.jgroups.protocols.Discovery.up(Discovery.java:264)
      [JBoss] 	at org.jgroups.protocols.PING.up(PING.java:273)
      [JBoss] 	at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319)
      [JBoss] 	at org.jgroups.protocols.TP.passMessageUp(TP.java:1253)
      [JBoss] 	at org.jgroups.protocols.TP.access$100(TP.java:50)
      [JBoss] 	at org.jgroups.protocols.TP$1.run(TP.java:1177)
      [JBoss] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1121)
      [JBoss] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
      [JBoss] 	at java.lang.Thread.run(Thread.java:769)
      
      [JBoss] SuspectedException
      [JBoss] 	at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:603)
      [JBoss] 	at org.jgroups.blocks.RpcDispatcher.callRemoteMethod(RpcDispatcher.java:323)
      [JBoss] 	at org.jgroups.blocks.RpcDispatcher.callRemoteMethod(RpcDispatcher.java:304)
      [JBoss] 	at org.jboss.ha.framework.server.ClusterPartition.callMethodOnNode(ClusterPartition.java:1201)
      [JBoss] 	at org.jboss.ha.timestamp.TimestampDiscrepancyService.statusCheck(TimestampDiscrepancyService.java:552)
      [JBoss] 	at org.jboss.ha.timestamp.TimestampDiscrepancyService.access$200(TimestampDiscrepancyService.java:60)
      [JBoss] 	at org.jboss.ha.timestamp.TimestampDiscrepancyService$1.run(TimestampDiscrepancyService.java:727)
      [JBoss] 	at java.lang.Thread.run(Thread.java:769)
      [JBoss] 08:06:03,316 INFO  [org.jboss.test.cluster.servlet.SessionTestServlet] (ajp-10.16.88.182-8009-5616) Setting up new session - iCEB2CcXgui6G0stoz6+vw__.perf04.
      [JBoss] 08:06:02,553 WARN  [org.jgroups.protocols.pbcast.NAKACK] (Incoming-149,10.16.88.182:55200) 10.16.88.182:55200] discarded message from non-member 10.16.88.186:55200, my view is [10.16.88.182:55200|7] [10.16.88.182:55200]
      [JBoss] 08:06:03,628 INFO  [org.jboss.test.cluster.servlet.SessionTestServlet] (ajp-10.16.88.182-8009-8949) Setting up new session - OJB2uLcf37VzttAZAoL3sA__.perf04.
      [JBoss] 08:06:03,316 INFO  [org.jboss.test.cluster.servlet.SessionTestServlet] (ajp-10.16.88.182-8009-8618) Setting up new session - GkMyPWRrrFAzlN+m5NNPWg__.perf04.
      [JBoss] 08:06:02,670 WARN  [org.jgroups.protocols.FD] (OOB-3649,10.16.88.182:55200) I was suspected by 10.16.88.185:55200; ignoring the SUSPECT message and sending back a HEARTBEAT_ACK
      [JBoss] 08:06:04,065 INFO  [org.jboss.ha.framework.interfaces.HAPartition.lifecycle.HttpReplTestPartition] (VERIFY_SUSPECT.TimerThread,HttpReplTestPartition,10.16.88.182:55200) Suspected member: 10.16.88.184:55200
      [JBoss] 08:06:06,212 INFO  [org.jboss.test.cluster.servlet.SessionTestServlet] (ajp-10.16.88.182-8009-5865) Setting up new session - FcI-pmZGjogDAfhpgyQCcw__.perf04.
      [JBoss] 08:06:07,312 ERROR [org.jboss.ha.timestamp.TimestampDiscrepancyService] (TimestampDiscrepancyService-AsyncStatusCheck) Caught exception requesting timestamp from node 10.16.88.186:1099
      [JBoss] SuspectedException
      [JBoss] 	at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:603)
      [JBoss] 	at org.jgroups.blocks.RpcDispatcher.callRemoteMethod(RpcDispatcher.java:323)
      [JBoss] 	at org.jgroups.blocks.RpcDispatcher.callRemoteMethod(RpcDispatcher.java:304)
      [JBoss] 	at org.jboss.ha.framework.server.ClusterPartition.callMethodOnNode(ClusterPartition.java:1201)
      [JBoss] 	at org.jboss.ha.timestamp.TimestampDiscrepancyService.statusCheck(TimestampDiscrepancyService.java:552)
      [JBoss] 	at org.jboss.ha.timestamp.TimestampDiscrepancyService.access$200(TimestampDiscrepancyService.java:60)
      [JBoss] 	at org.jboss.ha.timestamp.TimestampDiscrepancyService$1.run(TimestampDiscrepancyService.java:727)
      [JBoss] 	at java.lang.Thread.run(Thread.java:769)
      

      Attachments

        Issue Links

          Activity

            People

              navssurtani Navin Surtani (Inactive)
              jkudrnac@redhat.com Jitka Kozana
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: