Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-26757

[CLUSTERING] VERIFY_SUSPECT2 error when shutting down

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • None
    • 8.0.0.GA-CR3
    • Clustering, EJB
    • None
    • False
    • None
    • False

    Description

      In a 4 nodes cluster configured as in the following:

      embed-server --server-config=standalone-ha.xml
      if (outcome != success) of /subsystem=jgroups:read-attribute(name=default-stack)
      /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
      else
      /subsystem=jgroups:write-attribute(name=default-stack,value=tcp)
      /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
      end-if
      

      we deploy and EJB application and invoke it via "remote+http" e.g. "remote+http://10.0.201.46:8080,remote+http://10.0.201.126:8080,remote+http://10.0.201.250:8080,remote+http://10.0.201.92:8080"

      the application is then un-deployed & re-deployed on every node in sequence;

      when, at the end of the test, the nodes are shut down, we observed the following error on node 3:

      2024-03-02 01:42:50,584 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 121) WFLYCLJG0035: Disconnected 'ee' channel. 'wildfly3' left cluster 'ejb'
      2024-03-02 01:42:50,587 ERROR [org.jgroups.protocols.TCP] (thread-52,ejb,wildfly3) failed processing batch: java.util.concurrent.RejectedExecutionException: Task org.jgroups.protocols.VERIFY_SUSPECT2@70f7453b rejected from java.util.concurrent.ThreadPoolExecutor@67c6af13[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
      	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
      	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.VERIFY_SUSPECT2.startTask(VERIFY_SUSPECT2.java:240)
      	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.VERIFY_SUSPECT2.verifySuspect(VERIFY_SUSPECT2.java:195)
      	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:96)
      	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.stack.Protocol.up(Protocol.java:330)
      	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.suspect(FD_SOCK2.java:486)
      	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.handle(FD_SOCK2.java:302)
      	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:198)
      	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.MERGE3.up(MERGE3.java:288)
      	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.Discovery.up(Discovery.java:314)
      	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.RED.up(RED.java:119)
      	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.TP.passBatchUp(TP.java:1210)
      	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:287)
      	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:157)
      	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:276)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      	at org.wildfly.clustering.context@8.0.0.GA-redhat-00011//org.wildfly.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
      	at org.wildfly.clustering.context@8.0.0.GA-redhat-00011//org.wildfly.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:78)
      	at java.base/java.lang.Thread.run(Thread.java:833)
      
      2024-03-02 01:42:50,594 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: JBoss EAP 8.0.0.GA (WildFly Core 21.0.5.Final-redhat-00001) stopped in 232ms
      

      Complete logs in eap-logs.zip

      Attachments

        Activity

          People

            Unassigned Unassigned
            tborgato@redhat.com Tommaso Borgato
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: