Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-794

WARN ISPN000197: Error updating cluster member list at the boot up

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Verified (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 7.0.0.DR8, 7.0.0.ER2 (Beta), 7.0.0.ER4, 7.0.0.ER6, 7.0.0.ER7
    • Fix Version/s: 7.1.0.DR2
    • Component/s: Clustering
    • Labels:
      None
    • Target Release:
    • Steps to Reproduce:
      Hide

      1. Get two copies of EAP 7.0.0.ER6, no configuration modification needed.
      2. For both servers, copy cluterbench-ee7.ear to standalone/deployments folder.
      3. Start both servers with standalone-ha profile:

      Server 1 startup command

      		 
      ./bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node1

      Server 2 startup command

      		 
      ./bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node2 -Djboss.socket.binding.port-offset=100

      4. Check the server logs: you will get ISPN000197: Error updating cluster member list after one minute or so

      WARN  [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p14-t10) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node2

      Show
      1. Get two copies of EAP 7.0.0.ER6, no configuration modification needed. 2. For both servers, copy cluterbench-ee7.ear to standalone/deployments folder. 3. Start both servers with standalone-ha profile: Server 1 startup command ./bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node1 Server 2 startup command ./bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node2 -Djboss.socket.binding.port-offset=100 4. Check the server logs: you will get ISPN000197: Error updating cluster member list after one minute or so WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p14-t10) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node2
    • Affects:
      Release Notes
    • Release Notes Docs Status:
      Documented as Known Issue

      Description

      Update
      The description is no longer valid.
      This Jira is now allocated for WARN ISPN000197: Error updating cluster member list occurring at the boot up (#1 from this comment).
      ----------------------------------------------------
      Seen in ejb-ejbservlet and http-session scenarios intermittently (no matter what failover type or cache is used).

      When node perf18 is restarted after failover other servers log this error several times: 

      [JBossINF] [0m[31m16:11:43,595 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-107) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf18
      		 
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:752)
      		 
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$6(JGroupsTransport.java:599)
      		 
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$34/238012590.apply(Unknown Source)
      		 
      [JBossINF] 	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
      		 
      [JBossINF] 	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      		 
      [JBossINF] 	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      		 
      [JBossINF] 	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
      		 
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.RspListFuture.timeout(RspListFuture.java:40)
      		 
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$$Lambda$32/2073718099.run(Unknown Source)
      		 
      [JBossINF] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      		 
      [JBossINF] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      		 
      [JBossINF] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      		 
      [JBossINF] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      		 
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      		 
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      		 
      [JBossINF] 	at java.lang.Thread.run(Thread.java:745)

      Server log:
      http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-ejbservlet-jvmkill-repl-async/4/console-perf19/

      In this particular test run, after perf18 restarted , perf19 logged the first error in 2 seconds, perf20 in 30 seconds, perf21 in 10 seconds.

      timeline:

      perf18: [JBossINF] [0m[0m16:11:42,361 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: EAP 7.0.0.Alpha1 (WildFly Core 2.0.0.Beta1) started in 20244ms - Started 747 of 993 services (424 services are lazy, passive or on-demand)
      		 
      perf19: [JBossINF] [0m[31m16:11:43,595 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-107) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf18
      		 
      perf20: [JBossINF] [0m[31m16:12:12,836 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-51) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf18
      		 
      perf21: [JBossINF] [0m[31m16:11:52,826 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-22) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf18

      This error also intermittently appears after server is shutdown.

      Total number of errors for this particular test run: 1183

        Gliffy Diagrams

          Attachments

            Issue Links

            causes

            Bug - A problem which impairs or prevents the functions of the product. JBEAP-3663 Error updating cluster member list warning in ejb-multi-server

            • Critical - An upcoming version that is affected by this issue cannot be released until it's addressed. A critical bug is one that crashes the application, causes loss of data or severe memory leak.
            • Closed
            cloned to

            Bug - A problem which impairs or prevents the functions of the product. WFLY-5158 WARN ISPN000197: Error updating cluster member list at the boot up

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed
            is blocked by

            Task - A task that needs to be done. ISPN-6751 Backports for 8.1.5.Final

            • Blocker - An issue (bug, feature, task) that blocks development and/or testing work, production could not run. An upcoming version that is affected by this issue cannot be released until it's addressed.
            • Resolved
            is caused by

            Component Upgrade - A task tracking an update to a bundled component JBEAP-4841 (7.1.0) Upgrade Infinispan to 8.1.5.Final

            • Major - A request that should be considered seriously but is not a show stopper.
            • Closed

            Component Upgrade - A task tracking an update to a bundled component JBEAP-6048 (7.0.z) Upgrade Infinispan from 8.1.4.Final-redhat-1 to 8.1.6.Final

            • Major - A request that should be considered seriously but is not a show stopper.
            • Verified
            is duplicated by

            Bug - A problem which impairs or prevents the functions of the product. JBEAP-4054 ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: Replication timeout for X

            • Major - A request that should be considered seriously but is not a show stopper.
            • Closed
            is followed up by

            Bug - A problem which impairs or prevents the functions of the product. ISPN-6387 ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: Replication timeout for X

            • Major - A request that should be considered seriously but is not a show stopper.
            • New
            is related to

            Bug - A problem which impairs or prevents the functions of the product. JBEAP-2481 org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology after failover

            • Blocker - An issue (bug, feature, task) that blocks development and/or testing work, production could not run. An upcoming version that is affected by this issue cannot be released until it's addressed.
            • Closed
            relates to

            Bug - A problem which impairs or prevents the functions of the product. JBEAP-2263 Invalidating a session of an SSO on a different node than where the session was created does not logout the user

            • Blocker - An issue (bug, feature, task) that blocks development and/or testing work, production could not run. An upcoming version that is affected by this issue cannot be released until it's addressed.
            • Verified

              Activity

                People

                • Assignee:
                  rhusar Radoslav Husar
                  Reporter:
                  mvinkler Michal Vinkler
                • Votes:
                  2 Vote for this issue
                  Watchers:
                  16 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: