Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-2043

Node fails to join already formed replication mode cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • None
    • 5.1.4.FINAL
    • State Transfer
    • None

    Description

      During elasticity test with this timeline:
      start node1, REPLICATED clustering mode
      wait for start
      load data
      start node2
      wait for cluster to form
      start node3
      wait for cluster to form
      ...

      node3 fails to join the cluster with the following stack trace:
      node03: 06:58:06,731 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-3,node03/default) ISPN000172: Failed to prepare view CacheView

      {viewId=1, members=[node03/default]} for cache ___hotRodTopologyCache, rolling back to view CacheView{viewId=-1, members=[]}: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Cannot prepare new view CacheView{viewId=1, members=[node03/default]}

      on cache ___hotRodTopologyCache, we are currently preparing view CacheView

      {viewId=3, members=[node01/default, node02/default, node03/default]}
      node03: at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232) [rt.jar:1.6.0_30]
      node03: at java.util.concurrent.FutureTask.get(FutureTask.java:91) [rt.jar:1.6.0_30]
      node03: at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:321) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1]
      node03: at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:249) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1]
      node03: at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:875) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1]
      node03: at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_30]
      node03: at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_30]
      node03: at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
      node03: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
      node03: at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
      node03: Caused by: java.lang.IllegalStateException: Cannot prepare new view CacheView{viewId=1, members=[node03/default]} on cache ___hotRodTopologyCache, we are currently preparing view CacheView{viewId=3, members=[node01/default, node02/default, node03/default]}

      node03: at org.infinispan.cacheviews.CacheViewInfo.prepareView(CacheViewInfo.java:102) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1]
      node03: at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:481) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1]
      node03: at org.infinispan.cacheviews.CacheViewsManagerImpl$3.call(CacheViewsManagerImpl.java:312) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1]
      node03: at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_30]
      node03: at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_30]
      node03: at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
      node03: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
      node03: at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
      node03: at org.jboss.threads.JBossThread.run(JBossThread.java:122)
      node03:
      node03: 06:58:07,728 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-3,node03/default) ISPN000172: Failed to prepare view CacheView

      {viewId=3, members=[node03/default]}

      for cache ___hotRodTopologyCache, rolling back to view CacheView

      {viewId=2, members=[]}

      : java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Trying to block write commands but they are already blocked for view 3
      node03: at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232) [rt.jar:1.6.0_30]
      node03: at java.util.concurrent.FutureTask.get(FutureTask.java:91) [rt.jar:1.6.0_30]
      node03: at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:321) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1]
      node03: at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:249) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1]
      node03: at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:875) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1]
      node03: at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_30]
      node03: at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_30]
      node03: at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
      node03: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
      node03: at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
      node03: Caused by: java.lang.IllegalStateException: Trying to block write commands but they are already blocked for view 3
      node03: at org.infinispan.statetransfer.StateTransferLockImpl.blockNewTransactions(StateTransferLockImpl.java:233) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1]
      node03: at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:84) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1]
      node03: at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1]
      node03: at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:331) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1]
      node03: at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:485) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1]
      node03: at org.infinispan.cacheviews.CacheViewsManagerImpl$3.call(CacheViewsManagerImpl.java:312) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1]
      node03: at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_30]
      node03: at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_30]
      node03: at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
      node03: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
      node03: at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
      node03: at org.jboss.threads.JBossThread.run(JBossThread.java:122)
      node03:
      node03: 07:19:37,679 ERROR [org.infinispan.cacheviews.CacheViewInfo] (undefined) ISPN000164: Rolling back to cache view 2, but last committed view is 1
      node03: 07:19:37,687 ERROR [org.infinispan.cacheviews.CacheViewInfo] (undefined) ISPN000164: Rolling back to cache view 5, but last committed view is 1
      node03: 07:19:37,689 ERROR [org.infinispan.cacheviews.CacheViewInfo] (undefined) ISPN000164: Rolling back to cache view 6, but last committed view is 1
      node03: 07:19:47,684 ERROR [org.infinispan.cacheviews.CacheViewInfo] (undefined) ISPN000164: Rolling back to cache view 3, but last committed view is 5

      Attachments

        Activity

          People

            onevelik Ondrej Nevelik (Inactive)
            onevelik Ondrej Nevelik (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: