Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-9351

nodes did not join to cluster because of timeoutException

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Major Major
    • None
    • None
    • None
    • None
    • Hide

      join 15 nodes in parallel to 1 cluster

      AR nodes did not join because of various exceptions with cause of timeout
      ER all 15 nodes join to 1 cluster

      Show
      join 15 nodes in parallel to 1 cluster AR nodes did not join because of various exceptions with cause of timeout ER all 15 nodes join to 1 cluster

      I was trying to connect 15nodes to 1 cluster, however nodes did not join.
      In logs, all nodes mostly had 2 kinds of exceptions, all caused by TimeoutException:

      1:

      2018-07-03 07:35:31,670 DEBUG  [Camel (camel-1) thread #0 - seda://systemInitializer] (LocalTopologyManagerImpl.java:169) - Error sending join request for cache org.infinispan.CONFIG to coordinator
org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 2 from 0125f36a-2860-4107-810a-d087224c9105-21637

following by next exception 1 second after...
2018-07-03 07:35:31,968  INFO  [Camel (camel-1) thread #0 - seda://systemInitializer] (JGroupsTransport.java:702) - ISPN000080: Disconnecting JGroups channel cloud11-15
2018-07-03 07:35:32,262 DEBUG  [Camel (camel-1) thread #0 - seda://systemInitializer] (DefaultCacheManager.java:709) - Stopping cache manager cloud11-15 on null
2018-07-03 07:35:32,273  WARN  [Camel (camel-1) thread #0 - seda://systemInitializer] (DefaultCacheManager.java:736) - ISPN000189: While stopping a cache or cache manager, one of its components failed to stop
java.util.concurrent.CompletionException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
	at java.util.concurrent.CompletableFuture.reportJoin(Unknown Source) ~[?:1.8.0_131]
	at java.util.concurrent.CompletableFuture.join(Unknown Source) ~[?:1.8.0_131]
	at org.infinispan.manager.DefaultCacheManager.terminate(DefaultCacheManager.java:688) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.stopCaches(DefaultCacheManager.java:734) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:711) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 2 from 0125f36a-2860-4107-810a-d087224c9105-21637

      2:

      2018-07-03 09:12:55,788  WARN  [Camel (camel-1) thread #2 - seda://northboundProvider] (TransactionImpl.java:429) - ISPN000927: exception while committing
javax.transaction.xa.XAException: null
	at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:180) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.transaction.xa.XaTransactionTable.rollback(XaTransactionTable.java:137) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.transaction.xa.TransactionXaAdapter.rollback(TransactionXaAdapter.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:424) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.tx.TransactionImpl.rollbackResources(TransactionImpl.java:477) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:332) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:132) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.tx.TransactionManagerImpl.rollback(TransactionManagerImpl.java:80) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.CacheImpl.tryRollback(CacheImpl.java:1801) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.CacheImpl.executeCommandWithInjectedTx(CacheImpl.java:1731) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1707) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1370) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:655) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:544) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:674) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
.....
	Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 1043 from 0075f36a-2860-4107-810a-d087224c9105-32070

      in attached zip including
      -infinispan logs from all nodes
      -cluster config
      -jgroups health status csv files from nodes(comma separated, time in csv is 2hours before time in logs)

        1. 15nodesTryingToJoin1ClusterHowever5DidNotJoin .zip
          5.68 MB
        2. Rebooting14ControllrsOf15InCloud.zip
          5.37 MB

            Unassigned Unassigned
            robertcernak Robert Cernak (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: