Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-5463

IllegalStateException: channel is not connected exception when shutting down and replication is in progress

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Minor Minor
    • None
    • None
    • Clustering
    • None

      Hi,

      we're seeing the following exception:

      ...
      [JBossINF] �[0m�[33m20:12:53,073 WARN  [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 79) ISPN000100: Stopping, but there are 272 local transactions and 891 remote transactions that did not finish in time.
      [JBossINF] �[0m�[0m20:12:53,080 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 79) WFLYCLINF0003: Stopped clusterbench-ee7.ear.clusterbench-ee7-web-default.war cache from web container
      [JBossINF] �[0m�[0m20:12:53,097 INFO  [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 74) ISPN000029: Passivating all entries to disk
      [JBossINF] �[0m�[0m20:12:53,101 INFO  [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 74) ISPN000030: Passivated 3 entries in 3 milliseconds
      [JBossINF] �[0m�[0m20:12:53,111 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 74) WFLYCLINF0003: Stopped repl cache from web container
      [JBossINF] �[0m�[0m20:12:53,157 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel web
      [JBossINF] �[0m�[0m20:12:53,163 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel web
      [JBossINF] �[0m�[31m20:12:53,160 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (default task-72) ISPN000073: Unexpected error while replicating: org.infinispan.commons.CacheException: java.lang.IllegalStateException: channel is not connected
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.rethrowAsCacheException(CommandAwareRpcDispatcher.java:128)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:108)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:545)
      [JBossINF] 	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:170)
      [JBossINF] 	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:212)
      [JBossINF] 	at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.releaseLocksOnFailureBeforePrepare(PessimisticLockingInterceptor.java:248)
      [JBossINF] 	at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataWriteCommand(PessimisticLockingInterceptor.java:128)
      [JBossINF] 	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitReplaceCommand(AbstractLockingInterceptor.java:66)
      [JBossINF] 	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:57)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
      [JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:53)
      [JBossINF] 	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:57)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      [JBossINF] 	at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:366)
      [JBossINF] 	at org.infinispan.interceptors.TxInterceptor.visitReplaceCommand(TxInterceptor.java:235)
      [JBossINF] 	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:57)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
      [JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:53)
      [JBossINF] 	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:57)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.handleTxWriteCommand(StateTransferInterceptor.java:304)
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:286)
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.visitReplaceCommand(StateTransferInterceptor.java:128)
      [JBossINF] 	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:57)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      [JBossINF] 	at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:191)
      [JBossINF] 	at org.infinispan.interceptors.CacheMgmtInterceptor.visitReplaceCommand(CacheMgmtInterceptor.java:182)
      [JBossINF] 	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:57)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      [JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
      [JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
      [JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:53)
      [JBossINF] 	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:57)
      [JBossINF] 	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
      [JBossINF] 	at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1605)
      [JBossINF] 	at org.infinispan.cache.impl.CacheImpl.replaceInternal(CacheImpl.java:1150)
      [JBossINF] 	at org.infinispan.cache.impl.CacheImpl.replace(CacheImpl.java:1142)
      [JBossINF] 	at org.infinispan.cache.impl.DecoratedCache.replace(DecoratedCache.java:517)
      [JBossINF] 	at org.infinispan.cache.impl.AbstractDelegatingCache.replace(AbstractDelegatingCache.java:261)
      [JBossINF] 	at org.wildfly.clustering.ee.infinispan.CacheEntryMutator.mutate(CacheEntryMutator.java:52)
      [JBossINF] 	at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionAttributes.getAttribute(CoarseSessionAttributes.java:62)
      [JBossINF] 	at org.wildfly.clustering.web.undertow.session.DistributableSession.getAttribute(DistributableSession.java:131)
      [JBossINF] 	at io.undertow.servlet.spec.HttpSessionImpl.getAttribute(HttpSessionImpl.java:123)
      [JBossINF] 	at org.jboss.weld.context.beanstore.http.AbstractSessionBeanStore.getAttribute(AbstractSessionBeanStore.java:91)
      [JBossINF] 	at org.jboss.weld.context.beanstore.AttributeBeanStore.get(AttributeBeanStore.java:138)
      [JBossINF] 	at org.jboss.weld.context.beanstore.http.AbstractSessionBeanStore.get(AbstractSessionBeanStore.java:79)
      [JBossINF] 	at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:83)
      [JBossINF] 	at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:114)
      [JBossINF] 	at org.jboss.weld.context.PassivatingContextWrapper$AbstractPassivatingContextWrapper.get(PassivatingContextWrapper.java:70)
      [JBossINF] 	at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.getIfExists(ContextualInstanceStrategy.java:90)
      [JBossINF] 	at org.jboss.weld.bean.ContextualInstanceStrategy$CachingContextualInstanceStrategy.getIfExists(ContextualInstanceStrategy.java:165)
      [JBossINF] 	at org.jboss.weld.bean.ContextualInstance.getIfExists(ContextualInstance.java:63)
      [JBossINF] 	at org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:83)
      [JBossINF] 	at org.jboss.weld.bean.proxy.ProxyMethodHandler.getInstance(ProxyMethodHandler.java:125)
      [JBossINF] 	at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_WeldClientProxy.getSerialAndIncrement(Unknown Source)
      [JBossINF] 	at org.jboss.test.clusterbench.web.ejb.LocalEjbServlet.doGet(LocalEjbServlet.java:38)
      [JBossINF] 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
      [JBossINF] 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      [JBossINF] 	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)
      [JBossINF] 	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
      [JBossINF] 	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
      [JBossINF] 	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
      [JBossINF] 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      [JBossINF] 	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
      [JBossINF] 	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
      [JBossINF] 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      [JBossINF] 	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
      [JBossINF] 	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
      [JBossINF] 	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
      [JBossINF] 	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)
      [JBossINF] 	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
      [JBossINF] 	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
      [JBossINF] 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      [JBossINF] 	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
      [JBossINF] 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      [JBossINF] 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      [JBossINF] 	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282)
      [JBossINF] 	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)
      [JBossINF] 	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)
      [JBossINF] 	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)
      [JBossINF] 	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
      [JBossINF] 	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:778)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      [JBossINF] 	at java.lang.Thread.run(Thread.java:745)
      [JBossINF] Caused by: java.lang.IllegalStateException: channel is not connected
      [JBossINF] 	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:692)
      [JBossINF] 	at org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:175)
      [JBossINF] 	at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:360)
      [JBossINF] 	at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:103)
      [JBossINF] 	at org.jgroups.blocks.Request.execute(Request.java:83)
      [JBossINF] 	at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:367)
      [JBossINF] 	at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:373)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:300)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:104)
      [JBossINF] 	... 85 more
      ...
      

      It gets thrown when the server is shutting down and it seems that some replication is still in progress. The exception makes perfect sense due to the state in which the server is, however the ERROR log level seems high. Would it make sense to lower it? (INFO || WARN || DEBUG)

            pferraro@redhat.com Paul Ferraro
            rjanik@redhat.com Richard Janik
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: