Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-14463

EAP 7.2 clustering problems/errors/warnings on OpenShift

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • None
    • 7.3.0.CD16, 7.2.2.GA
    • Clustering
    • None

    Description

      I've run our OpenShift "chart tests". These tests form some cluster with given number of replicas (using dns-ping protocol). This cluster is then "stressed" with 256 client threads and we check the responses. There are many scenarios with operations like scale down, scale up, docker kill, etc. performed during the test.

      I can see quite a lot of "HTTP 500" responses in some of these scenarios. This isn't really happening with current EAP 7.1.1 image.

      For example scenario doing scale up from two nodes to six nodes (and there are other problematic for sure) there was 363 responses with 500 response code. I can see a lot of warnings from infinispan and jgroups.... and also timeouts. Timeout usually manifests as follows (I think it's the case when response took longer than 30 s) 

      �[0m�[31m07:32:56,351 ERROR [io.undertow.request] (default task-65) UT005023: Exception handling request to /Counter: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 376 from hsc-1-vq55d
        at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
        at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:500)
        at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:495)
        at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
        at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:637)
        at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:74)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:64)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:36)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:60)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:38)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:191)
        at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:176)
        at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
        at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
        at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:69)
        at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
        at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
        at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
        at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
        at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
        at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
        at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
        at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 376 from hsc-1-vq55d
        at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163)
        at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
        at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more

      and it is likely related with similar:

      �[0m�[31m07:34:26,860 ERROR [io.undertow.request] (default task-8) UT005023: Exception handling request to /Counter: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key SessionCreationMetaDataKey(_3LNzDdxMSsiqLKqUeVcA6B_Wdyo40WC8aystGYR) and requestor GlobalTx:hsc-1-5sd1p:114. Lock is held by GlobalTx:hsc-1-5sd1p:111
        at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
        at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:500)
        at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:495)
        at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
        at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:637)
        at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:74)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:64)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:36)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:60)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:38)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:191)
        at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:176)
        at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
        at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
        at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:69)
        at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
        at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
        at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
        at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
        at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
        at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
        at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
        at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key SessionCreationMetaDataKey(_3LNzDdxMSsiqLKqUeVcA6B_Wdyo40WC8aystGYR) and requestor GlobalTx:hsc-1-5sd1p:114. Lock is held by GlobalTx:hsc-1-5sd1p:111
        at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:253)
        at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:269)
        at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:161)
        at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:83)
        at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLock(PessimisticLockingInterceptor.java:93)
        at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:70)
        at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:110)
        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58)
        at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:374)
        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58)
        at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:189)
        at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommand(BaseStateTransferInterceptor.java:191)
        at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.visitGetKeyValueCommand(BaseStateTransferInterceptor.java:174)
        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:127)
        at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:96)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:60)
        at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
        at org.infinispan.interceptors.DDAsyncInterceptor.visitGetKeyValueCommand(DDAsyncInterceptor.java:106)
        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38)
        at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50)
        at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248)
        ... 43 more

      Some weird (and also suspicious) warnings are:

      �[0m�[33m07:34:23,043 WARN  [org.wildfly.clustering.web.undertow] (default task-1) javax.transaction.HeuristicMixedException: org.infinispan.commons.CacheException: javax.transaction.HeuristicMixedException
        at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:102)
        at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91)
        at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
        at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:577)
        at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339)
        at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
        at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
        at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
        at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
        at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
        at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.transaction.HeuristicMixedException
        at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:484)
        at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:493)
        at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
        at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
        at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
        ... 24 more
Caused by: javax.transaction.xa.XAException
        at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:216)
        at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:168)
        at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:126)
        at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
        at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:446)
        ... 28 more
Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 1 from 16c41d24-5c95-0b85-2cd2-fa319c6923cc,52e1cf91-625a-57f3-6bf5-3ba2c746a47b,0d3029c5-bf1b-0a8b-c27a-a71672755ad1
        at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
        at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:166)
        ... 31 more
Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 1 from 16c41d24-5c95-0b85-2cd2-fa319c6923cc,52e1cf91-625a-57f3-6bf5-3ba2c746a47b,0d3029c5-bf1b-0a8b-c27a-a71672755ad1
        at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163)
        at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
        at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more

      �[0m�[33m07:30:33,042 WARN  [org.infinispan.topology.ClusterTopologyManagerImpl] (MSC service thread 1-5) ISPN000329: Unable to read rebalancing status from coordinator hsc-1-7824g: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node hsc-1-7824g was suspected
        at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39)
        at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23)
        at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27)
        at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51)
        at org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:35)
        at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:53)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1328)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1238)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:121)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$receive$0(JGroupsTransport.java:1371)
        at org.jgroups.util.MessageBatch.forEach(MessageBatch.java:318)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1371)
        at org.jgroups.JChannel.up(JChannel.java:846)
        at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:160)
        at org.jgroups.stack.Protocol.up(Protocol.java:372)
        at org.jgroups.protocols.FORK.up(FORK.java:162)
        at org.jgroups.protocols.FRAG3.up(FRAG3.java:190)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:416)
        at org.jgroups.stack.Protocol.up(Protocol.java:372)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:293)
        at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1024)
        at org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:797)
        at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:470)
        at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:697)
        at org.jgroups.protocols.Encrypt.up(Encrypt.java:190)
        at org.jgroups.stack.Protocol.up(Protocol.java:372)
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:212)
        at org.jgroups.stack.Protocol.up(Protocol.java:372)
        at org.jgroups.stack.Protocol.up(Protocol.java:372)
        at org.jgroups.stack.Protocol.up(Protocol.java:372)
        at org.jgroups.protocols.TP.passBatchUp(TP.java:1274)
        at org.jgroups.util.SubmitToThreadPool$BatchHandler.passBatchUp(SubmitToThreadPool.java:140)
        at org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:136)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
        at java.lang.Thread.run(Thread.java:748)

      and also

      �[0m�[31m07:35:09,337 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p8-t6) ISPN000136: Error executing command PrepareCommand, writing keys []: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.lang.IllegalArgumentException] while invoking method [public void org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.removed(org.infinispan.notifications.cachelistener.event.CacheEntryRemovedEvent)] on listener instance: org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager@749e9c42
        at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:392)
        at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
        at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:409)
        at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1573)
        at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1569)
        at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1564)
        at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1543)
        at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryRemoved(CacheNotifierImpl.java:427)
        at org.infinispan.commands.write.RemoveCommand.notify(RemoveCommand.java:106)
        at org.infinispan.commands.write.RemoveCommand.performRemove(RemoveCommand.java:232)
        at org.infinispan.commands.write.RemoveCommand.perform(RemoveCommand.java:102)
        at org.infinispan.interceptors.impl.CallInterceptor.visitCommand(CallInterceptor.java:29)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:156)
        at org.infinispan.interceptors.distribution.TxDistributionInterceptor.handleTxWriteCommand(TxDistributionInterceptor.java:398)
        at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRemoveCommand(TxDistributionInterceptor.java:137)
        at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58)
        at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
        at org.infinispan.interceptors.DDAsyncInterceptor.visitRemoveCommand(DDAsyncInterceptor.java:65)
        at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58)
        at org.infinispan.interceptors.impl.CacheLoaderInterceptor.visitDataCommand(CacheLoaderInterceptor.java:201)
        at org.infinispan.interceptors.impl.CacheLoaderInterceptor.visitRemoveCommand(CacheLoaderInterceptor.java:168)
        at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58)
        at org.infinispan.interceptors.impl.EntryWrappingInterceptor$EntryWrappingVisitor.handleWriteCommand(EntryWrappingInterceptor.java:780)
        at org.infinispan.interceptors.impl.EntryWrappingInterceptor$EntryWrappingVisitor.visitRemoveCommand(EntryWrappingInterceptor.java:715)
        at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)
        at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyModificationsAndThen(EntryWrappingInterceptor.java:837)
        at org.infinispan.interceptors.impl.EntryWrappingInterceptor.wrapEntriesForPrepareAndApply(EntryWrappingInterceptor.java:823)
        at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:186)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58)
        at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
        at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:132)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:102)
        at org.infinispan.interceptors.impl.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:46)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:102)
        at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:133)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:187)
        at org.infinispan.interceptors.impl.TxInterceptor.handlePrepareCommand(TxInterceptor.java:147)
        at org.infinispan.interceptors.impl.TxInterceptor.visitPrepareCommand(TxInterceptor.java:136)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58)
        at org.infinispan.interceptors.BaseAsyncInterceptor.lambda$new$0(BaseAsyncInterceptor.java:21)
        at org.infinispan.interceptors.InvocationSuccessFunction.apply(InvocationSuccessFunction.java:21)
        at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:67)
        at org.infinispan.interceptors.InvocationStage.thenApply(InvocationStage.java:24)
        at org.infinispan.interceptors.BaseAsyncInterceptor.asyncInvokeNext(BaseAsyncInterceptor.java:228)
        at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:46)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:189)
        at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:203)
        at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:69)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:127)
        at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:96)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:60)
        at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
        at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:132)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
        at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50)
        at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:234)
        at org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:109)
        at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:102)
        at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
        at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
        at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Cannot create a transactional context without a valid Transaction instance.
        at org.infinispan.context.TransactionalInvocationContextFactory.createInvocationContext(TransactionalInvocationContextFactory.java:69)
        at org.infinispan.context.TransactionalInvocationContextFactory.createInvocationContext(TransactionalInvocationContextFactory.java:63)
        at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:948)
        at org.infinispan.cache.impl.DecoratedCache.lock(DecoratedCache.java:163)
        at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.lock(AbstractDelegatingAdvancedCache.java:253)
        at org.infinispan.cache.impl.EncoderCache.lock(EncoderCache.java:355)
        at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.lock(AbstractDelegatingAdvancedCache.java:253)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.remove(InfinispanSessionMetaDataFactory.java:116)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.purge(InfinispanSessionMetaDataFactory.java:111)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.purge(InfinispanSessionMetaDataFactory.java:36)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:67)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:38)
        at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.removed(InfinispanSessionManager.java:283)
        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:387)
        ... 73 more

      Attachments

        1. hsc-1-bf4m8.log
          465 kB
        2. hsc-1-rwd5n.log
          112 kB
        3. JBEAP-14463-logs-eapcd-ocp311.zip
          4.02 MB
        4. logs.tgz
          40 kB
        5. logs.zip
          188 kB
        6. scaleUpFromThreeToSix_logs.zip
          227 kB
        7. standalone-openshift.xml
          34 kB

        Issue Links

          is caused by

          Bug - A problem that impairs or prevents the functions of the product. WFLY-9890 "ISPN000476: Timed out waiting for responses for request X from node Y" immediately after node Y rejoins the cluster (failover)

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-14477 EAP CD clustering scale up issues on OpenShift

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          is incorporated by

          Tracker - Issue tracking a bug fix or improvement in another component JBEAP-17060 OpenShift HA Tests Stabilisation Tracker

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is related to

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-17072 Clustering warnings/errors during scaledown of EAP 7.2 pod on Openshift

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Verified

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-14842 Occasional Infinispan TimeoutException during various OpenShift failover scenarios

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          relates to

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-17072 Clustering warnings/errors during scaledown of EAP 7.2 pod on Openshift

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Verified

          Activity

            People

              rhn-engineering-rhusar Radoslav Husar
              tremes1@redhat.com Tomas Remes
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: