========================================================================= JBoss Bootstrap Environment JBOSS_HOME: /opt/jboss/infinispan-server JAVA: /usr/lib/jvm/java/bin/java JAVA_OPTS: -server -server -Xms64m -Xmx512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true ========================================================================= 07:44:19,957 INFO [org.jboss.modules] (main) JBoss Modules version 1.5.2.Final 07:44:20,214 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final 07:44:20,296 INFO [org.jboss.as] (MSC service thread 1-8) WFLYSRV0049: Infinispan Server 9.1.0.Final (WildFly Core 2.2.0.Final) starting 07:44:22,148 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 12) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=datagrid-jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 07:44:22,430 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http) 07:44:22,496 INFO [org.xnio] (MSC service thread 1-8) XNIO version 3.4.0.Final 07:44:22,506 INFO [org.xnio.nio] (MSC service thread 1-8) XNIO NIO Implementation Version 3.4.0.Final 07:44:22,602 INFO [org.jboss.remoting] (MSC service thread 1-8) JBoss Remoting version 4.0.21.Final 07:44:22,628 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) Activating Infinispan subsystem. 07:44:22,735 INFO [org.infinispan.server.jgroups] (ServerService Thread Pool -- 21) DGJGRP0001: Activating JGroups subsystem. 07:44:22,743 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 29) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique. 07:44:22,741 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 22) WFLYIO001: Worker 'default' has auto-configured to 10 core threads with 80 task threads based on your 5 available processors 07:44:22,760 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 26) WFLYNAM0001: Activating Naming Subsystem 07:44:22,779 INFO [org.jboss.as.security] (ServerService Thread Pool -- 28) WFLYSEC0002: Activating Security Subsystem 07:44:22,811 INFO [org.jboss.as.connector] (MSC service thread 1-8) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.4.Final) 07:44:22,819 INFO [org.jboss.as.security] (MSC service thread 1-8) WFLYSEC0001: Current PicketBox version=4.9.6.Final 07:44:22,821 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 18) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3) 07:44:22,821 INFO [org.jboss.as.naming] (MSC service thread 1-6) WFLYNAM0003: Starting Naming Service 07:44:22,896 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-6) WFLYJCA0018: Started Driver service with driver-name = h2 07:44:23,765 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-6) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS] 07:44:24,128 INFO [org.jgroups.protocols.kubernetes.KUBE_PING] (MSC service thread 1-4) namespace myproject set; clustering enabled 07:44:24,135 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss/infinispan-server/standalone/deployments 07:44:27,227 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000078: Starting JGroups channel clustered 07:44:27,251 INFO [org.infinispan.CLUSTER] (MSC service thread 1-6) ISPN000094: Received new cluster view for channel cluster: [datagrid-3-kg593|1] (2) [datagrid-3-kg593, datagrid-3-tzh6h] 07:44:27,295 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000079: Channel clustered local address is datagrid-3-tzh6h, physical addresses are [172.17.0.9:7600] 07:44:27,320 INFO [org.infinispan.CLUSTER] (jgroups-3,datagrid-3-tzh6h) ISPN000094: Received new cluster view for channel cluster: [datagrid-3-kg593|2] (3) [datagrid-3-kg593, datagrid-3-tzh6h, datagrid-3-t3z1r] 07:44:27,336 INFO [org.infinispan.CLUSTER] (jgroups-3,datagrid-3-tzh6h) ISPN100000: Node datagrid-3-t3z1r joined the cluster 07:44:27,358 INFO [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-6) ISPN000390: Persisted state, version=9.1.0.Final timestamp=2017-10-19T07:44:27.355Z 07:44:27,360 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-6) ISPN000128: Infinispan version: Infinispan 'Bastille' 9.1.0.Final 07:44:28,502 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) DGISPN0001: Started repl cache from clustered container 07:44:28,530 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) DGISPN0001: Started default cache from clustered container 07:44:28,561 INFO [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10000: HotRodServer starting 07:44:28,571 INFO [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10001: HotRodServer listening on 172.17.0.9:11222 07:44:28,579 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10000: REST starting 07:44:28,881 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10002: REST listening on 172.17.0.9:8080 (mapped to rest) 07:44:29,174 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://0.0.0.0:9990/management 07:44:29,176 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://0.0.0.0:9990 07:44:29,177 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Infinispan Server 9.1.0.Final (WildFly Core 2.2.0.Final) started in 9627ms - Started 192 of 279 services (165 services are lazy, passive or on-demand) 07:49:18,807 INFO [org.jboss.as.protocol] (management I/O-2) WFLYPRT0057: cancelled task by interrupting thread Thread[management-handler-thread - 2,5,management-handler-thread] 07:50:03,759 WARN [org.jgroups.protocols.FD_SOCK] (FD_SOCK pinger-11,datagrid-3-tzh6h) datagrid-3-tzh6h: peer datagrid-3-t3z1r closed socket (eof) 07:50:08,833 INFO [org.infinispan.CLUSTER] (jgroups-3,datagrid-3-tzh6h) ISPN000094: Received new cluster view for channel cluster: [datagrid-3-kg593|3] (3) [datagrid-3-kg593, datagrid-3-tzh6h, datagrid-3-t3z1r] 07:50:08,834 INFO [org.infinispan.CLUSTER] (jgroups-3,datagrid-3-tzh6h) ISPN100000: Node datagrid-3-t3z1r joined the cluster 07:50:08,836 INFO [org.infinispan.CLUSTER] (jgroups-3,datagrid-3-tzh6h) ISPN100001: Node datagrid-3-t3z1r left the cluster 07:50:08,850 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-4) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$null$3(JGroupsTransport.java:711) at org.infinispan.remoting.transport.impl.RequestRepository.lambda$forEach$0(RequestRepository.java:61) at java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1597) at org.infinispan.remoting.transport.impl.RequestRepository.forEach(RequestRepository.java:61) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$receiveClusterView$4(JGroupsTransport.java:711) at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:203) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:50:12,909 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:17,221 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:21,443 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:25,508 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:25,652 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p3-t1) ISPN000136: Error executing command PutKeyValueCommand, writing keys [WrappedByteArray{bytes=[B0x4A2E383530313136..[48], hashCode=-1944977757}]: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 11 at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:356) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:50:29,590 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-10) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:33,623 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-7) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:37,662 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-7) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:41,683 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-7) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:45,711 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:49,734 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:53,785 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:57,798 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:01,855 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:05,862 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:09,898 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:13,962 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-10) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:18,758 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:22,919 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-7) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:26,961 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-7) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:30,993 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-10) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:35,049 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-10) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:39,139 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-10) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:43,238 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-10) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:47,278 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:51,304 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:55,383 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:59,500 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:03,568 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-10) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:07,619 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:11,658 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:15,724 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-7) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:18,942 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:22,794 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-10) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:27,084 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-10) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:31,184 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-10) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:35,289 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-10) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:39,358 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-10) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:43,589 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-10) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:47,679 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-10) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:51,768 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-10) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:53:10,346 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-15) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355165 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,348 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-3) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355165 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355165 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,361 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-9) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355178 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,364 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-10) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355182 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,362 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-26) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355179 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,365 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-8) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355182 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355182 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,368 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-29) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355185 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,368 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-8) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355184 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,374 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-18) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355187 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,376 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-30) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355196 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,374 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-28) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355190 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,376 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-7) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355193 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,376 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-9) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355184 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355184 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,406 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-12) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355199 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,407 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-21) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355202 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,407 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-3) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355201 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,409 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-5) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355193 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355193 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,405 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-11) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355197 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,418 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-6) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355209 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,418 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-5) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355221 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,418 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-14) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355230 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,418 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-22) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355210 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,419 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-27) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355214 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,418 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-32) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355212 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,434 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-20) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355241 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,420 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-23) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355227 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,435 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-4) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355244 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,419 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-31) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355223 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,420 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-17) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355228 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,420 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-13) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355225 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,434 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-19) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355240 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,419 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-25) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355224 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,420 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-3) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355201 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355201 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,439 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-24) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355254 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,421 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-9) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355202 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355202 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,447 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-8) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355241 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355241 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,432 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t22) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@118ce379, topologyId=10, id=datagrid-3-kg593312907, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355231 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,433 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t15) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@520295cf, topologyId=10, id=datagrid-3-kg593312963, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355235 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,423 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-16) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355229 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,433 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t2) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@6af11c2, topologyId=10, id=datagrid-3-kg593312959, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355232 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,433 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t14) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@30325297, topologyId=10, id=datagrid-3-kg593312960, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355234 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,437 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t19) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@4e46308e, topologyId=10, id=datagrid-3-kg593312958, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355246 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,434 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t24) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@4367c6b3, topologyId=10, id=datagrid-3-kg593312942, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355237 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,433 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-2) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355230 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355230 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,434 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t13) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@6d6932b4, topologyId=10, id=datagrid-3-kg593312940, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355242 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,433 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t21) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@53ac6ed7, topologyId=10, id=datagrid-3-kg593312947, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355233 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,435 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t1) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@4e1ca54f, topologyId=10, id=datagrid-3-kg593312961, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355245 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,435 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-1) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355227 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355227 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,433 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t4) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@6c360732, topologyId=10, id=datagrid-3-kg593312909, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355236 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,438 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-3) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355221 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355221 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,437 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-5) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355228 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355228 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,468 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-2) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355244 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355244 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,468 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-1) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355225 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355225 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,469 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-2) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355254 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355254 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,438 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t20) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@7ace0ede, topologyId=10, id=datagrid-3-kg593312964, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355258 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,446 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t23) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@271cd7f0, topologyId=10, id=datagrid-3-kg593313024, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355263 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,446 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t9) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@6672068e, topologyId=10, id=datagrid-3-kg593313017, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355262 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,439 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t6) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@58b98131, topologyId=10, id=datagrid-3-kg593312956, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355249 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,439 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t5) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@1e901886, topologyId=10, id=datagrid-3-kg593312957, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355261 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,439 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t12) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@3421dab6, topologyId=10, id=datagrid-3-kg593312948, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355259 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,438 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t18) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@50e032f4, topologyId=10, id=datagrid-3-kg593312943, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355252 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,438 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t16) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@b355561, topologyId=10, id=datagrid-3-kg593312962, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355253 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,439 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t8) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@16bc8504, topologyId=10, id=datagrid-3-kg593313015, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355256 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,447 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t3) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@58a1fb65, topologyId=10, id=datagrid-3-kg593313016, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355255 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,438 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t7) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@5fd47513, topologyId=10, id=datagrid-3-kg593313014, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355257 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,438 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t10) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@796ec9d3, topologyId=10, id=datagrid-3-kg593312950, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355248 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,458 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-9) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355197 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355197 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,439 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t11) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@7e9ecdfb, topologyId=10, id=datagrid-3-kg593313022, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355260 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,439 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t25) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@675a9dc7, topologyId=10, id=datagrid-3-kg593313018, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355250 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,438 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t17) ISPN000071: Caught exception when handling command StreamRequestCommand{type=TERMINAL_REHASH, includeLoader=true, terminalOperation=org.infinispan.stream.impl.termop.SegmentRetryingOperation@484a7475, topologyId=10, id=datagrid-3-kg593313013, segments=[], keys=[], excludedKeys=[]}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355247 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:10,805 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-6) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:851) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.TimeoutException at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:877) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) ... 32 more 07:53:10,819 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-6) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355178 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355178 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,819 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-6) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355187 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355187 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,829 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-10) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355199 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355199 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,829 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-10) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355210 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355210 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,829 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-4) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355209 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355209 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,830 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-4) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355229 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355229 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:10,830 WARNING [io.netty.channel.DefaultChannelPipeline] (HotRod-ServerWorker-5-7) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355240 from datagrid-3-kg593 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:464) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:459) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:105) at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$0(ContextHandler.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 355240 from datagrid-3-kg593 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:53:21,126 WARN [org.infinispan.stream.impl.ClusterStreamManagerImpl] (timeout-thread--p3-t1) Unhandled remote stream exception encountered