========================================================================= JBoss Bootstrap Environment JBOSS_HOME: /opt/jboss/infinispan-server JAVA: /usr/lib/jvm/java/bin/java JAVA_OPTS: -server -server -Xms64m -Xmx512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true ========================================================================= 07:44:19,735 INFO [org.jboss.modules] (main) JBoss Modules version 1.5.2.Final 07:44:20,011 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final 07:44:20,125 INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0049: Infinispan Server 9.1.0.Final (WildFly Core 2.2.0.Final) starting 07:44:22,559 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 6) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=datagrid-jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 07:44:22,762 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http) 07:44:22,822 INFO [org.xnio] (MSC service thread 1-6) XNIO version 3.4.0.Final 07:44:22,838 INFO [org.xnio.nio] (MSC service thread 1-6) XNIO NIO Implementation Version 3.4.0.Final 07:44:22,919 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) Activating Infinispan subsystem. 07:44:22,932 INFO [org.infinispan.server.jgroups] (ServerService Thread Pool -- 21) DGJGRP0001: Activating JGroups subsystem. 07:44:22,955 INFO [org.jboss.as.connector] (MSC service thread 1-3) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.4.Final) 07:44:22,984 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 29) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique. 07:44:22,979 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 22) WFLYIO001: Worker 'default' has auto-configured to 10 core threads with 80 task threads based on your 5 available processors 07:44:23,007 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 26) WFLYNAM0001: Activating Naming Subsystem 07:44:23,016 INFO [org.jboss.as.security] (ServerService Thread Pool -- 28) WFLYSEC0002: Activating Security Subsystem 07:44:23,032 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 18) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3) 07:44:23,040 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0018: Started Driver service with driver-name = h2 07:44:23,057 INFO [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 4.0.21.Final 07:44:23,116 INFO [org.jboss.as.security] (MSC service thread 1-6) WFLYSEC0001: Current PicketBox version=4.9.6.Final 07:44:23,213 INFO [org.jboss.as.naming] (MSC service thread 1-3) WFLYNAM0003: Starting Naming Service 07:44:24,243 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS] 07:44:24,426 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-4) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss/infinispan-server/standalone/deployments 07:44:24,466 INFO [org.jgroups.protocols.kubernetes.KUBE_PING] (MSC service thread 1-6) namespace myproject set; clustering enabled 07:44:26,394 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000078: Starting JGroups channel clustered 07:44:26,397 INFO [org.infinispan.CLUSTER] (MSC service thread 1-4) ISPN000094: Received new cluster view for channel cluster: [datagrid-3-kg593|1] (2) [datagrid-3-kg593, datagrid-3-tzh6h] 07:44:26,440 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000079: Channel clustered local address is datagrid-3-kg593, physical addresses are [172.17.0.6:7600] 07:44:26,462 INFO [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-4) ISPN000390: Persisted state, version=9.1.0.Final timestamp=2017-10-19T07:44:26.451Z 07:44:26,463 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-4) ISPN000128: Infinispan version: Infinispan 'Bastille' 9.1.0.Final 07:44:27,183 INFO [org.infinispan.CLUSTER] (ViewHandler-10,datagrid-3-kg593) ISPN000094: Received new cluster view for channel cluster: [datagrid-3-kg593|2] (3) [datagrid-3-kg593, datagrid-3-tzh6h, datagrid-3-t3z1r] 07:44:27,214 INFO [org.infinispan.CLUSTER] (ViewHandler-10,datagrid-3-kg593) ISPN100000: Node datagrid-3-t3z1r joined the cluster 07:44:27,234 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) DGISPN0001: Started default cache from clustered container 07:44:27,250 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) DGISPN0001: Started repl cache from clustered container 07:44:27,301 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10000: HotRodServer starting 07:44:27,318 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10001: HotRodServer listening on 172.17.0.6:11222 07:44:27,331 INFO [org.infinispan.server.endpoint] (MSC service thread 1-8) DGENDPT10000: REST starting 07:44:27,526 INFO [org.infinispan.server.endpoint] (MSC service thread 1-8) DGENDPT10002: REST listening on 172.17.0.6:8080 (mapped to rest) 07:44:27,831 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://0.0.0.0:9990/management 07:44:27,833 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://0.0.0.0:9990 07:44:27,833 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Infinispan Server 9.1.0.Final (WildFly Core 2.2.0.Final) started in 8456ms - Started 192 of 279 services (165 services are lazy, passive or on-demand) 07:44:28,181 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t6) ISPN000310: Starting cluster-wide rebalance for cache ___protobuf_metadata, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[datagrid-3-kg593: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[datagrid-3-kg593: 122, datagrid-3-tzh6h: 134]}, unionCH=null, phase=READ_OLD_WRITE_ALL, actualMembers=[datagrid-3-kg593, datagrid-3-tzh6h], persistentUUIDs=[f6732a42-ca58-4bc0-9765-485ec573a3ca, d164e1af-9cbe-411e-bd2a-f9b7723c21a4]} 07:44:28,189 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t6) [Context=___protobuf_metadata][Context=datagrid-3-kg593]ISPN100002: Started rebalance with topology id 2 07:44:28,216 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t20) [Context=___protobuf_metadata][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 2 07:44:28,242 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t7) ISPN000310: Starting cluster-wide rebalance for cache repl, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[datagrid-3-kg593: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[datagrid-3-kg593: 122, datagrid-3-tzh6h: 134]}, unionCH=null, phase=READ_OLD_WRITE_ALL, actualMembers=[datagrid-3-kg593, datagrid-3-tzh6h], persistentUUIDs=[f6732a42-ca58-4bc0-9765-485ec573a3ca, d164e1af-9cbe-411e-bd2a-f9b7723c21a4]} 07:44:28,243 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t7) [Context=repl][Context=datagrid-3-kg593]ISPN100002: Started rebalance with topology id 2 07:44:28,255 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t23) [Context=repl][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 2 07:44:28,304 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=2, rebalanceId=2, currentCH=DefaultConsistentHash{ns=256, owners = (1)[datagrid-3-kg593: 256+0]}, pendingCH=DefaultConsistentHash{ns=256, owners = (2)[datagrid-3-kg593: 122+134, datagrid-3-tzh6h: 134+122]}, unionCH=null, phase=READ_OLD_WRITE_ALL, actualMembers=[datagrid-3-kg593, datagrid-3-tzh6h], persistentUUIDs=[f6732a42-ca58-4bc0-9765-485ec573a3ca, d164e1af-9cbe-411e-bd2a-f9b7723c21a4]} 07:44:28,305 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) [Context=default][Context=datagrid-3-kg593]ISPN100002: Started rebalance with topology id 2 07:44:28,325 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t1) [Context=default][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 2 07:44:28,417 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t17) [Context=___protobuf_metadata][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 2 07:44:28,419 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t17) ISPN000336: Finished cluster-wide rebalance for cache ___protobuf_metadata, topology id = 2 07:44:28,425 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t18) [Context=repl][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 2 07:44:28,449 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t7) [Context=___protobuf_metadata][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 3 07:44:28,436 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t18) ISPN000336: Finished cluster-wide rebalance for cache repl, topology id = 2 07:44:28,453 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t19) [Context=default][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 2 07:44:28,456 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t20) [Context=___protobuf_metadata][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 3 07:44:28,457 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t19) ISPN000336: Finished cluster-wide rebalance for cache default, topology id = 2 07:44:28,463 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t8) [Context=repl][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 3 07:44:28,470 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t21) [Context=repl][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 3 07:44:28,466 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t10) [Context=default][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 3 07:44:28,465 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t13) [Context=___protobuf_metadata][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 4 07:44:28,474 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t22) [Context=___protobuf_metadata][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 4 07:44:28,479 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t19) [Context=repl][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 4 07:44:28,479 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t23) [Context=default][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 3 07:44:28,487 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t24) [Context=repl][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 4 07:44:28,492 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t23) [Context=default][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 4 07:44:28,494 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) [Context=default][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 4 07:44:28,624 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t1) ISPN000310: Starting cluster-wide rebalance for cache ___script_cache, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[datagrid-3-kg593: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[datagrid-3-kg593: 122, datagrid-3-tzh6h: 134]}, unionCH=null, phase=READ_OLD_WRITE_ALL, actualMembers=[datagrid-3-kg593, datagrid-3-tzh6h], persistentUUIDs=[f6732a42-ca58-4bc0-9765-485ec573a3ca, d164e1af-9cbe-411e-bd2a-f9b7723c21a4]} 07:44:28,624 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t1) [Context=___script_cache][Context=datagrid-3-kg593]ISPN100002: Started rebalance with topology id 2 07:44:28,642 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t8) [Context=___script_cache][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 2 07:44:28,770 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t3) [Context=___script_cache][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 2 07:44:28,772 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t3) ISPN000336: Finished cluster-wide rebalance for cache ___script_cache, topology id = 2 07:44:28,775 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t13) [Context=___script_cache][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 3 07:44:28,793 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t4) [Context=___script_cache][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 3 07:44:28,796 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t17) [Context=___script_cache][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 4 07:44:28,797 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t5) [Context=___script_cache][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 4 07:44:28,884 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t6) ISPN000310: Starting cluster-wide rebalance for cache ___hotRodTopologyCache, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[datagrid-3-kg593: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[datagrid-3-kg593: 122, datagrid-3-tzh6h: 134]}, unionCH=null, phase=READ_OLD_WRITE_ALL, actualMembers=[datagrid-3-kg593, datagrid-3-tzh6h], persistentUUIDs=[f6732a42-ca58-4bc0-9765-485ec573a3ca, d164e1af-9cbe-411e-bd2a-f9b7723c21a4]} 07:44:28,885 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t6) [Context=___hotRodTopologyCache][Context=datagrid-3-kg593]ISPN100002: Started rebalance with topology id 2 07:44:28,897 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t25) [Context=___hotRodTopologyCache][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 2 07:44:28,921 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t9) [Context=___hotRodTopologyCache][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 2 07:44:28,927 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t9) ISPN000336: Finished cluster-wide rebalance for cache ___hotRodTopologyCache, topology id = 2 07:44:28,930 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t1) [Context=___hotRodTopologyCache][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 3 07:44:28,939 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) [Context=___hotRodTopologyCache][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 3 07:44:28,942 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t6) [Context=___hotRodTopologyCache][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 4 07:44:28,944 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t13) [Context=___hotRodTopologyCache][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 4 07:44:29,308 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t14) ISPN000310: Starting cluster-wide rebalance for cache ___protobuf_metadata, topology CacheTopology{id=6, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[datagrid-3-kg593: 122, datagrid-3-tzh6h: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[datagrid-3-kg593: 85, datagrid-3-tzh6h: 88, datagrid-3-t3z1r: 83]}, unionCH=null, phase=READ_OLD_WRITE_ALL, actualMembers=[datagrid-3-kg593, datagrid-3-tzh6h, datagrid-3-t3z1r], persistentUUIDs=[f6732a42-ca58-4bc0-9765-485ec573a3ca, d164e1af-9cbe-411e-bd2a-f9b7723c21a4, fabf42f9-be60-4ac4-b7a9-892df4c059b0]} 07:44:29,309 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t14) [Context=___protobuf_metadata][Context=datagrid-3-kg593]ISPN100002: Started rebalance with topology id 6 07:44:29,311 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t16) [Context=___protobuf_metadata][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 6 07:44:29,314 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t14) [Context=___protobuf_metadata][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 6 07:44:29,332 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t15) ISPN000310: Starting cluster-wide rebalance for cache repl, topology CacheTopology{id=6, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[datagrid-3-kg593: 122, datagrid-3-tzh6h: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[datagrid-3-kg593: 85, datagrid-3-tzh6h: 88, datagrid-3-t3z1r: 83]}, unionCH=null, phase=READ_OLD_WRITE_ALL, actualMembers=[datagrid-3-kg593, datagrid-3-tzh6h, datagrid-3-t3z1r], persistentUUIDs=[f6732a42-ca58-4bc0-9765-485ec573a3ca, d164e1af-9cbe-411e-bd2a-f9b7723c21a4, fabf42f9-be60-4ac4-b7a9-892df4c059b0]} 07:44:29,332 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t15) [Context=repl][Context=datagrid-3-kg593]ISPN100002: Started rebalance with topology id 6 07:44:29,333 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t16) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=6, rebalanceId=3, currentCH=DefaultConsistentHash{ns=256, owners = (2)[datagrid-3-kg593: 122+134, datagrid-3-tzh6h: 134+122]}, pendingCH=DefaultConsistentHash{ns=256, owners = (3)[datagrid-3-kg593: 85+83, datagrid-3-tzh6h: 88+85, datagrid-3-t3z1r: 83+88]}, unionCH=null, phase=READ_OLD_WRITE_ALL, actualMembers=[datagrid-3-kg593, datagrid-3-tzh6h, datagrid-3-t3z1r], persistentUUIDs=[f6732a42-ca58-4bc0-9765-485ec573a3ca, d164e1af-9cbe-411e-bd2a-f9b7723c21a4, fabf42f9-be60-4ac4-b7a9-892df4c059b0]} 07:44:29,334 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t16) [Context=default][Context=datagrid-3-kg593]ISPN100002: Started rebalance with topology id 6 07:44:29,334 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t14) [Context=repl][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 6 07:44:29,340 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t17) [Context=default][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 6 07:44:29,340 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t18) [Context=repl][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 6 07:44:29,342 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t14) [Context=default][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 6 07:44:29,433 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) [Context=default][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 6 07:44:29,434 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) ISPN000336: Finished cluster-wide rebalance for cache default, topology id = 6 07:44:29,436 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t24) [Context=default][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 7 07:44:29,441 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) [Context=default][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 7 07:44:29,447 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t3) [Context=repl][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 6 07:44:29,447 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t3) ISPN000336: Finished cluster-wide rebalance for cache repl, topology id = 6 07:44:29,448 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t2) [Context=repl][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 7 07:44:29,449 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t4) [Context=default][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 7 07:44:29,450 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t5) [Context=___protobuf_metadata][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 6 07:44:29,450 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t5) ISPN000336: Finished cluster-wide rebalance for cache ___protobuf_metadata, topology id = 6 07:44:29,451 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t1) [Context=default][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 8 07:44:29,456 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t12) [Context=___protobuf_metadata][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 7 07:44:29,457 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t6) [Context=default][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 8 07:44:29,457 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t7) [Context=___protobuf_metadata][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 7 07:44:29,457 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t8) [Context=repl][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 7 07:44:29,464 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t9) [Context=default][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 8 07:44:29,467 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) [Context=repl][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 7 07:44:29,468 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t13) [Context=___protobuf_metadata][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 7 07:44:29,478 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t4) [Context=repl][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 8 07:44:29,488 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t11) [Context=repl][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 8 07:44:29,490 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t12) [Context=___protobuf_metadata][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 8 07:44:29,491 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t14) [Context=___protobuf_metadata][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 8 07:44:29,495 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t14) [Context=___protobuf_metadata][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 8 07:44:29,505 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t15) [Context=repl][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 8 07:44:29,627 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t16) ISPN000310: Starting cluster-wide rebalance for cache ___script_cache, topology CacheTopology{id=6, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[datagrid-3-kg593: 122, datagrid-3-tzh6h: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[datagrid-3-kg593: 85, datagrid-3-tzh6h: 88, datagrid-3-t3z1r: 83]}, unionCH=null, phase=READ_OLD_WRITE_ALL, actualMembers=[datagrid-3-kg593, datagrid-3-tzh6h, datagrid-3-t3z1r], persistentUUIDs=[f6732a42-ca58-4bc0-9765-485ec573a3ca, d164e1af-9cbe-411e-bd2a-f9b7723c21a4, fabf42f9-be60-4ac4-b7a9-892df4c059b0]} 07:44:29,631 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t16) [Context=___script_cache][Context=datagrid-3-kg593]ISPN100002: Started rebalance with topology id 6 07:44:29,636 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t23) [Context=___script_cache][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 6 07:44:29,640 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t17) [Context=___script_cache][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 6 07:44:29,779 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t20) [Context=___script_cache][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 6 07:44:29,781 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t20) ISPN000336: Finished cluster-wide rebalance for cache ___script_cache, topology id = 6 07:44:29,782 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t7) [Context=___script_cache][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 7 07:44:29,796 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t19) [Context=___script_cache][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 7 07:44:29,804 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t21) [Context=___script_cache][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 7 07:44:29,821 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t9) [Context=___script_cache][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 8 07:44:29,824 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t22) [Context=___script_cache][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 8 07:44:29,830 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t23) [Context=___script_cache][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 8 07:44:29,857 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t24) ISPN000310: Starting cluster-wide rebalance for cache ___hotRodTopologyCache, topology CacheTopology{id=6, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[datagrid-3-kg593: 122, datagrid-3-tzh6h: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[datagrid-3-kg593: 85, datagrid-3-tzh6h: 88, datagrid-3-t3z1r: 83]}, unionCH=null, phase=READ_OLD_WRITE_ALL, actualMembers=[datagrid-3-kg593, datagrid-3-tzh6h, datagrid-3-t3z1r], persistentUUIDs=[f6732a42-ca58-4bc0-9765-485ec573a3ca, d164e1af-9cbe-411e-bd2a-f9b7723c21a4, fabf42f9-be60-4ac4-b7a9-892df4c059b0]} 07:44:29,858 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t24) [Context=___hotRodTopologyCache][Context=datagrid-3-kg593]ISPN100002: Started rebalance with topology id 6 07:44:29,861 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t13) [Context=___hotRodTopologyCache][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 6 07:44:29,863 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) [Context=___hotRodTopologyCache][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 6 07:44:29,915 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t4) [Context=___hotRodTopologyCache][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 6 07:44:29,916 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t4) ISPN000336: Finished cluster-wide rebalance for cache ___hotRodTopologyCache, topology id = 6 07:44:29,917 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t17) [Context=___hotRodTopologyCache][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 7 07:44:29,918 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t5) [Context=___hotRodTopologyCache][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 7 07:44:29,922 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t7) [Context=___hotRodTopologyCache][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 7 07:44:29,924 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t22) [Context=___hotRodTopologyCache][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 8 07:44:29,927 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t8) [Context=___hotRodTopologyCache][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 8 07:44:29,928 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t6) [Context=___hotRodTopologyCache][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 8 07:50:08,818 INFO [org.infinispan.CLUSTER] (ViewHandler-21,datagrid-3-kg593) ISPN000094: Received new cluster view for channel cluster: [datagrid-3-kg593|3] (3) [datagrid-3-kg593, datagrid-3-tzh6h, datagrid-3-t3z1r] 07:50:08,819 INFO [org.infinispan.CLUSTER] (ViewHandler-21,datagrid-3-kg593) ISPN100000: Node datagrid-3-t3z1r joined the cluster 07:50:08,822 INFO [org.infinispan.CLUSTER] (ViewHandler-21,datagrid-3-kg593) ISPN100001: Node datagrid-3-t3z1r left the cluster 07:50:08,830 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-1) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$null$3(JGroupsTransport.java:711) at org.infinispan.remoting.transport.impl.RequestRepository.lambda$forEach$0(RequestRepository.java:61) at java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1597) at org.infinispan.remoting.transport.impl.RequestRepository.forEach(RequestRepository.java:61) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$receiveClusterView$4(JGroupsTransport.java:711) at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:203) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 07:50:10,596 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t21) ISPN000310: Starting cluster-wide rebalance for cache ___protobuf_metadata, topology CacheTopology{id=10, rebalanceId=4, currentCH=ReplicatedConsistentHash{ns = 256, owners = (3)[datagrid-3-kg593: 85, datagrid-3-tzh6h: 88, datagrid-3-t3z1r: 83]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (4)[datagrid-3-kg593: 60, datagrid-3-tzh6h: 67, datagrid-3-t3z1r: 67, datagrid-3-t3z1r: 62]}, unionCH=null, phase=READ_OLD_WRITE_ALL, actualMembers=[datagrid-3-kg593, datagrid-3-tzh6h, datagrid-3-t3z1r, datagrid-3-t3z1r], persistentUUIDs=[f6732a42-ca58-4bc0-9765-485ec573a3ca, d164e1af-9cbe-411e-bd2a-f9b7723c21a4, fabf42f9-be60-4ac4-b7a9-892df4c059b0, c8112fbc-d483-4c12-8e5a-7e883554f213]} 07:50:10,598 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t21) [Context=___protobuf_metadata][Context=datagrid-3-kg593]ISPN100002: Started rebalance with topology id 10 07:50:10,600 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t7) [Context=___protobuf_metadata][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 10 07:50:10,603 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t20) [Context=___protobuf_metadata][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 10 07:50:10,632 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t22) ISPN000310: Starting cluster-wide rebalance for cache repl, topology CacheTopology{id=10, rebalanceId=4, currentCH=ReplicatedConsistentHash{ns = 256, owners = (3)[datagrid-3-kg593: 85, datagrid-3-tzh6h: 88, datagrid-3-t3z1r: 83]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (4)[datagrid-3-kg593: 60, datagrid-3-tzh6h: 67, datagrid-3-t3z1r: 67, datagrid-3-t3z1r: 62]}, unionCH=null, phase=READ_OLD_WRITE_ALL, actualMembers=[datagrid-3-kg593, datagrid-3-tzh6h, datagrid-3-t3z1r, datagrid-3-t3z1r], persistentUUIDs=[f6732a42-ca58-4bc0-9765-485ec573a3ca, d164e1af-9cbe-411e-bd2a-f9b7723c21a4, fabf42f9-be60-4ac4-b7a9-892df4c059b0, c8112fbc-d483-4c12-8e5a-7e883554f213]} 07:50:10,632 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t22) [Context=repl][Context=datagrid-3-kg593]ISPN100002: Started rebalance with topology id 10 07:50:10,633 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t20) [Context=repl][Context=datagrid-3-kg593]ISPN100003: Node datagrid-3-kg593 finished rebalance phase with topology id 10 07:50:10,635 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t23) [Context=repl][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 10 07:50:10,646 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=10, rebalanceId=4, currentCH=DefaultConsistentHash{ns=256, owners = (3)[datagrid-3-kg593: 85+83, datagrid-3-tzh6h: 88+85, datagrid-3-t3z1r: 83+88]}, pendingCH=DefaultConsistentHash{ns=256, owners = (4)[datagrid-3-kg593: 60+57, datagrid-3-tzh6h: 67+59, datagrid-3-t3z1r: 67+73, datagrid-3-t3z1r: 62+67]}, unionCH=null, phase=READ_OLD_WRITE_ALL, actualMembers=[datagrid-3-kg593, datagrid-3-tzh6h, datagrid-3-t3z1r, datagrid-3-t3z1r], persistentUUIDs=[f6732a42-ca58-4bc0-9765-485ec573a3ca, d164e1af-9cbe-411e-bd2a-f9b7723c21a4, fabf42f9-be60-4ac4-b7a9-892df4c059b0, c8112fbc-d483-4c12-8e5a-7e883554f213]} 07:50:10,647 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) [Context=default][Context=datagrid-3-kg593]ISPN100002: Started rebalance with topology id 10 07:50:10,650 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t1) [Context=default][Context=datagrid-3-tzh6h]ISPN100003: Node datagrid-3-tzh6h finished rebalance phase with topology id 10 07:50:10,654 WARN [org.infinispan.statetransfer.InboundTransferTask] (stateTransferExecutor-thread--p7-t1) ISPN000210: Failed to request state of cache default from node datagrid-3-t3z1r, segments {84 225}: org.infinispan.commons.CacheException: CacheNotFoundResponse at org.infinispan.statetransfer.InboundTransferTask.startTransfer(InboundTransferTask.java:147) at org.infinispan.statetransfer.InboundTransferTask.requestSegments(InboundTransferTask.java:117) at org.infinispan.statetransfer.StateConsumerImpl.lambda$addTransfer$8(StateConsumerImpl.java:1102) at org.infinispan.executors.LimitedExecutor.lambda$executeAsync$1(LimitedExecutor.java:101) at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144) at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33) at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:50:11,173 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t24) [Context=___protobuf_metadata][Context=datagrid-3-t3z1r]ISPN100003: Node datagrid-3-t3z1r finished rebalance phase with topology id 10 07:50:12,878 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:16,905 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:21,327 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-1) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:25,388 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-1) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:29,422 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-1) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:33,462 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-1) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:37,484 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-1) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:41,536 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-1) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:45,572 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-1) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:49,612 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-1) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:53,665 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-1) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:50:57,721 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:01,760 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-1) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:05,785 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:09,831 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:13,887 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:18,843 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-3) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:22,863 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-1) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:26,909 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:30,971 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:35,012 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-1) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:39,051 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:43,106 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:47,173 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:51,219 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-9) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:55,280 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:51:59,380 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:03,421 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-9) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:07,455 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-9) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:11,519 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-9) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:15,560 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-9) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:18,833 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-9) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:22,805 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:27,072 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:31,170 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:35,335 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-9) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:39,645 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-9) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:43,672 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:47,902 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-9) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:52:51,881 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node datagrid-3-t3z1r was suspected at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:39) at org.infinispan.remoting.transport.jgroups.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:23) at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:27) at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:51) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onNewView(SingleTargetRequest.java:42) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addRequest(JGroupsTransport.java:949) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:850) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.performSyncRemoteInvocation(JGroupsTransport.java:1094) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:281) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:127) at org.infinispan.distexec.DefaultExecutorService$RemoteDistributedTaskPart.execute(DefaultExecutorService.java:980) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:495) at org.infinispan.distexec.DefaultExecutorService.submitEverywhere(DefaultExecutorService.java:476) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:98) ... 31 more 07:53:10,596 WARN [org.infinispan.stream.impl.ClusterStreamManagerImpl] (timeout-thread--p3-t1) Unhandled remote stream exception encountered 07:53:11,299 ERROR [org.infinispan.stats.impl.ClusterCacheStatsImpl] (HotRod-ServerWorker-5-5) Could not execute cluster wide cache stats operation : java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:851) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:100) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:119) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:198) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsInt(AbstractClusterStats.java:193) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getNumberOfEntries(ClusterCacheStatsImpl.java:196) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getCurrentNumberOfEntries(ClusterCacheStatsImpl.java:259) at org.infinispan.server.hotrod.Decoder2x.createStatsResponse(Decoder2x.java:615) at org.infinispan.server.hotrod.LocalContextHandler.realChannelRead(LocalContextHandler.java:57) at org.infinispan.server.hotrod.LocalContextHandler.channelRead(LocalContextHandler.java:33) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.getResult(DefaultExecutorService.java:894) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.innerGet(DefaultExecutorService.java:868) at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:849) ... 32 more 07:53:25,883 WARN [org.infinispan.stream.impl.ClusterStreamManagerImpl] (timeout-thread--p3-t1) Unhandled remote stream exception encountered 07:53:30,198 WARN [org.infinispan.stream.impl.ClusterStreamManagerImpl] (timeout-thread--p3-t1) Unhandled remote stream exception encountered 07:53:30,333 WARN [org.infinispan.stream.impl.ClusterStreamManagerImpl] (timeout-thread--p3-t1) Unhandled remote stream exception encountered 07:53:31,154 WARN [org.infinispan.stream.impl.ClusterStreamManagerImpl] (timeout-thread--p3-t1) Unhandled remote stream exception encountered 07:53:34,093 WARN [org.infinispan.stream.impl.ClusterStreamManagerImpl] (timeout-thread--p3-t1) Unhandled remote stream exception encountered 07:53:34,093 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-55) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 535256 from datagrid-3-tzh6h at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:35,383 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-72) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 535244 from datagrid-3-tzh6h at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 07:53:34,455 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-ServerHandler-6-67) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 535266 from datagrid-3-tzh6h at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86) at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)