Details
-
Bug
-
Resolution: Done
-
Blocker
-
5.1.0.FINAL
-
None
-
None
Description
Another resilience test: 4 nodes node2 goes down and up again:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/job/edg-60-failover-dist-basic/16/artifact/report/stats-throughput.png
Test timeline goes like this:
2012-02-01 04:39:21,993: 20 hotrod clients started
2012-02-01 04:42:21,995: node02 killed
2012-02-01 04:45:27,950: node02 up again
Endpoints:
172.18.1.1:11222 - node01
172.18.1.3:11222 - node02
172.18.1.5:11222 - node03
172.18.1.7:11222 - node04
in the client side log
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/job/edg-60-failover-dist-basic/16/console-edg-perf06/consoleText
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/job/edg-60-failover-dist-basic/16/console-edg-perf07/consoleText
some warnings about connection refused and then node02 being removed from the pool by Timer thread:
2012-02-01 04:43:15,608 254109 INFO [org.infinispan.client.hotrod.impl.protocol.Codec11] (Timer-1:) ISPN004006: New topology: [/172.18.1.1:11222, /172.18.1.7:11222, /172.18.1.5:11222] 2012-02-01 04:43:15,609 254110 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004014: New server added(/172.18.1.1:11222), adding to the pool. 2012-02-01 04:43:15,610 254111 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004014: New server added(/172.18.1.7:11222), adding to the pool. 2012-02-01 04:43:15,611 254112 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004014: New server added(/172.18.1.5:11222), adding to the pool. 2012-02-01 04:43:15,612 254113 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004016: Server not in cluster anymore(/172.18.1.3:11222), removing from the pool.
Log of Client-19
2012-02-01 04:39:20,962 19437 DEBUG [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) Statically configured servers: [/172.18.1.7:11222, /172.18.1.5:11222, /172.18.1.3:11222, /172.18.1.1:11222]
2012-02-01 04:39:20,962 19437 DEBUG [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) Load balancer class: org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy
2012-02-01 04:39:20,963 19438 DEBUG [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) Tcp no delay = true; client socket timeout = 120000 ms; connect timeout = 120000 ms
2012-02-01 04:39:20,996 19471 INFO [org.infinispan.client.hotrod.impl.protocol.Codec11] (Client-19:) ISPN004006: New topology: [/172.18.1.1:11222, /172.18.1.3:11222, /172.18.1.7:11222, /172.18.1.5:11222]
2012-02-01 04:39:20,997 19472 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.1:11222), adding to the pool.
2012-02-01 04:39:20,998 19473 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.3:11222), adding to the pool.
2012-02-01 04:39:20,999 19474 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.7:11222), adding to the pool.
2012-02-01 04:39:21,000 19475 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.5:11222), adding to the pool.
2012-02-01 04:45:26,202 384677 INFO [org.infinispan.client.hotrod.impl.protocol.Codec11] (Client-19:) ISPN004006: New topology: [/172.18.1.1:11222, /172.18.1.7:11222, /172.18.1.5:11222]
2012-02-01 04:45:26,206 384681 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.1:11222), adding to the pool.
2012-02-01 04:45:26,207 384682 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.7:11222), adding to the pool.
2012-02-01 04:45:26,208 384683 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.5:11222), adding to the pool.
it seems like they get new topology update but without newly started node02
