Uploaded image for project: 'JGroups'
  1. JGroups
  2. JGRP-343

Unable to bring additional node to a cluster due to WARN ClientGmsImpl:331 - there was more than 1 candidate for coord and WARN ClientGmsImpl:118 - join(10.20.111.1:1362) failed, retrying

    XMLWordPrintable

Details

    • Bug
    • Resolution: Can't Do
    • Major
    • None
    • 2.2.8
    • None

    Description

      We have an environment that consists of approx 20-30 nodes all running Hibernate 3.1.1->OSCache->Jgroups 2.2.8

      In some cases the state of the cluster becomes inconsitent due to multiple nodes being rebooted at random times and this results in some nodes thnking that Coordinator is NODE-X and other nodes thinking that Coordinator is NODE-Y. It is actually very easy to repsoduc it in this 20-node environment by rebooting the coordinator node.

      The confused nodes never recover from the wrong state even when NODE-X goes off-line. The only way to correct is to reboot all ~20 nodes which really sucks

      During the confused cluster state i am unable to bring additional nodes up - the JOIN error keeps scrolling and they never become part of the cluster:

      19:20:04,500 WARN ClientGmsImpl:118 - join(10.20.111.1:1362) failed, retrying
      ...

      With debug on we were able to determine that the reason for it is the following:

      16:51:56,203 WARN ClientGmsImpl:331 - there was more than 1 candidate for coord
      inator:

      {10.40.10.52:4775=1, 10.20.110.1:2715=1}

      This was when node 10.20.110.1 has been dead for at least an hour.

      The state of the cluster (each nodes' oppinion on who should be the coordinator is nicely reported here:

      16:52:06,375 DEBUG PingWaiter:95 - clearing old responses: [[own_addr=10.20.111.
      1:1117, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:651
      80, coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, c
      oord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.20.110.3:45282, coord_
      addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, coord_addr=1
      0.20.110.1:2715, is_server=true], [own_addr=10.40.10.51:3426, coord_addr=10.20.1
      10.1:2715, is_server=true], [own_addr=10.40.10.52:4775, coord_addr=10.20.110.1:2
      715, is_server=true], [own_addr=10.40.121.3:4249, coord_addr=10.20.110.1:2715, i
      s_server=true], [own_addr=10.40.125.3:1739, coord_addr=10.20.110.1:2715, is_serv
      er=true], [own_addr=10.40.126.2:4228, coord_addr=10.20.110.1:2715, is_server=tru
      e], [own_addr=10.40.124.1:64677, coord_addr=10.40.124.1:64677, is_server=false],
      [own_addr=10.40.124.2:31634, coord_addr=10.20.110.1:2715, is_server=true], [own
      _addr=10.40.121.2:1403, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=
      10.40.120.3:4838, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.
      126.3:4204, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.126.1:
      58260, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.125.2:3377,
      coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.124.4:1764, coord
      _addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.121.4:1832, coord_addr=
      10.20.110.1:2715, is_server=true], [own_addr=10.40.125.4:2799, coord_addr=10.20.
      110.1:2715, is_server=true], [own_addr=10.40.120.4:3696, coord_addr=10.20.110.1:
      2715, is_server=true], [own_addr=10.40.120.2:2013, coord_addr=10.20.110.1:2715,
      is_server=true], [own_addr=10.40.126.4:21141, coord_addr=10.20.110.1:2715, is_se
      rver=true], [own_addr=10.40.124.3:15988, coord_addr=10.20.110.1:2715, is_server=
      true], [own_addr=10.40.125.1:3612, coord_addr=10.20.110.1:2715, is_server=true],
      [own_addr=10.40.124.1:64720, coord_addr=10.40.124.1:64720, is_server=false], [o
      wn_addr=10.40.121.1:3810, coord_addr=10.20.110.1:2715, is_server=true]]

      Majority of the nodes think that our DEAD node should be the Coordinator.

      And finally here is the full debug trace from an attempt to bring a new node into the cluster:

      D:\sbroot\r6.1.0.0\Bin>startEmulator.bat
      16:51:49,578 DEBUG ClassConfigurator:97 - mapping is:
      1: class org.jgroups.stack.IpAddress
      2: class org.jgroups.protocols.CAUSAL$CausalHeader
      3: class org.jgroups.protocols.FD$FdHeader
      4: class org.jgroups.protocols.FD_PID$FdHeader
      5: class org.jgroups.protocols.FD_PROB$FdHeader
      6: class org.jgroups.protocols.FD_SOCK$FdHeader
      7: class org.jgroups.protocols.FragHeader
      8: class org.jgroups.protocols.MERGE$MergeHeader
      9: class org.jgroups.protocols.NakAckHeader
      10: class org.jgroups.protocols.PARTITIONER$PartitionerHeader
      11: class org.jgroups.protocols.PerfHeader
      12: class org.jgroups.protocols.PIGGYBACK$PiggybackHeader
      13: class org.jgroups.protocols.PingHeader
      14: class org.jgroups.protocols.TcpHeader
      15: class org.jgroups.protocols.TOTAL$Header
      16: class org.jgroups.protocols.TOTAL_OLD$TotalHeader
      17: class org.jgroups.protocols.TOTAL_TOKEN$TotalTokenHeader
      18: class org.jgroups.protocols.TOTAL_TOKEN$RingTokenHeader
      19: class org.jgroups.protocols.TunnelHeader
      20: class org.jgroups.protocols.UdpHeader
      21: class org.jgroups.protocols.UNICAST$UnicastHeader
      22: class org.jgroups.protocols.VERIFY_SUSPECT$VerifyHeader
      23: class org.jgroups.protocols.WANPIPE$WanPipeHeader
      24: class org.jgroups.protocols.pbcast.GMS$GmsHeader
      25: class org.jgroups.protocols.pbcast.NakAckHeader
      26: class org.jgroups.protocols.pbcast.PbcastHeader
      27: class org.jgroups.protocols.pbcast.STABLE$StableHeader
      28: class org.jgroups.protocols.pbcast.STATE_TRANSFER$StateHeader
      29: class org.jgroups.protocols.SMACK$SmackHeader
      30: class org.jgroups.Message
      31: class org.jgroups.View
      32: class org.jgroups.ViewId
      33: class org.jgroups.util.List
      34: interface org.jgroups.Address
      35: class [Lorg.jgroups.Address;
      36: class org.jgroups.protocols.PingRsp
      37: class [Ljava.lang.Object;
      38: class java.util.Vector
      39: class org.jgroups.protocols.pbcast.JoinRsp
      40: class org.jgroups.protocols.pbcast.Digest
      41: class java.util.Hashtable
      50: class org.jgroups.service.lease.LeaseRequestHeader
      51: class org.jgroups.service.lease.LeaseResponseHeader
      52: class org.jgroups.service.lease.DenyResponseHeader
      53: class org.jgroups.protocols.COMPRESS$CompressHeader
      54: class org.jgroups.protocols.FC$FcHeader
      55: class org.jgroups.protocols.WanPipeAddress

      16:51:49,765 DEBUG AUTOCONF:166 - frag_size=64000
      16:51:49,843 DEBUG GMS:93 - changed role to org.jgroups.protocols.pbcast.ClientG
      msImpl
      16:51:49,890 DEBUG UDP:359 - creating sockets and starting threads
      16:51:49,906 INFO UDP:1048 - sockets will use interface 10.20.111.1
      16:51:49,921 INFO UDP:1094 - socket information:
      local_addr=10.20.111.1:1172, mcast_addr=231.12.21.132:45566, bind_addr=/10.20.11
      1.1, ttl=32
      sock: bound to 10.20.111.1:1172, receive buffer size=64000, send buffer size=320
      00
      mcast_recv_sock: bound to 10.20.111.1:45566, send buffer size=150000, receive bu
      ffer size=80000
      mcast_send_sock: bound to 10.20.111.1:1173, send buffer size=150000, receive buf
      fer size=80000

      -------------------------------------------------------
      GMS: address is 10.20.111.1:1172
      -------------------------------------------------------
      16:51:50,015 DEBUG UDP:1347 - created unicast receiver thread
      16:51:50,062 DEBUG FD_SOCK:872 - waiting for client connections on 0.0.0.0/0.0.0
      .0:1174
      16:51:50,062 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=20
      00, got 0 rsps
      16:51:50,062 DEBUG PingSender:57 - sending GET_MBRS_REQ
      16:51:50,125 DEBUG UDP:803 - sending msg to 231.12.21.132:45566 (src=10.20.111.1
      :1172), headers are

      {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:channel_ name=OSCacheBus]}

      16:51:50,156 DEBUG UDP:820 - looped back local message [dst: 231.12.21.132:45566
      , src: 10.20.111.1:1172 (2 headers), size = 0 bytes]
      16:51:50,187 DEBUG PING:183 - discarded my own discovery request
      16:51:50,187 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1173 (s
      ize=51 bytes)
      16:51:50,187 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.1:1117
      16:51:50,218 DEBUG UDP:748 - discarded own loopback multicast packet
      16:51:50,234 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.1
      :1117 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1117, coord_addr=10 .20.111.1:1117, is_server=false]]}

      16:51:50,281 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.3:65180
      16:51:50,281 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.1:1
      117, coord_addr=10.20.111.1:1117, is_server=false]
      16:51:50,296 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.3
      :65180 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.3:65180, coord_addr= 10.40.10.52:4775, is_server=true]]}

      16:51:50,312 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=17
      50, got 1 rsps
      16:51:50,359 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
      size=420 bytes)
      16:51:50,359 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.3:6
      5180, coord_addr=10.40.10.52:4775, is_server=true]
      16:51:50,359 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.2:1620
      16:51:50,406 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      4.4:1764 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =537, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:50,421 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=16
      41, got 2 rsps
      16:51:50,437 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.2
      :1620 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.2:1620, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:50,484 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
      size=420 bytes)
      16:51:50,484 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:51:50,546 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.3:45282
      16:51:50,546 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.2:1
      620, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:50,562 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      4.4:1764 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =538, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:50,593 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.3
      :45282 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.3:45282, coord_addr= 10.40.10.52:4775, is_server=true]]}

      16:51:50,609 DEBUG PingWaiter:122 - initial mbrs are [[own_addr=10.20.111.1:1117
      , coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180, co
      ord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, coord_ad
      dr=10.20.110.1:2715, is_server=true]]
      16:51:50,640 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:51:50,687 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.2:56053
      16:51:50,687 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.3:4
      5282, coord_addr=10.40.10.52:4775, is_server=true]
      16:51:50,750 DEBUG ClientGmsImpl:66 - initial_mbrs are [[own_addr=10.20.111.1:11
      17, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180,
      coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, coord_
      addr=10.20.110.1:2715, is_server=true]]
      16:51:50,765 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.2
      :56053 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.2:56053, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:51:50,828 WARN ClientGmsImpl:331 - there was more than 1 candidate for coord
      inator:

      {10.40.10.52:4775=1, 10.20.110.1:2715=1}

      16:51:50,875 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.52:4775
      16:51:50,875 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.2:5
      6053, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:50,890 DEBUG ClientGmsImpl:113 - sending handleJoin(10.20.111.1:1172) to 1
      0.40.10.52:4775
      16:51:50,906 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
      :4775 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.52:4775, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:50,953 DEBUG UNICAST:222 - [10.20.111.1:1172] --> DATA(10.40.10.52:4775: #
      80, first=true)
      16:51:51,000 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.52:4
      775, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:51,000 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.51:3426
      16:51:51,015 DEBUG STABLE:324 - stable task started; num_gossip_runs=3, max_goss
      ip_runs=3
      16:51:51,046 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.51
      :3426 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.51:3426, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:51,062 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
      72), headers are

      {UNICAST=[UNICAST: DATA, seqno=80], GMS=GmsHeader[JOIN_REQ]: mb r=10.20.111.1:1172, UDP=[UDP:channel_name=OSCacheBus]}

      16:51:51,109 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.2:4228
      16:51:51,109 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.51:3
      426, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:51,156 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.2
      :4228 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.2:4228, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:51,234 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.3:4838
      16:51:51,234 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.2:4
      228, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:51,234 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.3
      :4838 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.3:4838, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:51,296 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.1:58260
      16:51:51,296 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.3:4
      838, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:51,312 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.1
      :58260 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.1:58260, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:51:51,390 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.4:3696
      16:51:51,390 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.1:5
      8260, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:51,390 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.4
      :3696 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.4:3696, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:51,453 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.1:3810
      16:51:51,453 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.4:3
      696, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:51,468 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.1
      :3810 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.1:3810, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:51,531 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.1:3
      810, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:51,531 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.3:4249
      16:51:51,578 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.3
      :4249 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.3:4249, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:51,609 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64677
      16:51:51,609 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.3:4
      249, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:51,625 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
      :64677 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64677, coord_addr= 10.40.124.1:64677, is_server=false]]}

      16:51:51,687 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.1:3612
      16:51:51,687 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
      4677, coord_addr=10.40.124.1:64677, is_server=false]
      16:51:51,703 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.1
      :3612 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.1:3612, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:51,765 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.1:3
      612, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:51,765 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.4:1764
      16:51:51,796 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.4
      :1764 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.4:1764, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:51,843 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.3:4204
      16:51:51,843 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.4:1
      764, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:51,859 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.3
      :4204 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.3:4204, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:51,921 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.2:1403
      16:51:51,921 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.3:4
      204, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:51,937 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.2
      :1403 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.2:1403, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:52,000 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.2:1
      403, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:52,000 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.4:1832
      16:51:52,031 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.4
      :1832 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.4:1832, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:52,062 DEBUG UDP:247 - received (mcast) 1098 bytes from /10.40.125.2:3378
      (size=1098 bytes)
      16:51:52,078 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.2:2013
      16:51:52,078 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.4:1
      832, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:52,093 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      5.2:3377 (3 headers), size = 0 bytes], headers are

      {STABLE=[STABLE_GOSSIP]: dige st is [10.20.110.1:2715: [0 : 595 (595)], 10.20.110.2:56053: [1 : 1490 (1490)], 10.20.110.3:45282: [0 : 630 (630)], 10.20.111.1:53038: [0 : 21 (21)], 10.20.111. 2:1620: [0 : 952 (952)], 10.20.111.3:65180: [0 : 379 (379)], 10.40.10.51:3426: [ 5 : 3906 (3906)], 10.40.10.52:4775: [16 : 3457 (3457)], 10.40.120.2:2013: [0 : 6 78 (678)], 10.40.120.3:4838: [0 : 674 (674)], 10.40.120.4:3696: [0 : 658 (658)], 10.40.121.1:3810: [0 : 884 (884)], 10.40.121.2:1403: [0 : 631 (631)], 10.40.121 .3:4249: [0 : 598 (598)], 10.40.121.4:1832: [0 : 773 (773)], 10.40.124.1:55644: [0 : 512 (512)], 10.40.124.2:31634: [0 : 652 (652)], 10.40.124.3:15988: [0 : 790 (790)], 10.40.124.4:1764: [0 : 538 (538)], 10.40.125.1:3612: [0 : 804 (804)], 1 0.40.125.2:3377: [0 : 539 (539)], 10.40.125.3:1739: [0 : 774 (774)], 10.40.125.4 :2799: [0 : 756 (756)], 10.40.126.1:58260: [0 : 52 (52)], 10.40.126.2:4228: [0 : 50 (50)], 10.40.126.3:4204: [0 : 51 (51)], 10.40.126.4:21141: [0 : 643 (643)]], NAKACK=[NAKACK: MSG, seqno=540, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:52,109 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.2
      :2013 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.2:2013, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:52,296 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:51:52,343 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.2:2
      013, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:52,343 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.3:15988
      16:51:52,390 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.3
      :15988 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.3:15988, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:51:52,437 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.3:1
      5988, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:52,437 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.4:2799
      16:51:52,468 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.4
      :2799 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.4:2799, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:52,515 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.2:3377
      16:51:52,515 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.4:2
      799, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:52,531 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.2
      :3377 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.2:3377, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:52,593 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.2:31634
      16:51:52,593 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.2:3
      377, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:52,609 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.2
      :31634 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.2:31634, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:51:52,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.4:21141
      16:51:52,671 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.2:3
      1634, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:52,687 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.4
      :21141 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.4:21141, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:51:52,750 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64720
      16:51:52,750 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.4:2
      1141, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:52,765 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
      :64720 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64720, coord_addr= 10.40.124.1:64720, is_server=false]]}

      16:51:52,828 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
      4720, coord_addr=10.40.124.1:64720, is_server=false]
      16:51:52,828 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.3:1739
      16:51:52,875 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.3
      :1739 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.3:1739, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:52,906 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.3:1
      739, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:52,906 DEBUG UDP:1542 - received (ucast) 62 bytes from /10.40.10.52:4775
      16:51:52,953 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
      :4775 (2 headers), size = 0 bytes], headers are

      {UNICAST=[UNICAST: DATA_ACK, seq no=80], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:52,984 DEBUG UDP:1542 - received (ucast) 82 bytes from /10.40.10.52:4775
      16:51:52,984 DEBUG UNICAST:387 - [10.20.111.1:1172] <-- ACK(10.40.10.52:4775: #8
      0)
      16:51:53,000 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
      :4775 (3 headers), size = 0 bytes], headers are

      {UNICAST=[UNICAST: DATA, seqno=6 1], GMS=GmsHeader[JOIN_RSP]: join_rsp=null, UDP=[UDP:channel_name=OSCacheBus]}

      16:51:53,046 DEBUG UNICAST:402 - [10.20.111.1:1172] --> ACK(10.40.10.52:4775: #6
      1)
      16:51:53,078 DEBUG UNICAST:344 - [10.20.111.1:1172] <-- DATA(10.40.10.52:4775: #
      61, first=true
      16:51:53,078 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
      72), headers are

      {UNICAST=[UNICAST: DATA_ACK, seqno=61], UDP=[UDP:channel_name=O SCacheBus]}

      16:51:53,093 WARN ClientGmsImpl:118 - join(10.20.111.1:1172) failed, retrying
      16:51:54,812 DEBUG UDP:247 - received (mcast) 51 bytes from /10.40.124.1:64678 (
      size=51 bytes)
      16:51:54,828 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      4.1:64677 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCac heBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}

      16:51:54,859 DEBUG PING:195 - received GET_MBRS_REQ from 10.40.124.1:64677, send
      ing response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_add
      r=10.20.111.1:1172, is_server=false]]
      16:51:54,890 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1118 (s
      ize=51 bytes)
      16:51:54,890 DEBUG UDP:803 - sending msg to 10.40.124.1:64677 (src=10.20.111.1:1
      172), headers are

      {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172 , coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheB us]}

      16:51:54,921 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.20.11
      1.1:1117 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCach eBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}

      16:51:55,000 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.2:1404 (
      size=420 bytes)
      16:51:55,000 DEBUG PING:195 - received GET_MBRS_REQ from 10.20.111.1:1117, sendi
      ng response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_addr
      =10.20.111.1:1172, is_server=false]]
      16:51:55,015 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      1.2:1403 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =632, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:55,046 DEBUG UDP:803 - sending msg to 10.20.111.1:1117 (src=10.20.111.1:11
      72), headers are

      {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheBu s]}

      16:51:55,078 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.2:1404 (
      size=420 bytes)
      16:51:55,078 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:51:55,140 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      1.2:1403 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =633, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:55,156 DEBUG PingWaiter:95 - clearing old responses: [[own_addr=10.20.111.
      1:1117, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:651
      80, coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, co
      ord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.20.110.3:45282, coord_a
      ddr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, coord_addr=1
      0.20.110.1:2715, is_server=true], [own_addr=10.40.10.52:4775, coord_addr=10.20.1
      10.1:2715, is_server=true], [own_addr=10.40.10.51:3426, coord_addr=10.20.110.1:2
      715, is_server=true], [own_addr=10.40.126.2:4228, coord_addr=10.20.110.1:2715, i
      s_server=true], [own_addr=10.40.120.3:4838, coord_addr=10.20.110.1:2715, is_serv
      er=true], [own_addr=10.40.126.1:58260, coord_addr=10.20.110.1:2715, is_server=tr
      ue], [own_addr=10.40.120.4:3696, coord_addr=10.20.110.1:2715, is_server=true], [
      own_addr=10.40.121.1:3810, coord_addr=10.20.110.1:2715, is_server=true], [own_ad
      dr=10.40.121.3:4249, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.
      40.124.1:64677, coord_addr=10.40.124.1:64677, is_server=false], [own_addr=10.40.
      125.1:3612, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.124.4:
      1764, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.126.3:4204,
      coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.121.2:1403, coord_
      addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.121.4:1832, coord_addr=1
      0.20.110.1:2715, is_server=true], [own_addr=10.40.120.2:2013, coord_addr=10.20.1
      10.1:2715, is_server=true], [own_addr=10.40.124.3:15988, coord_addr=10.20.110.1:
      2715, is_server=true], [own_addr=10.40.125.4:2799, coord_addr=10.20.110.1:2715,
      is_server=true], [own_addr=10.40.125.2:3377, coord_addr=10.20.110.1:2715, is_ser
      ver=true], [own_addr=10.40.124.2:31634, coord_addr=10.20.110.1:2715, is_server=t
      rue], [own_addr=10.40.126.4:21141, coord_addr=10.20.110.1:2715, is_server=true],
      [own_addr=10.40.124.1:64720, coord_addr=10.40.124.1:64720, is_server=false], [o
      wn_addr=10.40.125.3:1739, coord_addr=10.20.110.1:2715, is_server=true]]
      16:51:55,187 DEBUG UDP:247 - received (mcast) 51 bytes from /10.40.124.1:64721 (
      size=51 bytes)
      16:51:55,187 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:51:55,468 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=20
      00, got 0 rsps
      16:51:55,484 DEBUG PingSender:57 - sending GET_MBRS_REQ
      16:51:55,500 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      4.1:64720 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCac heBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}

      16:51:55,546 DEBUG UDP:803 - sending msg to 231.12.21.132:45566 (src=10.20.111.1
      :1172), headers are

      {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:channel_ name=OSCacheBus]}

      16:51:55,578 DEBUG PING:195 - received GET_MBRS_REQ from 10.40.124.1:64720, send
      ing response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_add
      r=10.20.111.1:1172, is_server=false]]
      16:51:55,609 DEBUG UDP:820 - looped back local message [dst: 231.12.21.132:45566
      , src: 10.20.111.1:1172 (2 headers), size = 0 bytes]
      16:51:55,671 DEBUG PING:183 - discarded my own discovery request
      16:51:55,671 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1173 (s
      ize=51 bytes)
      16:51:55,671 DEBUG UDP:803 - sending msg to 10.40.124.1:64720 (src=10.20.111.1:1
      172), headers are

      {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172 , coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheB us]}

      16:51:55,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.1:1117
      16:51:55,703 DEBUG UDP:748 - discarded own loopback multicast packet
      16:51:55,750 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.1
      :1117 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1117, coord_addr=10 .20.111.1:1117, is_server=false]]}

      16:51:55,812 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.1:1
      117, coord_addr=10.20.111.1:1117, is_server=false]
      16:51:55,812 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.3:65180
      16:51:55,828 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=16
      40, got 1 rsps
      16:51:55,843 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.3
      :65180 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.3:65180, coord_addr= 10.40.10.52:4775, is_server=true]]}

      16:51:55,906 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.3:6
      5180, coord_addr=10.40.10.52:4775, is_server=true]
      16:51:55,906 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.2:56053
      16:51:55,937 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=15
      31, got 2 rsps
      16:51:55,937 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.2
      :56053 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.2:56053, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:51:56,015 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.3:45282
      16:51:56,015 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.2:5
      6053, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:56,015 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.3
      :45282 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.3:45282, coord_addr= 10.40.10.52:4775, is_server=true]]}

      16:51:56,046 DEBUG PingWaiter:122 - initial mbrs are [[own_addr=10.20.111.1:1117
      , coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180, co
      ord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, coord_a
      ddr=10.20.110.1:2715, is_server=true]]
      16:51:56,078 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.2:1620
      16:51:56,078 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.3:4
      5282, coord_addr=10.40.10.52:4775, is_server=true]
      16:51:56,125 DEBUG ClientGmsImpl:66 - initial_mbrs are [[own_addr=10.20.111.1:11
      17, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180,
      coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, coord
      _addr=10.20.110.1:2715, is_server=true]]
      16:51:56,140 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.2
      :1620 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.2:1620, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:56,203 WARN ClientGmsImpl:331 - there was more than 1 candidate for coord
      inator:

      {10.40.10.52:4775=1, 10.20.110.1:2715=1}

      16:51:56,250 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.51:3426
      16:51:56,250 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.2:1
      620, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:56,281 DEBUG ClientGmsImpl:113 - sending handleJoin(10.20.111.1:1172) to 1
      0.40.10.52:4775
      16:51:56,281 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.51
      :3426 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.51:3426, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:56,328 DEBUG UNICAST:222 - [10.20.111.1:1172] --> DATA(10.40.10.52:4775: #
      81, first=false)
      16:51:56,375 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64677
      16:51:56,375 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.51:3
      426, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:56,406 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
      72), headers are

      {UNICAST=[UNICAST: DATA, seqno=81], GMS=GmsHeader[JOIN_REQ]: mb r=10.20.111.1:1172, UDP=[UDP:channel_name=OSCacheBus]}

      16:51:56,406 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
      :64677 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64677, coord_addr= 10.40.124.1:64677, is_server=false]]}

      16:51:56,515 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.1:3810
      16:51:56,515 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
      4677, coord_addr=10.40.124.1:64677, is_server=false]
      16:51:56,515 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.1
      :3810 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.1:3810, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:56,593 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.1:3
      810, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:56,593 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.52:4775
      16:51:56,625 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
      :4775 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.52:4775, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:56,671 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.52:4
      775, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:56,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.3:15988
      16:51:56,703 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.3
      :15988 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.3:15988, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:51:56,750 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.2:4228
      16:51:56,750 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.3:1
      5988, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:56,765 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.2
      :4228 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.2:4228, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:56,765 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
      size=420 bytes)
      16:51:56,828 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.3:4838
      16:51:56,828 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.2:4
      228, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:56,843 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      4.4:1764 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =539, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:56,859 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.3
      :4838 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.3:4838, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:56,921 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
      size=420 bytes)
      16:51:56,921 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:51:56,953 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.4:2799
      16:51:56,953 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.3:4
      838, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:56,984 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      4.4:1764 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =540, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:57,000 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.4
      :2799 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.4:2799, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:57,062 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:51:57,109 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.1:58260
      16:51:57,109 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.4:2
      799, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:57,125 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.1
      :58260 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.1:58260, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:51:57,203 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.1:5
      8260, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:57,203 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.4:1832
      16:51:57,234 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.4
      :1832 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.4:1832, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:57,281 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.2:1403
      16:51:57,281 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.4:1
      832, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:57,296 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.2
      :1403 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.2:1403, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:57,359 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.2:1
      403, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:57,359 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.3:4204
      16:51:57,390 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.3
      :4204 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.3:4204, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:57,437 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.4:1764
      16:51:57,437 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.3:4
      204, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:57,453 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.4
      :1764 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.4:1764, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:57,515 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.3:1739
      16:51:57,515 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.4:1
      764, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:57,531 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.3
      :1739 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.3:1739, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:57,593 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.3:1
      739, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:57,593 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.3:4249
      16:51:57,625 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.3
      :4249 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.3:4249, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:57,671 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.3:4
      249, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:57,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.2:31634
      16:51:57,703 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.2
      :31634 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.2:31634, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:51:57,750 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.2:3
      1634, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:57,750 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.2:3377
      16:51:57,781 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.2
      :3377 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.2:3377, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:57,828 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.4:3696
      16:51:57,828 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.2:3
      377, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:57,843 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.4
      :3696 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.4:3696, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:57,906 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.4:3
      696, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:57,906 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.1:3612
      16:51:57,937 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.1
      :3612 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.1:3612, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:57,984 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.2:2013
      16:51:57,984 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.1:3
      612, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:58,000 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.2
      :2013 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.2:2013, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:51:58,046 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
      size=420 bytes)
      16:51:58,062 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.4:21141
      16:51:58,062 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.2:2
      013, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:58,078 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      4.4:1764 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =541, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:58,093 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.4
      :21141 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.4:21141, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:51:58,156 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
      size=420 bytes)
      16:51:58,156 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:51:58,187 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.4:2
      1141, coord_addr=10.20.110.1:2715, is_server=true]
      16:51:58,187 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64720
      16:51:58,218 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      4.4:1764 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =542, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:58,265 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
      :64720 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64720, coord_addr= 10.40.124.1:64720, is_server=false]]}

      16:51:58,296 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:51:58,343 DEBUG UDP:1542 - received (ucast) 62 bytes from /10.40.10.52:4775
      16:51:58,343 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
      4720, coord_addr=10.40.124.1:64720, is_server=false]
      16:51:58,375 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
      :4775 (2 headers), size = 0 bytes], headers are

      {UNICAST=[UNICAST: DATA_ACK, seq no=81], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:58,406 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.2:1404 (
      size=420 bytes)
      16:51:58,421 DEBUG UDP:1542 - received (ucast) 82 bytes from /10.40.10.52:4775
      16:51:58,421 DEBUG UNICAST:387 - [10.20.111.1:1172] <-- ACK(10.40.10.52:4775: #8
      1)
      16:51:58,453 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      1.2:1403 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =634, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:58,453 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
      :4775 (3 headers), size = 0 bytes], headers are

      {UNICAST=[UNICAST: DATA, seqno=6 2], GMS=GmsHeader[JOIN_RSP]: join_rsp=null, UDP=[UDP:channel_name=OSCacheBus]}

      16:51:58,515 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.2:1404 (
      size=420 bytes)
      16:51:58,515 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:51:58,578 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      1.2:1403 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =635, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:58,578 DEBUG UNICAST:402 - [10.20.111.1:1172] --> ACK(10.40.10.52:4775: #6
      2)
      16:51:58,625 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
      size=420 bytes)
      16:51:58,625 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:51:58,640 DEBUG UNICAST:344 - [10.20.111.1:1172] <-- DATA(10.40.10.52:4775: #
      62, first=false
      16:51:58,640 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
      72), headers are

      {UNICAST=[UNICAST: DATA_ACK, seqno=62], UDP=[UDP:channel_name=O SCacheBus]}

      16:51:58,656 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      1.3:4249 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =599, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:58,703 WARN ClientGmsImpl:118 - join(10.20.111.1:1172) failed, retrying
      16:51:58,765 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
      size=420 bytes)
      16:51:58,765 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:51:58,796 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      1.3:4249 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =600, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:58,843 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
      size=420 bytes)
      16:51:58,843 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:51:58,875 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      4.4:1764 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =543, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:58,906 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
      size=420 bytes)
      16:51:58,906 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:51:58,937 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      4.4:1764 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =544, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:51:58,984 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:52:00,734 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.125.1:3613 (
      size=420 bytes)
      16:52:00,750 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      5.1:3612 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =805, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:52:00,781 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.125.1:3613 (
      size=420 bytes)
      16:52:00,781 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:52:00,796 DEBUG PingWaiter:95 - clearing old responses: [[own_addr=10.20.111.
      1:1117, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:651
      80, coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, c
      oord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.20.110.3:45282, coord_
      addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, coord_addr=1
      0.20.110.1:2715, is_server=true], [own_addr=10.40.10.51:3426, coord_addr=10.20.1
      10.1:2715, is_server=true], [own_addr=10.40.124.1:64677, coord_addr=10.40.124.1:
      64677, is_server=false], [own_addr=10.40.121.1:3810, coord_addr=10.20.110.1:2715
      , is_server=true], [own_addr=10.40.10.52:4775, coord_addr=10.20.110.1:2715, is_s
      erver=true], [own_addr=10.40.124.3:15988, coord_addr=10.20.110.1:2715, is_server
      =true], [own_addr=10.40.126.2:4228, coord_addr=10.20.110.1:2715, is_server=true]
      , [own_addr=10.40.120.3:4838, coord_addr=10.20.110.1:2715, is_server=true], [own
      _addr=10.40.125.4:2799, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=
      10.40.126.1:58260, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40
      .121.4:1832, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.121.2
      :1403, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.126.3:4204,
      coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.124.4:1764, coord
      _addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.125.3:1739, coord_addr=
      10.20.110.1:2715, is_server=true], [own_addr=10.40.121.3:4249, coord_addr=10.20.
      110.1:2715, is_server=true], [own_addr=10.40.124.2:31634, coord_addr=10.20.110.1
      :2715, is_server=true], [own_addr=10.40.125.2:3377, coord_addr=10.20.110.1:2715,
      is_server=true], [own_addr=10.40.120.4:3696, coord_addr=10.20.110.1:2715, is_se
      rver=true], [own_addr=10.40.125.1:3612, coord_addr=10.20.110.1:2715, is_server=t
      rue], [own_addr=10.40.120.2:2013, coord_addr=10.20.110.1:2715, is_server=true],
      [own_addr=10.40.126.4:21141, coord_addr=10.20.110.1:2715, is_server=true], [own_
      addr=10.40.124.1:64720, coord_addr=10.40.124.1:64720, is_server=false]]
      16:52:00,812 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      5.1:3612 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =806, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:52:01,093 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=20
      00, got 0 rsps
      16:52:01,093 DEBUG PingSender:57 - sending GET_MBRS_REQ
      16:52:01,140 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:52:01,171 DEBUG UDP:803 - sending msg to 231.12.21.132:45566 (src=10.20.111.1
      :1172), headers are

      {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:channel_ name=OSCacheBus]}

      16:52:01,218 DEBUG UDP:820 - looped back local message [dst: 231.12.21.132:45566
      , src: 10.20.111.1:1172 (2 headers), size = 0 bytes]
      16:52:01,234 DEBUG PING:183 - discarded my own discovery request
      16:52:01,234 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1173 (s
      ize=51 bytes)
      16:52:01,234 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.1:1117
      16:52:01,265 DEBUG UDP:748 - discarded own loopback multicast packet
      16:52:01,281 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.1
      :1117 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1117, coord_addr=10 .20.111.1:1117, is_server=false]]}

      16:52:01,343 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.3:65180
      16:52:01,343 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.1:1
      117, coord_addr=10.20.111.1:1117, is_server=false]
      16:52:01,359 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.3
      :65180 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.3:65180, coord_addr= 10.40.10.52:4775, is_server=true]]}

      16:52:01,375 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=17
      18, got 1 rsps
      16:52:01,421 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.2:56053
      16:52:01,421 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.3:6
      5180, coord_addr=10.40.10.52:4775, is_server=true]
      16:52:01,453 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.2
      :56053 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.2:56053, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:52:01,484 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=16
      09, got 2 rsps
      16:52:01,531 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.2:5
      6053, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:01,531 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.3:45282
      16:52:01,562 DEBUG PingWaiter:122 - initial mbrs are [[own_addr=10.20.111.1:1117
      , coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180, co
      ord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, coord_a
      ddr=10.20.110.1:2715, is_server=true]]
      16:52:01,578 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.3
      :45282 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.3:45282, coord_addr= 10.40.10.52:4775, is_server=true]]}

      16:52:01,625 DEBUG ClientGmsImpl:66 - initial_mbrs are [[own_addr=10.20.111.1:11
      17, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180,
      coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, coord
      _addr=10.20.110.1:2715, is_server=true]]
      16:52:01,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.2:1620
      16:52:01,671 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.3:4
      5282, coord_addr=10.40.10.52:4775, is_server=true]
      16:52:01,718 WARN ClientGmsImpl:331 - there was more than 1 candidate for coord
      inator:

      {10.40.10.52:4775=1, 10.20.110.1:2715=1}

      16:52:01,718 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.2
      :1620 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.2:1620, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:01,765 DEBUG ClientGmsImpl:113 - sending handleJoin(10.20.111.1:1172) to 1
      0.40.10.52:4775
      16:52:01,812 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.51:3426
      16:52:01,812 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.2:1
      620, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:01,812 DEBUG UDP:247 - received (mcast) 51 bytes from /10.40.124.1:64678 (
      size=51 bytes)
      16:52:01,828 DEBUG UNICAST:222 - [10.20.111.1:1172] --> DATA(10.40.10.52:4775: #
      82, first=false)
      16:52:01,843 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.51
      :3426 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.51:3426, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:01,890 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      4.1:64677 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCac heBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}

      16:52:01,906 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
      72), headers are

      {UNICAST=[UNICAST: DATA, seqno=82], GMS=GmsHeader[JOIN_REQ]: mb r=10.20.111.1:1172, UDP=[UDP:channel_name=OSCacheBus]}

      16:52:01,953 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.52:4775
      16:52:01,953 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.51:3
      426, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:01,984 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1118 (s
      ize=51 bytes)
      16:52:02,031 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
      :4775 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.52:4775, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:02,062 DEBUG PING:195 - received GET_MBRS_REQ from 10.40.124.1:64677, send
      ing response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_add
      r=10.20.111.1:1172, is_server=false]]
      16:52:02,078 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.20.11
      1.1:1117 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCach eBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}

      16:52:02,125 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.3:4249
      16:52:02,156 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.52:4
      775, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:02,156 DEBUG UDP:803 - sending msg to 10.40.124.1:64677 (src=10.20.111.1:1
      172), headers are

      {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172 , coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheB us]}

      16:52:02,187 DEBUG UDP:247 - received (mcast) 51 bytes from /10.40.124.1:64721 (
      size=51 bytes)
      16:52:02,203 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.3
      :4249 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.3:4249, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:02,234 DEBUG PING:195 - received GET_MBRS_REQ from 10.20.111.1:1117, sendi
      ng response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_addr
      =10.20.111.1:1172, is_server=false]]
      16:52:02,296 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      4.1:64720 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCac heBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}

      16:52:02,343 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.3:1739
      16:52:02,375 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.3:4
      249, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:02,375 DEBUG UDP:803 - sending msg to 10.20.111.1:1117 (src=10.20.111.1:11
      72), headers are

      {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheBu s]}

      16:52:02,421 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.3
      :1739 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.3:1739, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:02,437 DEBUG PING:195 - received GET_MBRS_REQ from 10.40.124.1:64720, send
      ing response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_add
      r=10.20.111.1:1172, is_server=false]]
      16:52:02,531 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.2:4228
      16:52:02,562 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.3:1
      739, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:02,562 DEBUG UDP:803 - sending msg to 10.40.124.1:64720 (src=10.20.111.1:1
      172), headers are

      {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172 , coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheB us]}

      16:52:02,578 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.2
      :4228 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.2:4228, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:02,687 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.2:4
      228, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:02,687 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64677
      16:52:02,718 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
      :64677 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64677, coord_addr= 10.40.124.1:64677, is_server=false]]}

      16:52:02,765 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.2:31634
      16:52:02,765 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
      4677, coord_addr=10.40.124.1:64677, is_server=false]
      16:52:02,781 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.2
      :31634 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.2:31634, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:52:02,843 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.2:3
      1634, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:02,843 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.2:1403
      16:52:02,875 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.2
      :1403 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.2:1403, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:02,906 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
      size=420 bytes)
      16:52:02,921 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.3:4838
      16:52:02,921 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.2:1
      403, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:02,937 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      1.3:4249 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =601, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:52:02,953 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.3
      :4838 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.3:4838, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:03,015 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
      size=420 bytes)
      16:52:03,015 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:52:03,046 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.3:4204
      16:52:03,046 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.3:4
      838, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:03,078 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      1.3:4249 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =602, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:52:03,093 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.3
      :4204 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.3:4204, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:03,156 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:52:03,203 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.1:58260
      16:52:03,203 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.3:4
      204, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:03,218 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.1
      :58260 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.1:58260, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:52:03,296 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.1:5
      8260, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:03,296 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.2:3377
      16:52:03,328 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.2
      :3377 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.2:3377, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:03,375 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.4:1764
      16:52:03,375 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.2:3
      377, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:03,390 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.4
      :1764 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.4:1764, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:03,453 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.4:1832
      16:52:03,453 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.4:1
      764, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:03,468 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.4
      :1832 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.4:1832, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:03,531 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.4:1
      832, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:03,531 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.4:2799
      16:52:03,562 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.4
      :2799 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.4:2799, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:03,609 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.4:2
      799, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:03,609 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.4:3696
      16:52:03,640 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.4
      :3696 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.4:3696, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:03,687 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.4:3
      696, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:03,687 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.2:2013
      16:52:03,718 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.2
      :2013 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.2:2013, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:03,765 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.4:21141
      16:52:03,765 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.2:2
      013, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:03,781 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.4
      :21141 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.4:21141, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:52:03,843 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.4:2
      1141, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:03,843 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.3:15988
      16:52:03,875 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.3
      :15988 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.3:15988, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:52:03,921 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.1:3612
      16:52:03,921 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.3:1
      5988, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:03,937 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.1
      :3612 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.1:3612, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:04,000 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.1:3
      612, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:04,000 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64720
      16:52:04,031 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
      :64720 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64720, coord_addr= 10.40.124.1:64720, is_server=false]]}

      16:52:04,078 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.1:3810
      16:52:04,078 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
      4720, coord_addr=10.40.124.1:64720, is_server=false]
      16:52:04,093 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.1
      :3810 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.1:3810, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:04,156 DEBUG UDP:1542 - received (ucast) 62 bytes from /10.40.10.52:4775
      16:52:04,156 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.1:3
      810, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:04,171 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
      :4775 (2 headers), size = 0 bytes], headers are

      {UNICAST=[UNICAST: DATA_ACK, seq no=82], UDP=[UDP:channel_name=OSCacheBus]}

      16:52:04,218 DEBUG UDP:1542 - received (ucast) 82 bytes from /10.40.10.52:4775
      16:52:04,218 DEBUG UNICAST:387 - [10.20.111.1:1172] <-- ACK(10.40.10.52:4775: #8
      2)
      16:52:04,234 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
      :4775 (3 headers), size = 0 bytes], headers are

      {UNICAST=[UNICAST: DATA, seqno=6 3], GMS=GmsHeader[JOIN_RSP]: join_rsp=null, UDP=[UDP:channel_name=OSCacheBus]}

      16:52:04,281 DEBUG UNICAST:402 - [10.20.111.1:1172] --> ACK(10.40.10.52:4775: #6
      3)
      16:52:04,312 DEBUG UNICAST:344 - [10.20.111.1:1172] <-- DATA(10.40.10.52:4775: #
      63, first=false
      16:52:04,312 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
      72), headers are

      {UNICAST=[UNICAST: DATA_ACK, seqno=63], UDP=[UDP:channel_name=O SCacheBus]}

      16:52:04,328 WARN ClientGmsImpl:118 - join(10.20.111.1:1172) failed, retrying
      16:52:05,312 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
      size=420 bytes)
      16:52:05,343 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      1.3:4249 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =603, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:52:05,375 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
      size=420 bytes)
      16:52:05,375 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:52:05,390 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
      1.3:4249 (2 headers), size = 355 bytes], headers are

      {NAKACK=[NAKACK: MSG, seqno =604, range=null], UDP=[UDP:channel_name=OSCacheBus]}

      16:52:05,437 DEBUG NAKACK:340 - message was discarded (not yet server)
      16:52:06,375 DEBUG PingWaiter:95 - clearing old responses: [[own_addr=10.20.111.
      1:1117, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:651
      80, coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, c
      oord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.20.110.3:45282, coord_
      addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, coord_addr=1
      0.20.110.1:2715, is_server=true], [own_addr=10.40.10.51:3426, coord_addr=10.20.1
      10.1:2715, is_server=true], [own_addr=10.40.10.52:4775, coord_addr=10.20.110.1:2
      715, is_server=true], [own_addr=10.40.121.3:4249, coord_addr=10.20.110.1:2715, i
      s_server=true], [own_addr=10.40.125.3:1739, coord_addr=10.20.110.1:2715, is_serv
      er=true], [own_addr=10.40.126.2:4228, coord_addr=10.20.110.1:2715, is_server=tru
      e], [own_addr=10.40.124.1:64677, coord_addr=10.40.124.1:64677, is_server=false],
      [own_addr=10.40.124.2:31634, coord_addr=10.20.110.1:2715, is_server=true], [own
      _addr=10.40.121.2:1403, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=
      10.40.120.3:4838, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.
      126.3:4204, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.126.1:
      58260, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.125.2:3377,
      coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.124.4:1764, coord
      _addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.121.4:1832, coord_addr=
      10.20.110.1:2715, is_server=true], [own_addr=10.40.125.4:2799, coord_addr=10.20.
      110.1:2715, is_server=true], [own_addr=10.40.120.4:3696, coord_addr=10.20.110.1:
      2715, is_server=true], [own_addr=10.40.120.2:2013, coord_addr=10.20.110.1:2715,
      is_server=true], [own_addr=10.40.126.4:21141, coord_addr=10.20.110.1:2715, is_se
      rver=true], [own_addr=10.40.124.3:15988, coord_addr=10.20.110.1:2715, is_server=
      true], [own_addr=10.40.125.1:3612, coord_addr=10.20.110.1:2715, is_server=true],
      [own_addr=10.40.124.1:64720, coord_addr=10.40.124.1:64720, is_server=false], [o
      wn_addr=10.40.121.1:3810, coord_addr=10.20.110.1:2715, is_server=true]]
      16:52:06,640 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=20
      00, got 0 rsps
      16:52:06,640 DEBUG PingSender:57 - sending GET_MBRS_REQ
      16:52:06,671 DEBUG UDP:803 - sending msg to 231.12.21.132:45566 (src=10.20.111.1
      :1172), headers are

      {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:channel_ name=OSCacheBus]}

      16:52:06,703 DEBUG UDP:820 - looped back local message [dst: 231.12.21.132:45566
      , src: 10.20.111.1:1172 (2 headers), size = 0 bytes]
      16:52:06,734 DEBUG PING:183 - discarded my own discovery request
      16:52:06,734 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1173 (s
      ize=51 bytes)
      16:52:06,734 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.1:1117
      16:52:06,765 DEBUG UDP:748 - discarded own loopback multicast packet
      16:52:06,781 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.1
      :1117 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1117, coord_addr=10 .20.111.1:1117, is_server=false]]}

      16:52:06,828 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.3:65180
      16:52:06,828 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.1:1
      117, coord_addr=10.20.111.1:1117, is_server=false]
      16:52:06,843 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.3
      :65180 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.3:65180, coord_addr= 10.40.10.52:4775, is_server=true]]}

      16:52:06,859 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=17
      81, got 1 rsps
      16:52:06,906 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.3:45282
      16:52:06,906 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.3:6
      5180, coord_addr=10.40.10.52:4775, is_server=true]
      16:52:06,937 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.3
      :45282 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.3:45282, coord_addr= 10.40.10.52:4775, is_server=true]]}

      16:52:06,968 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=16
      72, got 2 rsps
      16:52:07,000 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.2:1620
      16:52:07,000 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.3:4
      5282, coord_addr=10.40.10.52:4775, is_server=true]
      16:52:07,031 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.2
      :1620 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.2:1620, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:07,062 DEBUG PingWaiter:122 - initial mbrs are [[own_addr=10.20.111.1:1117
      , coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180, co
      ord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.3:45282, coord_a
      ddr=10.40.10.52:4775, is_server=true]]
      16:52:07,109 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.52:4775
      16:52:07,109 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.2:1
      620, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:07,156 DEBUG ClientGmsImpl:66 - initial_mbrs are [[own_addr=10.20.111.1:11
      17, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180,
      coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.3:45282, coord
      _addr=10.40.10.52:4775, is_server=true]]
      16:52:07,156 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
      :4775 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.52:4775, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:07,234 DEBUG ClientGmsImpl:113 - sending handleJoin(10.20.111.1:1172) to 1
      0.40.10.52:4775
      16:52:07,265 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.3:1739
      16:52:07,265 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.52:4
      775, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:07,296 DEBUG UNICAST:222 - [10.20.111.1:1172] --> DATA(10.40.10.52:4775: #
      83, first=false)
      16:52:07,312 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.3
      :1739 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.3:1739, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:07,359 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
      72), headers are

      {UNICAST=[UNICAST: DATA, seqno=83], GMS=GmsHeader[JOIN_REQ]: mb r=10.20.111.1:1172, UDP=[UDP:channel_name=OSCacheBus]}

      16:52:07,406 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.3:4249
      16:52:07,406 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.3:1
      739, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:07,453 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.3
      :4249 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.3:4249, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:07,515 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.51:3426
      16:52:07,515 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.3:4
      249, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:07,531 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.51
      :3426 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.51:3426, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:07,593 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.51:3
      426, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:07,593 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.2:4228
      16:52:07,625 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.2
      :4228 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.2:4228, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:07,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.1:58260
      16:52:07,671 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.2:4
      228, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:07,687 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.1
      :58260 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheB us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.1:58260, coord_addr= 10.20.110.1:2715, is_server=true]]}

      16:52:07,750 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.4:2799
      16:52:07,750 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.1:5
      8260, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:07,765 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.4
      :2799 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.4:2799, coord_addr=10 .20.110.1:2715, is_server=true]]}

      16:52:07,828 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.4:2
      799, coord_addr=10.20.110.1:2715, is_server=true]
      16:52:07,828 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.1:3810
      16:52:07,859 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.1
      :3810 (2 headers), size = 0 bytes], headers are

      {UDP=[UDP:channel_name=OSCacheBu s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.1:3810, coord_addr=10 .20.110.1:2715, is_server=true]]}

      Terminate batch job (Y/N)? y

      D:\sbroot\r6.1.0.0\Bin>

      Attachments

        Activity

          People

            rhn-engineering-bban Bela Ban
            maciek.kolesnik Maciek Kolesnik (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: