Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-9759

Hot Rod server non-hash aware topology updates can include non-members

    XMLWordPrintable

Details

    • Bug
    • Resolution: Obsolete
    • Critical
    • None
    • 9.4.2.Final
    • Test Suite

    Description

      When sending a non-hash aware topology update, the server includes all the servers that are present in the topology cache. This includes both servers that don't have the cache running yet (if the cache was started dynamically) and servers that are shutting down or already shutdown (because a node doesn't remove itself from the address cache before shutting down).

      When a node shut down, the remaining nodes eventually see the view change and remove the stopped server from the address cache, but likely after sending a topology update with the new topology id to the clients. In cases where a rebalance is not necessary (e.g. replicated caches, or a single node is alive), a corrected topology update is never sent to the client.

      This is causing random failures in HotRod10ReplicationTest.testReplicatedPutWithTopologyChanges,HotRodReplicationTest.testReplicatedPutWithTopologyChanges and HotRod12ReplicationTest.testReplicatedPutWithTopologyChanges

      I saw it on a pull request build first, but I see it (and its subclasses) has been randomly failing in master as well. I am able to reliably reproduce the failure on my machine if I add a small delay in CrashedMemberDetectorListener.detectCrashedMember().

      16:24:18,288 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.server.hotrod.HotRod12ReplicationTest.testReplicatedPutWithTopologyChanges
      java.lang.AssertionError: expected:<3> but was:<2>
      	at org.testng.AssertJUnit.fail(AssertJUnit.java:59) ~[testng-6.9.9.jar:?]
      	at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364) ~[testng-6.9.9.jar:?]
      	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80) ~[testng-6.9.9.jar:?]
      	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245) ~[testng-6.9.9.jar:?]
      	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:252) ~[testng-6.9.9.jar:?]
      	at org.infinispan.server.hotrod.HotRodReplicationTest.testReplicatedPutWithTopologyChanges(HotRodReplicationTest.java:145) ~[test-classes/:?]
      

      https://ci.infinispan.org/job/Infinispan/job/master/878/

      Attachments

        Activity

          People

            Unassigned Unassigned
            dberinde@redhat.com Dan Berindei (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: