Loading...

Type: Bug
Resolution: Obsolete
Priority: Major
Fix Version/s: None
Affects Version/s: 9.2.1.Final
Component/s: Hot Rod
Labels:
- redhat-summit-18

Steps to Reproduce:
Hide

I was not able to reliably reproduce why the client listener is re-registered twice without debugger. I've needed to use the debugger and this helps to reliable reproduce it in my environment.

1) Unzip infinispan-server 9.2.1.Final and run with:

cd infinispan-server-9.2.1.Final/bin/ ./standalone.sh -c clustered.xml

2) Run this Java application from IDE in debug mode https://github.com/mposolda/misc/blob/master/ispn-client-listener/src/main/java/org/mposolda/ClientListenerFailoverTest.java . Make sure that infinispan version (Version of infinispan on client side) is 9.2.1.Final (property "ispn.version" in this file: https://github.com/mposolda/misc/blob/master/ispn-client-listener/pom.xml

After application is executed, you can see that items are added and client listener is triggered as expected:

May 30, 2018 9:53:12 AM org.mposolda.ClientListenerFailoverTest main INFO: RemoteCache created May 30, 2018 9:53:13 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-0 May 30, 2018 9:53:13 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-0 May 30, 2018 9:53:14 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-1 May 30, 2018 9:53:14 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-1 May 30, 2018 9:53:15 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-2 May 30, 2018 9:53:15 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-2 May 30, 2018 9:53:16 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-3 May 30, 2018 9:53:16 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-3 May 30, 2018 9:53:17 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-4 May 30, 2018 9:53:17 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-4 May 30, 2018 9:53:18 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-5 May 30, 2018 9:53:18 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-5 ....

3) Add breakpoint to class org.infinispan.client.hotrod.impl.operations.AddClientListenerOperation to the beginning of method "execute(Channel channel)"

4) Restart infinispan-server

5) After server is restarted, the method AddClientListenerOperation.execute is invoked. Wait with breakpoint at least 10 seconds to reproduce the issue.

6) Continue with breakpoint and see that method is executed another time. Continue with breakpoint again.

7) Now I can see that clientListener is executed twice for each added item, which is NOT ok.

.... INFO: Adding key: key-189 May 30, 2018 9:56:23 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-189 May 30, 2018 9:56:23 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-189 May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-190 May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-190 May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-190 May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-191 May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-191 May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-191 May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-192 May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-192 May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-192 ...

Sorry for the brittle steps to reproduce with the need of debugger, but that was the only way to simulate it reliably. Without debugger, the listener is usually re-registered once (correct behaviour), but sometimes twice (incorrect behaviour), so I think some issue is here.
Show
I was not able to reliably reproduce why the client listener is re-registered twice without debugger. I've needed to use the debugger and this helps to reliable reproduce it in my environment. 1) Unzip infinispan-server 9.2.1.Final and run with: cd infinispan-server-9.2.1.Final/bin/ ./standalone.sh -c clustered.xml 2) Run this Java application from IDE in debug mode https://github.com/mposolda/misc/blob/master/ispn-client-listener/src/main/java/org/mposolda/ClientListenerFailoverTest.java . Make sure that infinispan version (Version of infinispan on client side) is 9.2.1.Final (property "ispn.version" in this file: https://github.com/mposolda/misc/blob/master/ispn-client-listener/pom.xml After application is executed, you can see that items are added and client listener is triggered as expected: May 30, 2018 9:53:12 AM org.mposolda.ClientListenerFailoverTest main INFO: RemoteCache created May 30, 2018 9:53:13 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-0 May 30, 2018 9:53:13 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-0 May 30, 2018 9:53:14 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-1 May 30, 2018 9:53:14 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-1 May 30, 2018 9:53:15 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-2 May 30, 2018 9:53:15 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-2 May 30, 2018 9:53:16 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-3 May 30, 2018 9:53:16 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-3 May 30, 2018 9:53:17 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-4 May 30, 2018 9:53:17 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-4 May 30, 2018 9:53:18 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-5 May 30, 2018 9:53:18 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-5 .... 3) Add breakpoint to class org.infinispan.client.hotrod.impl.operations.AddClientListenerOperation to the beginning of method "execute(Channel channel)" 4) Restart infinispan-server 5) After server is restarted, the method AddClientListenerOperation.execute is invoked. Wait with breakpoint at least 10 seconds to reproduce the issue. 6) Continue with breakpoint and see that method is executed another time. Continue with breakpoint again. 7) Now I can see that clientListener is executed twice for each added item, which is NOT ok. .... INFO: Adding key: key-189 May 30, 2018 9:56:23 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-189 May 30, 2018 9:56:23 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-189 May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-190 May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-190 May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-190 May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-191 May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-191 May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-191 May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest main INFO: Adding key: key-192 May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-192 May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created INFO: Listener: created: key-192 ... Sorry for the brittle steps to reproduce with the need of debugger, but that was the only way to simulate it reliably. Without debugger, the listener is usually re-registered once (correct behaviour), but sometimes twice (incorrect behaviour), so I think some issue is here.

During test of summit, we saw some issue with client listeners. Setup was this:

Every site had one hotrod client (RHSSO), which used infinispan of version 8.2.8.Final and one infinispan-server, which I think was version 9.2.1.Final
Client (RH-SSO) had some client listener registered against remoteCache
After restart of infinispan-server, the RH-SSO was able to re-connect to it and CRUD the items to the remoteCache. But previously registered clientListeners were lost.

Note that setup demo used just 1 infinispan-server on every site, so there were not any topology changes (no possibility of re-balance client listener to other node).

I've tried to do simple reproducer for the scenario above and verified that with client 8.2.8.Final and server 9.2.1.Final, the client listener is not automatically re-registered after restart.

BUT after upgrade client to 9.2.1.Final, re-registration worked fine and client listener was re-registered automatically. This is important as from long term perspective, we are upgrading to infinispan 9 on RH-SSO side. But I had the opposite issue, that sometimes the client listener is registered twice. See steps to reproduce for the details.

Details

Description

Attachments

Activity

People

Dates