Details
-
Bug
-
Resolution: Done
-
Minor
-
7.2.0.Beta1, 7.1.1.Final
-
None
Description
Sometimes the ClientListenerNotifier keeps printing extremely high number of warning and quickly fill GBs of disk space:
11:28:16,805 WARN [ClientListenerNotifier] (Client-Listener-aacb322ffc60405e) ISPN004039: Unable to complete reading event from server null java.nio.channels.IllegalBlockingModeException at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:179) at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:282) at org.infinispan.client.hotrod^C.impl.protocol.Codec20.readEvent(Codec20.java:126) at org.infinispan.client.hotrod.event.ClientListenerNotifier$EventDispatcher.run(ClientListenerNotifier.java:237) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
This can be reproduced when running the server test suite, more specifically the test ExampleConfigsIT. Right after the server is started the above warning is logged in a loop for the duration of the tests.
I've started to observe this after commit (1), which changed the behaviour to retry on SocketTimeouts.
Before the aforementioned commit, log were printing:
10:49:56,361 ERROR [ClientListenerNotifier] (Client-Listener-b53963d5ab2b4234) ISPN004043: Unrecoverable error reading event from server null, exiting event reader thread org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:184) at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:282) at org.infinispan.client.hotrod.impl.protocol.Codec20.readEvent(Codec20.java:126) at org.infinispan.client.hotrod.event.ClientListenerNotifier$EventDispatcher.run(ClientListenerNotifier.java:236) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketTimeoutException at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:179) ... 8 more
And thus the listener thread was exiting,
(1) https://github.com/infinispan/infinispan/commit/1a03a0a84307ae8b95c06b398216eb36f6065f31
