Details
-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
None
-
None
-
None
Description
We are working in an application that uses WebSockets on WildFly. Basically application sends notification to connected clients through WebSockets. From time to time we found threads get stuck running trying to send a message. We tried configuring Undertow write timeout but did not help. Threads will never ever recover after that happens.
"asyncSendExecutor-47" - Thread t@379 java.lang.Thread.State: RUNNABLE at org.xnio.channels.Channels.writeBlocking(Channels.java:97) at io.undertow.websockets.jsr.WebSocketSessionRemoteEndpoint$BasicWebSocketSessionRemoteEndpoint.sendText(WebSocketSessionRemoteEndpoint.java:283) at org.springframework.web.socket.adapter.standard.StandardWebSocketSession.sendTextMessage(StandardWebSocketSession.java:197) at org.springframework.web.socket.adapter.AbstractWebSocketSession.sendMessage(AbstractWebSocketSession.java:104) at org.springframework.web.socket.sockjs.transport.session.WebSocketServerSockJsSession.writeFrameInternal(WebSocketServerSockJsSession.java:222) at org.springframework.web.socket.sockjs.transport.session.AbstractSockJsSession.writeFrame(AbstractSockJsSession.java:322) at org.springframework.web.socket.sockjs.transport.session.WebSocketServerSockJsSession.sendMessageInternal(WebSocketServerSockJsSession.java:212) at org.springframework.web.socket.sockjs.transport.session.AbstractSockJsSession.sendMessage(AbstractSockJsSession.java:166) at com.optima.midtier.services.content.AbstractWebSocketResponseCallback.sendPending(AbstractWebSocketResponseCallback.java:142) at com.optima.midtier.services.content.AbstractWebSocketResponseCallback.access$200(AbstractWebSocketResponseCallback.java:33) at com.optima.midtier.services.content.AbstractWebSocketResponseCallback$SendTask.run(AbstractWebSocketResponseCallback.java:312) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
VisualVM showed that thread gets stuck in while loop of writeBlocking:
public static <C extends WritableByteChannel & SuspendableWriteChannel> int writeBlocking(C channel, ByteBuffer buffer) throws IOException { int t = 0; while (buffer.hasRemaining()) { final int res = channel.write(buffer); if (res == 0) { channel.awaitWritable(); } else { t += res; } } return t; }
We were able to reproduce issue on Wildfly 8.2 to 10.1. Unfortunately we could reproduce only on production environment but never on development so we can not provide a case to reproduce issue.
Also those threads uses high amount of CPU. You can easily determine a server has the issue just checking CPU on machine is always above 100%.
Our guess is SocketChannel is still open but writing is not enabled. So write returns 0 and awaitWritable returns immediately, but is just a guess.
