-
Bug
-
Resolution: Done
-
Blocker
-
7.1.0.CR3
-
None
-
Regression, Blocks Testing
-
-
-
-
-
-
-
We started seeing intermittent "connection refused" on the client side in failover tests, when the client is invoking a cluster and some of the servers go down (but at least one server is always running).
Similar issue also arises when the application gets undeployed on one server (in which case the invocation fails with NoSuchEJBException even though the bean is still available on other servers).
I noticed that the same can be reproduced even without failover having to occur. The scenario is that a client has two PROVIDER_URLs specified, but only one server is running. The client will start correctly invoking on the running server, but intermittently, it will fail with "connection refused" because the client tried connecting to the stopped server, which failed, and didn't retry on the running server. The reproducer I'm attaching shows this scenario. The issue arises quite sparsely with this reproducer, but for some reason, in our automated failover tests, it happens much more often, in most runs of the test suite, we see at least one test failure due to this. It also looks like it happens a bit less often when running over IPv6, but it happens too.
Customer impact: EJB invocations can intermittently fail even when they should succeed.
Stack trace when this happens looks like this:
Exception in thread "main" org.jboss.ejb.client.RequestSendFailedException: java.net.ConnectException: Connection refused
at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.handleFailed(RemoteEJBReceiver.java:101)
at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.handleFailed(RemoteEJBReceiver.java:74)
at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:215)
at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:926)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.xnio.nio.WorkerThread$ConnectHandle.handleReady(WorkerThread.java:327)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
at ...asynchronous invocation...(Unknown Source)
at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:570)
at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:536)
at org.jboss.remoting3.ConnectionInfo$None.getConnection(ConnectionInfo.java:83)
at org.jboss.remoting3.ConnectionInfo.getConnection(ConnectionInfo.java:56)
at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:487)
at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:433)
at org.jboss.remoting3.UncloseableEndpoint.getConnectedIdentity(UncloseableEndpoint.java:51)
at org.jboss.remoting3.Endpoint.getConnectedIdentity(Endpoint.java:122)
at org.jboss.ejb.protocol.remote.RemoteEJBReceiver.lambda$getConnection$2(RemoteEJBReceiver.java:185)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.ejb.protocol.remote.RemoteEJBReceiver.getConnection(RemoteEJBReceiver.java:185)
at org.jboss.ejb.protocol.remote.RemoteEJBReceiver.processInvocation(RemoteEJBReceiver.java:128)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:454)
at org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocation(RemotingEJBClientInterceptor.java:51)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocation(TransactionPostDiscoveryInterceptor.java:79)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:90)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:66)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:165)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:466)
at org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:302)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:161)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:100)
at com.sun.proxy.$Proxy2.hello(Unknown Source)
at client.Client.main(Client.java:26)
- incorporates
-
EJBCLIENT-290 Introduce org.jboss.ejb.client.max-retries system property
-
- Resolved
-
-
-
- Resolved
-
-
-
- Resolved
-
- is incorporated by
-
JBEAP-13630 Upgrade ejb-client to 4.0.8.Final
-
- Verified
-
- is related to
-
-
- Closed
-
