Details
-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
None
-
None
-
None
Description
There is problem with classloading when EAP 6.3.0.ER9 server is started by Arquillian. Problem was hit in following test scenario:
- Start two EAP 6.3.0.ER9 servers
- HornetQ is configured in collocated HA topology with shared store (EAP servers contains HornetQ backup for each other - so there are 2 live/backup pairs)
- Start producer which sends messages to "jms/queue/testQueue0" to first EAP server
- Start consumer which receives messages from "jms/queue/testQueue0" to first EAP server
- Kill -9 "first EAP server"
- Producer and consumer failovers to backup
Problem:
When producer and consumer failovers to backup then it fails with:
13:58:40,146 Thread-2 (HornetQ-client-global-threads-2060674296) ERROR [org.hornetq.core.client.impl.ClientSessionImpl:1198] HQ214003: Failed to handle failover HornetQException[errorType=SECURITY_EXCEPTION message=HQ119031: Unable to validate user: null] at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:394) at org.hornetq.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:1087) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:1075) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:717) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:587) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionException(ClientSessionFactoryImpl.java:441) at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$2.run(NettyConnector.java:924) at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722)
Second EAP server logs:
13:58:40,144 ERROR [org.hornetq.core.server] (Old I/O server worker (parentId: 223510377, [id: 0x0d527f69, /192.168.40.2:5446])) HQ224018: Failed to create session: HornetQException[errorType=SECURITY_EXCEPTION message=HQ119031: Unable to validate user: null] at org.hornetq.core.security.impl.SecurityStoreImpl.authenticate(SecurityStoreImpl.java:146) [hornetq-server-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.server.impl.HornetQServerImpl.createSession(HornetQServerImpl.java:979) [hornetq-server-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.protocol.core.impl.HornetQPacketHandler.handleCreateSession(HornetQPacketHandler.java:150) [hornetq-server-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.protocol.core.impl.HornetQPacketHandler.handlePacket(HornetQPacketHandler.java:78) [hornetq-server-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:636) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:564) [hornetq-server-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_15] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_15] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_15]
It failed because it was not authenticated on second EAP server. When debugging the problem I figured that root cause for this failure is that
javax.security.auth.login.LoginContext.invoke(String methodName):746 is using reflection to load class - org.jboss.as.security.RealmDirectLoginModule which was not found in provided classloader:
Class c = Class.forName(moduleStack[i].entry.getLoginModuleName(), true, contextClassLoader);
When I try the same test scenario manually then producer and consumer failovers without problem (=authenticates). Is there a way how EAP 6 classloader can be influenced by Arquillian?
