$ java -Dprogram.name=run.sh -server -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.lang.ClassLoader.allowArraySyntax=true -Djava.net.preferIPv4Stack=true -Djboss.shared.directory.nodeA=/mnt/jbm/hudson/shared-san-nodeA -Djboss.shared.directory.nodeB=/mnt/jbm/hudson/shared-san-nodeB -jar /tmp/hudson//jboss-eap-5.1/jboss-as/bin/run.jar -c production -b messaging-21 -Djboss.partition.name=DisconnectJournal -Dsf.pid=54ItZdWMoLNkMSYG 06:48:07,089 INFO [ServerImpl] Starting JBoss (Microcontainer)... 06:48:07,089 INFO [ServerImpl] Release ID: JBoss [EAP] 5.1.1 (build: SVNTag=JBPAPP_5_1_1 date=201105171607) 06:48:07,089 INFO [ServerImpl] Bootstrap URL: null 06:48:07,089 INFO [ServerImpl] Home Dir: /tmp/hudson/jboss-eap-5.1/jboss-as 06:48:07,090 INFO [ServerImpl] Home URL: file:/tmp/hudson/jboss-eap-5.1/jboss-as/ 06:48:07,090 INFO [ServerImpl] Library URL: file:/tmp/hudson/jboss-eap-5.1/jboss-as/lib/ 06:48:07,091 INFO [ServerImpl] Patch URL: null 06:48:07,091 INFO [ServerImpl] Common Base URL: file:/tmp/hudson/jboss-eap-5.1/jboss-as/common/ 06:48:07,091 INFO [ServerImpl] Common Library URL: file:/tmp/hudson/jboss-eap-5.1/jboss-as/common/lib/ 06:48:07,091 INFO [ServerImpl] Server Name: production 06:48:07,091 INFO [ServerImpl] Server Base Dir: /tmp/hudson/jboss-eap-5.1/jboss-as/server 06:48:07,091 INFO [ServerImpl] Server Base URL: file:/tmp/hudson/jboss-eap-5.1/jboss-as/server/ 06:48:07,091 INFO [ServerImpl] Server Config URL: file:/tmp/hudson/jboss-eap-5.1/jboss-as/server/production/conf/ 06:48:07,091 INFO [ServerImpl] Server Home Dir: /tmp/hudson/jboss-eap-5.1/jboss-as/server/production 06:48:07,091 INFO [ServerImpl] Server Home URL: file:/tmp/hudson/jboss-eap-5.1/jboss-as/server/production/ 06:48:07,091 INFO [ServerImpl] Server Data Dir: /tmp/hudson/jboss-eap-5.1/jboss-as/server/production/data 06:48:07,091 INFO [ServerImpl] Server Library URL: file:/tmp/hudson/jboss-eap-5.1/jboss-as/server/production/lib/ 06:48:07,091 INFO [ServerImpl] Server Log Dir: /tmp/hudson/jboss-eap-5.1/jboss-as/server/production/log 06:48:07,091 INFO [ServerImpl] Server Native Dir: /tmp/hudson/jboss-eap-5.1/jboss-as/server/production/tmp/native 06:48:07,091 INFO [ServerImpl] Server Temp Dir: /tmp/hudson/jboss-eap-5.1/jboss-as/server/production/tmp 06:48:07,092 INFO [ServerImpl] Server Temp Deploy Dir: /tmp/hudson/jboss-eap-5.1/jboss-as/server/production/tmp/deploy 06:48:07,737 INFO [ServerImpl] Starting Microcontainer, bootstrapURL=file:/tmp/hudson/jboss-eap-5.1/jboss-as/server/production/conf/bootstrap.xml 06:48:08,312 INFO [VFSCacheFactory] Initializing VFSCache [org.jboss.virtual.plugins.cache.CombinedVFSCache] 06:48:08,315 INFO [VFSCacheFactory] Using VFSCache [CombinedVFSCache[real-cache: null]] 06:48:08,565 INFO [CopyMechanism] VFS temp dir: /tmp/hudson/jboss-eap-5.1/jboss-as/server/production/tmp 06:48:08,566 INFO [ZipEntryContext] VFS force nested jars copy-mode is enabled. 06:48:09,443 INFO [ServerInfo] Java version: 1.6.0_24,Sun Microsystems Inc. 06:48:09,443 INFO [ServerInfo] Java Runtime: Java(TM) SE Runtime Environment (build 1.6.0_24-b07) 06:48:09,444 INFO [ServerInfo] Java VM: Java HotSpot(TM) 64-Bit Server VM 19.1-b02,Sun Microsystems Inc. 06:48:09,444 INFO [ServerInfo] OS-System: Linux 2.6.32-71.14.1.el6.x86_64,amd64 06:48:09,444 INFO [ServerInfo] VM arguments: -Dprogram.name=run.sh -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.lang.ClassLoader.allowArraySyntax=true -Djava.net.preferIPv4Stack=true -Djboss.shared.directory.nodeA=/mnt/jbm/hudson/shared-san-nodeA -Djboss.shared.directory.nodeB=/mnt/jbm/hudson/shared-san-nodeB 06:48:09,474 INFO [JMXKernel] Legacy JMX core initialized 06:48:12,976 INFO [WebService] Using RMI server codebase: http://messaging-21:8083/ 06:48:16,291 INFO [NativeServerConfig] JBoss Web Services - Stack Native Core 06:48:16,291 INFO [NativeServerConfig] 3.1.2.SP10 06:48:17,022 INFO [LogNotificationListener] Adding notification listener for logging mbean "jboss.system:service=Logging,type=Log4jService" to server org.jboss.mx.server.MBeanServerImpl@76d78df0[ defaultDomain='jboss' ] 06:48:23,349 WARN [config] jacorb.home unset! Will use '.' 06:48:23,350 WARN [config] File ./jacorb.properties for configuration jacorb not found 06:48:23,400 WARN [config] jacorb.home unset! Will use '.' 06:48:23,400 WARN [config] File ./jacorb.properties for configuration jacorb not found 06:48:23,804 INFO [CorbaNamingService] CORBA Naming Started 06:48:24,988 INFO [MailService] Mail Service bound to java:/Mail 06:48:25,145 INFO [SnmpAgentService] SNMP agent going active 06:48:26,829 INFO [DisconnectJournal] Initializing partition DisconnectJournal 06:48:26,864 INFO [STDOUT] --------------------------------------------------------- GMS: address is 10.16.100.46:55200 (cluster=DisconnectJournal) --------------------------------------------------------- 06:48:26,937 INFO [PlatformMBeanServerRegistration] JBossCache MBeans were successfully registered to the platform mbean server. 06:48:26,980 INFO [STDOUT] --------------------------------------------------------- GMS: address is 10.16.100.46:55200 (cluster=DisconnectJournal-HAPartitionCache) --------------------------------------------------------- 06:48:27,994 INFO [DisconnectJournal] Number of cluster members: 2 06:48:27,994 INFO [DisconnectJournal] Other members: 1 06:48:28,064 INFO [RPCManagerImpl] Received new cluster view: [10.16.100.44:55200|1] [10.16.100.44:55200, 10.16.100.46:55200] 06:48:28,111 INFO [LegacyStateTransferIntegrator] Using version 4096 06:48:28,122 INFO [RPCManagerImpl] Cache local address is 10.16.100.46:55200 06:48:28,124 INFO [RPCManagerImpl] state was retrieved successfully (in 1.14 seconds) 06:48:28,137 INFO [ComponentRegistry] JBoss Cache version: JBossCache 'Malagueta' 3.2.7.GA 06:48:28,140 INFO [DisconnectJournal] Fetching serviceState (will wait for 30000 milliseconds): 06:48:28,219 INFO [DisconnectJournal] serviceState was retrieved successfully (in 79 milliseconds) 06:48:28,407 INFO [HANamingService] Started HAJNDI bootstrap; jnpPort=1100, backlog=50, bindAddress=messaging-21/10.16.100.46 06:48:28,413 INFO [DetachedHANamingService$AutomaticDiscovery] Listening on /10.16.100.46:1102, group=230.0.0.4, HA-JNDI address=10.16.100.46:1100 06:48:28,561 WARNING [FileConfigurationParser] AIO wasn't located on this platform, it will fall back to using pure Java NIO. If your platform is Linux, install LibAIO to enable the AIO journal 06:48:28,661 WARNING [FileConfigurationParser] AIO wasn't located on this platform, it will fall back to using pure Java NIO. If your platform is Linux, install LibAIO to enable the AIO journal 06:48:29,303 INFO [UnifiedInvokerHA] Service name is jboss:service=invoker,type=unifiedha 06:48:30,047 INFO [HornetQServerImpl] live server is starting with configuration HornetQ Configuration (clustered=true,backup=false,sharedStore=true,journalDirectory=/mnt/jbm/hudson/shared-san-nodeB/hornetq/journal,bindingsDirectory=/mnt/jbm/hudson/shared-san-nodeB/hornetq/bindings,largeMessagesDirectory=/mnt/jbm/hudson/shared-san-nodeB/hornetq/largemessages,pagingDirectory=/mnt/jbm/hudson/shared-san-nodeB/hornetq/paging) 06:48:30,048 INFO [HornetQServerImpl] Waiting to obtain live lock 06:48:30,112 INFO [JournalStorageManager] Using NIO Journal 06:48:30,169 WARNING [HornetQServerImpl] Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this. 06:48:30,314 INFO [FileLockNodeManager] Waiting to obtain live lock 06:48:30,316 INFO [FileLockNodeManager] Live Server Obtained live lock 06:48:32,700 INFO [NettyAcceptor] Started Netty Acceptor version 3.2.3.Final-r1 messaging-21:5445 for CORE protocol 06:48:32,702 INFO [NettyAcceptor] Started Netty Acceptor version 3.2.3.Final-r1 messaging-21:5455 for CORE protocol 06:48:32,734 INFO [HornetQServerImpl] Server is now live 06:48:32,735 INFO [HornetQServerImpl] HornetQ Server version 2.2.5.GA (HQ_2_2_5_GA_EAP, 121) [3fd25546-df88-11e0-8fc9-001b217d6d88] started 06:48:32,985 INFO [HornetQServerImpl] backup server is starting with configuration HornetQ Configuration (clustered=true,backup=true,sharedStore=true,journalDirectory=/mnt/jbm/hudson/shared-san-nodeA/hornetq/journal,bindingsDirectory=/mnt/jbm/hudson/shared-san-nodeA/hornetq/bindings,largeMessagesDirectory=/mnt/jbm/hudson/shared-san-nodeA/hornetq/largemessages,pagingDirectory=/mnt/jbm/hudson/shared-san-nodeA/hornetq/paging) 06:48:32,987 INFO [FileLockNodeManager] Waiting to become backup node 06:48:32,989 INFO [FileLockNodeManager] ** got backup lock 06:48:32,990 INFO [JournalStorageManager] Using NIO Journal 06:48:32,991 WARNING [HornetQServerImpl] Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this. 06:48:33,097 INFO [ClusterManagerImpl] announcing backup 06:48:33,125 INFO [HornetQServerImpl] HornetQ Backup Server version 2.2.5.GA (HQ_2_2_5_GA_EAP, 121) [3f2b8dba-df88-11e0-ab4f-001b217d6d91] started, waiting live to fail before it gets active 06:48:33,178 INFO [TransactionManagerService] JBossTS Transaction Service (JTA version - tag:JBOSSTS_4_6_1_GA_CP11) - JBoss Inc. 06:48:33,178 INFO [TransactionManagerService] Setting up property manager MBean and JMX layer 06:48:33,300 INFO [TransactionManagerService] Initializing recovery manager 06:48:33,379 INFO [TransactionManagerService] Recovery manager configured 06:48:33,380 INFO [TransactionManagerService] Binding TransactionManager JNDI Reference 06:48:33,409 INFO [TransactionManagerService] Starting transaction recovery manager 06:48:33,701 INFO [AprLifecycleListener] The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /qa/tools/opt/x86_64/jdk1.6.0_24/jre/lib/amd64/server:/qa/tools/opt/x86_64/jdk1.6.0_24/jre/lib/amd64:/qa/tools/opt/x86_64/jdk1.6.0_24/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 06:48:33,745 INFO [Http11Protocol] Initializing Coyote HTTP/1.1 on http-messaging-21%2F10.16.100.46-8080 06:48:33,746 INFO [AjpProtocol] Initializing Coyote AJP/1.3 on ajp-messaging-21%2F10.16.100.46-8009 06:48:33,760 INFO [StandardService] Starting service jboss.web 06:48:33,762 INFO [StandardEngine] Starting Servlet Engine: JBoss Web/2.1.11.GA 06:48:33,797 INFO [Catalina] Server startup in 51 ms 06:48:33,812 INFO [TomcatDeployment] deploy, ctxPath=/web-console 06:48:34,359 INFO [TomcatDeployment] deploy, ctxPath=/juddi 06:48:34,390 INFO [RegistryServlet] Loading jUDDI configuration. 06:48:34,390 INFO [RegistryServlet] Resources loaded from: /WEB-INF/juddi.properties 06:48:34,390 INFO [RegistryServlet] Initializing jUDDI components. 06:48:34,498 INFO [TomcatDeployment] deploy, ctxPath=/jbossws 06:48:34,518 INFO [TomcatDeployment] deploy, ctxPath=/invoker 06:48:34,778 INFO [RARDeployment] Required license terms exist, view vfsfile:/tmp/hudson/jboss-eap-5.1/jboss-as/server/production/deploy/jboss-local-jdbc.rar/META-INF/ra.xml 06:48:34,788 INFO [RARDeployment] Required license terms exist, view vfsfile:/tmp/hudson/jboss-eap-5.1/jboss-as/server/production/deploy/jboss-xa-jdbc.rar/META-INF/ra.xml 06:48:34,794 INFO [RARDeployment] Required license terms exist, view vfsfile:/tmp/hudson/jboss-eap-5.1/jboss-as/server/production/deploy/jms-ra.rar/META-INF/ra.xml 06:48:34,811 INFO [HornetQResourceAdapter] HornetQ resource adaptor started 06:48:34,817 INFO [RARDeployment] Required license terms exist, view vfsfile:/tmp/hudson/jboss-eap-5.1/jboss-as/server/production/deploy/mail-ra.rar/META-INF/ra.xml 06:48:34,831 INFO [RARDeployment] Required license terms exist, view vfsfile:/tmp/hudson/jboss-eap-5.1/jboss-as/server/production/deploy/quartz-ra.rar/META-INF/ra.xml 06:48:34,895 INFO [SimpleThreadPool] Job execution threads will use class loader of thread: main 06:48:34,917 INFO [QuartzScheduler] Quartz Scheduler v.1.5.2 created. 06:48:34,919 INFO [RAMJobStore] RAMJobStore initialized. 06:48:34,920 INFO [StdSchedulerFactory] Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource file in Quartz package: 'quartz.properties' 06:48:34,920 INFO [StdSchedulerFactory] Quartz scheduler version: 1.5.2 06:48:34,921 INFO [QuartzScheduler] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started. 06:48:35,208 INFO [ConnectionFactoryBindingService] Bound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=DefaultDS' to JNDI name 'java:DefaultDS' 06:48:35,429 INFO [ConnectionFactoryBindingService] Bound ConnectionManager 'jboss.jca:service=ConnectionFactoryBinding,name=JmsXA' to JNDI name 'java:JmsXA' 06:48:35,520 INFO [TomcatDeployment] deploy, ctxPath=/admin-console 06:48:35,579 INFO [config] Initializing Mojarra (1.2_13-b01-FCS) for context '/admin-console' 06:48:36,518 INFO [ClusterManagerImpl] backup announced 06:48:36,870 INFO [BridgeImpl] Connecting bridge sf.failover-cluster.3f2b8dba-df88-11e0-ab4f-001b217d6d91 to its destination [3fd25546-df88-11e0-8fc9-001b217d6d88] 06:48:36,957 INFO [BridgeImpl] Bridge sf.failover-cluster.3f2b8dba-df88-11e0-ab4f-001b217d6d91 is connected [3fd25546-df88-11e0-8fc9-001b217d6d88-> sf.failover-cluster.3f2b8dba-df88-11e0-ab4f-001b217d6d91] 06:48:37,323 INFO [TomcatDeployment] deploy, ctxPath=/ 06:48:37,362 INFO [TomcatDeployment] deploy, ctxPath=/jmx-console 06:48:37,382 INFO [HornetQJMSRealDeployer] Deploying unit AbstractVFSDeploymentContext@290850743{vfsfile:/tmp/hudson/jboss-eap-5.1/jboss-as/server/production/deploy/destinations-hornetq-jms.xml} with config org.hornetq.jms.server.config.impl.JMSConfigurationImpl@229e76ae 06:48:37,420 INFO [HornetQServerImpl] trying to deploy queue jms.queue.OutQueue 06:48:37,451 INFO [HornetQServerImpl] trying to deploy queue jms.queue.TradeBrokerQueue 06:48:37,462 INFO [HornetQServerImpl] trying to deploy queue jms.topic.TradeStreamerTopic 06:48:37,476 INFO [HornetQServerImpl] trying to deploy queue jms.queue.InQueue 06:48:37,486 INFO [HornetQJMSRealDeployer] Deploying unit AbstractVFSDeploymentContext@64855122{vfsfile:/tmp/hudson/jboss-eap-5.1/jboss-as/server/production/deploy/hornetq/hornetq-jms.xml} with config org.hornetq.jms.server.config.impl.JMSConfigurationImpl@2be4d228 06:48:37,529 INFO [HornetQServerImpl] trying to deploy queue jms.queue.ExpiryQueue 06:48:37,560 INFO [HornetQServerImpl] trying to deploy queue jms.queue.DLQ 06:48:37,780 INFO [ProfileServiceBootstrap] Loading profile: ProfileKey@3f6a5d72[domain=default, server=default, name=production] 06:48:37,789 INFO [Http11Protocol] Starting Coyote HTTP/1.1 on http-messaging-21%2F10.16.100.46-8080 06:48:37,803 INFO [AjpProtocol] Starting Coyote AJP/1.3 on ajp-messaging-21%2F10.16.100.46-8009 06:48:37,809 INFO [ServerImpl] JBoss (Microcontainer) [5.1.1 (build: SVNTag=JBPAPP_5_1_1 date=201105171607)] Started in 30s:714ms 06:49:58,451 WARN [BridgeImpl] sf.failover-cluster.3f2b8dba-df88-11e0-ab4f-001b217d6d91::Connection failed before reconnect HornetQException[errorCode=2 message=Channel disconnected] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:362) at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:687) at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 06:49:59,936 INFO [DisconnectJournal] Suspected member: 10.16.100.44:55200 06:50:00,062 INFO [DisconnectJournal] New cluster view for partition DisconnectJournal (id: 2, delta: -1) : [10.16.100.46:1099] 06:50:00,061 INFO [RPCManagerImpl] Received new cluster view: [10.16.100.46:55200|2] [10.16.100.46:55200] 06:50:00,068 INFO [DisconnectJournal] I am (10.16.100.46:1099) received membershipChanged event: 06:50:00,068 INFO [DisconnectJournal] Dead members: 1 ([10.16.100.44:1099]) 06:50:00,068 INFO [DisconnectJournal] New Members : 0 ([]) 06:50:00,068 INFO [DisconnectJournal] All Members : 1 ([10.16.100.46:1099]) 06:50:02,540 WARN [JournalImpl] Uncommitted transaction with id 225 found and discarded 06:50:03,042 INFO [NettyAcceptor] Started Netty Acceptor version 3.2.3.Final-r1 messaging-21:5465 for CORE protocol 06:50:03,044 WARN [ClientSessionFactoryImpl] Failed to connect to server. 06:50:03,247 INFO [HornetQServerImpl] Backup Server is now live 06:50:03,252 INFO [BridgeImpl] Connecting bridge sf.failover-cluster.3fd25546-df88-11e0-8fc9-001b217d6d88 to its destination [3f2b8dba-df88-11e0-ab4f-001b217d6d91] 06:50:03,261 WARN [BridgeImpl] sf.failover-cluster.3f2b8dba-df88-11e0-ab4f-001b217d6d91::Connection failed with failedOver=true HornetQException[errorCode=2 message=Channel disconnected] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:362) at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:687) at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 06:50:03,266 INFO [BridgeImpl] Bridge sf.failover-cluster.3fd25546-df88-11e0-8fc9-001b217d6d88 is connected [3f2b8dba-df88-11e0-ab4f-001b217d6d91-> sf.failover-cluster.3fd25546-df88-11e0-8fc9-001b217d6d88] 06:50:03,273 WARN [ChannelImpl] Can't find packet to clear: last received command id 3 first stored command id 3 06:50:03,273 WARN [ChannelImpl] Can't find packet to clear: last received command id 4 first stored command id 4 06:50:03,274 WARN [ChannelImpl] Can't find packet to clear: last received command id 5 first stored command id 5 06:50:03,276 WARN [ChannelImpl] Can't find packet to clear: last received command id 6 first stored command id 6 06:50:04,246 WARN [DiscoveryGroupImpl] There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=3f2b8dba-df88-11e0-ab4f-001b217d6d91 06:51:03,064 INFO [DisconnectJournal] New cluster view for partition DisconnectJournal (id: 3, delta: 1) : [10.16.100.46:1099, 10.16.100.44:1099] 06:51:03,066 INFO [DisconnectJournal] I am (10.16.100.46:1099) received membershipChanged event: 06:51:03,066 INFO [DisconnectJournal] Dead members: 0 ([]) 06:51:03,066 INFO [DisconnectJournal] New Members : 1 ([10.16.100.44:1099]) 06:51:03,066 INFO [DisconnectJournal] All Members : 2 ([10.16.100.46:1099, 10.16.100.44:1099]) 06:51:03,176 INFO [RPCManagerImpl] Received new cluster view: [10.16.100.46:55200|3] [10.16.100.46:55200, 10.16.100.44:55200] 06:51:10,335 INFO [HornetQServerImpl] live server wants to restart, restarting server in backup 06:51:10,355 INFO [BridgeImpl] Bridge sf.failover-cluster.3fd25546-df88-11e0-8fc9-001b217d6d88 being stopped 06:51:10,358 INFO [BridgeImpl] stopped bridge sf.failover-cluster.3fd25546-df88-11e0-8fc9-001b217d6d88 06:51:10,374 WARN [RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4] 06:51:10,374 WARN [RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4] 06:51:10,374 WARN [RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4] 06:51:10,375 WARN [BridgeImpl] sf.failover-cluster.3f2b8dba-df88-11e0-ab4f-001b217d6d91::Connection failed before reconnect HornetQException[errorCode=4 message=The connection was disconnected because of server shutdown] at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1278) at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 06:51:10,384 WARN [ManagementServiceImpl] On ManagementService stop, there are 1 unexpected registered MBeans: [jms.server] 06:51:10,408 INFO [HornetQServerImpl] HornetQ Server version 2.2.5.GA (HQ_2_2_5_GA_EAP, 121) [3f2b8dba-df88-11e0-ab4f-001b217d6d91] stopped 06:51:10,408 INFO [HornetQServerImpl] backup server is starting with configuration HornetQ Configuration (clustered=true,backup=true,sharedStore=true,journalDirectory=/mnt/jbm/hudson/shared-san-nodeA/hornetq/journal,bindingsDirectory=/mnt/jbm/hudson/shared-san-nodeA/hornetq/bindings,largeMessagesDirectory=/mnt/jbm/hudson/shared-san-nodeA/hornetq/largemessages,pagingDirectory=/mnt/jbm/hudson/shared-san-nodeA/hornetq/paging) 06:51:10,408 INFO [FileLockNodeManager] Waiting to become backup node 06:51:10,409 INFO [FileLockNodeManager] ** got backup lock 06:51:10,410 INFO [JournalStorageManager] Using NIO Journal 06:51:10,410 WARNING [HornetQServerImpl] Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this. 06:51:10,428 INFO [ClusterManagerImpl] announcing backup 06:51:10,429 INFO [HornetQServerImpl] HornetQ Backup Server version 2.2.5.GA (HQ_2_2_5_GA_EAP, 121) [3f2b8dba-df88-11e0-ab4f-001b217d6d91] started, waiting live to fail before it gets active 06:51:12,721 INFO [ClusterManagerImpl] backup announced 06:51:13,473 WARN [BridgeImpl] sf.failover-cluster.3f2b8dba-df88-11e0-ab4f-001b217d6d91::Connection failed with failedOver=true HornetQException[errorCode=4 message=The connection was disconnected because of server shutdown] at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1278) at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 06:51:13,502 WARN [ChannelImpl] Can't find packet to clear: last received command id 3 first stored command id 3 06:51:13,512 WARN [ChannelImpl] Can't find packet to clear: last received command id 4 first stored command id 4 06:51:13,512 WARN [ChannelImpl] Can't find packet to clear: last received command id 5 first stored command id 5 06:51:13,516 WARN [ChannelImpl] Can't find packet to clear: last received command id 6 first stored command id 6 06:51:18,336 WARN [DiscoveryGroupImpl] There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=3f2b8dba-df88-11e0-ab4f-001b217d6d91 06:51:38,715 WARN [RemotingConnectionImpl] Connection failure has been detected: Did not receive data from /10.16.100.44:53825. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3] 06:51:38,717 WARN [ServerSessionImpl] Client connection failed, clearing up resources for session 43bd23d1-df88-11e0-ab4f-001b217d6d91 06:51:38,717 WARN [ServerSessionImpl] Cleared up resources for session 43bd23d1-df88-11e0-ab4f-001b217d6d91 06:58:16,563 WARN [BridgeImpl] Unable to send message, will try again once bridge reconnects HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 71] at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:276) at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:287) at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:142) at org.hornetq.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:407) at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2017) at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1587) at org.hornetq.core.server.impl.QueueImpl.doPoll(QueueImpl.java:1472) at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:72) at org.hornetq.core.server.impl.QueueImpl$ConcurrentPoller.run(QueueImpl.java:2299) at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 06:59:13,405 WARN [RemotingConnectionImpl] Connection failure has been detected: Did not receive data from server for org.hornetq.core.remoting.impl.netty.NettyConnection@5df56405[local= /10.16.100.46:55277, remote=messaging-20/10.16.100.44:5445] [code=3] 06:59:13,405 WARN [BridgeImpl] sf.failover-cluster.3f2b8dba-df88-11e0-ab4f-001b217d6d91::Connection failed before reconnect HornetQException[errorCode=3 message=Did not receive data from server for org.hornetq.core.remoting.impl.netty.NettyConnection@5df56405[local= /10.16.100.46:55277, remote=messaging-20/10.16.100.44:5445]] at org.hornetq.core.client.impl.ClientSessionFactoryImpl$PingRunnable.run(ClientSessionFactoryImpl.java:1382) at org.hornetq.core.client.impl.ClientSessionFactoryImpl$ActualScheduledPinger.run(ClientSessionFactoryImpl.java:1353) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 06:59:43,409 ERROR [ClientSessionImpl] Failed to handle failover HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 32] at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:276) at org.hornetq.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:921) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:872) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:587) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:481) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:77) at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1334) at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528) at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298) at org.hornetq.core.client.impl.ClientSessionFactoryImpl$PingRunnable$1.run(ClientSessionFactoryImpl.java:1392) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 06:59:43,409 WARN [BridgeImpl] sf.failover-cluster.3f2b8dba-df88-11e0-ab4f-001b217d6d91::Connection failed with failedOver=true HornetQException[errorCode=3 message=Did not receive data from server for org.hornetq.core.remoting.impl.netty.NettyConnection@5df56405[local= /10.16.100.46:55277, remote=messaging-20/10.16.100.44:5445]] at org.hornetq.core.client.impl.ClientSessionFactoryImpl$PingRunnable.run(ClientSessionFactoryImpl.java:1382) at org.hornetq.core.client.impl.ClientSessionFactoryImpl$ActualScheduledPinger.run(ClientSessionFactoryImpl.java:1353) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 07:00:13,072 ERROR [BridgeImpl] Failed to cancel refs HornetQException[errorCode=6 message=Input/output error] at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:286) at org.hornetq.core.client.impl.ClientSessionImpl.internalCreateQueue(ClientSessionImpl.java:1756) at org.hornetq.core.client.impl.ClientSessionImpl.createQueue(ClientSessionImpl.java:304) at org.hornetq.core.client.impl.DelegatingSession.createQueue(DelegatingSession.java:284) at org.hornetq.core.server.cluster.impl.ClusterConnectionBridge.setupNotificationConsumer(ClusterConnectionBridge.java:198) at org.hornetq.core.server.cluster.impl.ClusterConnectionBridge.afterConnect(ClusterConnectionBridge.java:223) at org.hornetq.core.server.cluster.impl.BridgeImpl.fail(BridgeImpl.java:486) at org.hornetq.core.server.cluster.impl.BridgeImpl.connectionFailed(BridgeImpl.java:427) at org.hornetq.core.server.cluster.impl.ClusterConnectionBridge.connectionFailed(ClusterConnectionBridge.java:249) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:817) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:611) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:481) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:77) at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1334) at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528) at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298) at org.hornetq.core.client.impl.ClientSessionFactoryImpl$PingRunnable$1.run(ClientSessionFactoryImpl.java:1392) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 07:00:16,050 INFO [JournalStorageManager] Deleting unreferenced message id=2147485151 from the journal 07:00:16,105 INFO [NettyAcceptor] Started Netty Acceptor version 3.2.3.Final-r1 messaging-21:5465 for CORE protocol 07:00:16,110 INFO [HornetQServerImpl] Backup Server is now live 07:00:16,463 WARN [DiscoveryGroupImpl] There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=3f2b8dba-df88-11e0-ab4f-001b217d6d91 07:00:17,746 INFO [BridgeImpl] Connecting bridge sf.failover-cluster.3fd25546-df88-11e0-8fc9-001b217d6d88 to its destination [3f2b8dba-df88-11e0-ab4f-001b217d6d91] 07:00:17,759 INFO [BridgeImpl] Bridge sf.failover-cluster.3fd25546-df88-11e0-8fc9-001b217d6d88 is connected [3f2b8dba-df88-11e0-ab4f-001b217d6d91-> sf.failover-cluster.3fd25546-df88-11e0-8fc9-001b217d6d88] 07:00:28,740 WARN [RemotingConnectionImpl] Connection failure has been detected: Did not receive data from /10.34.3.115:50362. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3] 07:00:28,741 WARN [ServerSessionImpl] Client connection failed, clearing up resources for session a4a07e67-df89-11e0-afcb-f0def106737a 07:00:28,741 WARN [ServerSessionImpl] Cleared up resources for session a4a07e67-df89-11e0-afcb-f0def106737a 07:00:28,741 WARN [ServerSessionImpl] Client connection failed, clearing up resources for session a4cff2d8-df89-11e0-afcb-f0def106737a 07:00:28,742 WARN [ServerSessionImpl] Cleared up resources for session a4cff2d8-df89-11e0-afcb-f0def106737a 07:00:44,743 WARN [RemotingConnectionImpl] Connection failure has been detected: Did not receive data from /10.34.3.115:50367. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3] 07:00:44,743 WARN [ServerSessionImpl] Client connection failed, clearing up resources for session adadd94b-df89-11e0-8fbd-f0def106737a 07:00:44,744 WARN [ServerSessionImpl] Cleared up resources for session adadd94b-df89-11e0-8fbd-f0def106737a 07:00:44,744 WARN [ServerSessionImpl] Client connection failed, clearing up resources for session addcff9c-df89-11e0-8fbd-f0def106737a 07:00:44,745 WARN [ServerSessionImpl] Cleared up resources for session addcff9c-df89-11e0-8fbd-f0def106737a 07:02:24,431 WARN [RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4] 07:02:24,431 WARN [BridgeImpl] sf.failover-cluster.3f2b8dba-df88-11e0-ab4f-001b217d6d91::Connection failed before reconnect HornetQException[errorCode=4 message=The connection was disconnected because of server shutdown] at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1278) at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 07:02:24,432 WARN [RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4] 07:02:24,438 WARN [RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4] 07:02:24,438 WARN [RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4] 07:02:24,440 WARN [BridgeImpl] sf.failover-cluster.3f2b8dba-df88-11e0-ab4f-001b217d6d91::Connection failed with failedOver=true HornetQException[errorCode=4 message=The connection was disconnected because of server shutdown] at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1278) at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 07:02:24,449 WARN [ChannelImpl] Can't find packet to clear: last received command id 4 first stored command id 4 07:02:24,695 INFO [RPCManagerImpl] Received new cluster view: [10.16.100.46:55200|4] [10.16.100.46:55200] 07:02:24,760 INFO [DisconnectJournal] New cluster view for partition DisconnectJournal (id: 4, delta: -1) : [10.16.100.46:1099] 07:02:24,760 INFO [DisconnectJournal] I am (10.16.100.46:1099) received membershipChanged event: 07:02:24,760 INFO [DisconnectJournal] Dead members: 1 ([10.16.100.44:1099]) 07:02:24,760 INFO [DisconnectJournal] New Members : 0 ([]) 07:02:24,760 INFO [DisconnectJournal] All Members : 1 ([10.16.100.46:1099])