Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-85

Connection.start sometimes fails to connect when master container is destroyed when using replicated levelDB and fabric discovery.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Obsolete
    • Major
    • None
    • None
    • None
    • None
    • Hide

      To reproduce:
      1)Start fuse
      2)run command:

      fabric:create --wait-for-provisioning
      

      3)Unpack test attached reconnect-test.zip and run with mvn clean install

      Show
      To reproduce: 1)Start fuse 2)run command: fabric:create --wait- for -provisioning 3)Unpack test attached reconnect-test.zip and run with mvn clean install

    Description

      This is a stacktrace of exception:

      [mIteration=77
      �[0;32m10-12-2014 17:19:35,189 | INFO | [FabricDiscoveryAgent] | call - Using local ZKClient
      �[m�[0;32m10-12-2014 17:19:35,190 | INFO | [CuratorFrameworkImpl] | start - Starting
      �[m�[0;32m10-12-2014 17:19:35,194 | INFO | [ZooKeeper] | <init> - Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@41e65439
      �[m�[0;32m10-12-2014 17:19:35,195 | INFO | [ClientCnxn] | logStartConnect - Opening socket connection to server dhcp-4-224.brq.redhat.com/127.0.0.1:2181
      �[m�[0;32m10-12-2014 17:19:35,196 | INFO | [ClientCnxn] | primeConnection - Socket connection established to dhcp-4-224.brq.redhat.com/127.0.0.1:2181, initiating session
      �[m�[0;32m10-12-2014 17:19:35,206 | INFO | [ClientCnxn] | onConnected - Session establishment complete on server dhcp-4-224.brq.redhat.com/127.0.0.1:2181, sessionid = 0x14a34f50ff001a1, negotiated timeout = 40000
      �[m�[0;32m10-12-2014 17:19:35,207 | INFO | [ConnectionStateManager] | postState - State change: CONNECTED
      �[m�[0;32m10-12-2014 17:19:35,794 | INFO | [SSHClient] | executeCommand - Command: activemq:list
      �[m�[0;32m10-12-2014 17:19:35,997 | INFO | [SSHClient] | executeCommand - Response: brokerName = broker-nossl
       
       
      �[m!!!Restart finished container 0
      !!!Stopping container 1
      !!!!Killing container broker-nossl-c1
      �[0;32m10-12-2014 17:19:35,997 | INFO | [SSHClient] | executeCommand - Command: shell:exec kill -9 21056
      �[m�[0;32m10-12-2014 17:19:36,237 | INFO | [DiscoveryTransport] | onServiceAdd - Adding new broker connection URL: tcp://dhcp-10-40-3-26.brq.redhat.com:39445
      �[m�[0;32m10-12-2014 17:19:36,241 | INFO | [FailoverTransport] | doReconnect - Successfully connected to tcp://dhcp-10-40-3-26.brq.redhat.com:39445
      �[m�[0;33m10-12-2014 17:19:36,286 | WARN | [FailoverTransport] | handleTransportFailure - Transport (tcp://dhcp-10-40-3-26.brq.redhat.com/10.40.3.26:39445@54170) failed, attempting to automatically reconnect
      org.apache.activemq.transport.InactivityIOException: Cannot send, channel has already failed: tcp://10.40.3.26:39445
              at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297)
              at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
              at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
              at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
              at org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:665)
              at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
              at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
              at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
              at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
              at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406)
              at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522)
              at org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
              at org.jboss.fuse.qa.amq.threads.ConnectionRestarter.performAction(ConnectionRestarter.java:19)
              at org.jboss.fuse.qa.amq.threads.AbstractSenderReceiver.run(AbstractSenderReceiver.java:63)
              at java.lang.Thread.run(Thread.java:744)
      �[m�[0;32m10-12-2014 17:19:36,470 | INFO | [SSHClient] | executeCommand - Response:
      �[m�[0;32m10-12-2014 17:19:36,471 | INFO | [Container] | killContainerAtLocalHost - Result of killing master process :
      �[m�[0;32m10-12-2014 17:19:36,471 | INFO | [SSHClient] | executeCommand - Command: shell:exec kill -9 21056
      �[m�[0;32m10-12-2014 17:19:36,780 | INFO | [SSHClient] | executeCommand - Response:
      �[m�[0;32m10-12-2014 17:19:36,781 | INFO | [Container] | killContainerAtLocalHost - Result of killing master process :
      �[m!!!!Waiting for empty process broker-nossl-c1
      

      In fabric there is created 3 containers with replicated levelDB:

      mq-create --no-ssl --parent-profile=mq-replicated --group a broker-nossl
      container-create-child --profile mq-broker-a.broker-nossl root broker-nossl-c1
      container-create-child --profile mq-broker-a.broker-nossl root broker-nossl-c2
      container-create-child --profile mq-broker-a.broker-nossl root broker-nossl-c3
      

      One thread kills and then starts containers with replicated levelDB one by one.
      Anpther parallel thread continuously starts and stops connection to url:

      discovery:(fabric://a)?useExponentialBackoff=false&trace=true
      

      using the following code:

      Connection con = getConnectionFactory().createConnection();
      		con.start();
      		Session session = con.createSession(false, Session.AUTO_ACKNOWLEDGE);
      		Thread.sleep(10);
      		session.close();
      		con.stop();
      		con.close();
      

      Attachments

        1. logs.zip
          106 kB
        2. reconnect-test.zip
          74 kB

        Issue Links

          Activity

            People

              Unassigned Unassigned
              emedvede Elena Medvedeva (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: