Uploaded image for project: 'HornetQ'
  1. HornetQ
  2. HORNETQ-252

Connections not closed if client/server version mismatch

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 2.0.0.CR2
    • Fix Version/s: 2.0.0.CR3
    • Component/s: None
    • Labels:
      None

      Description

      If the client and server versions don't match (e.g. during an upgrade where the server is upgraded first) the server logs an error but then leaves the connection open instead of closing it. In our case we have clients which try to reconnect every 10 seconds which means every 10 seconds another connection is left open. We have about 30 clients so this means that within minutes we hit the "too many open files" limit on linux. Here is output from grepping number of connections to one of our clients over time:

      jms1:~# netstat | grep streamer12 | wc -l
      8
      jms1:~# netstat | grep streamer12 | wc -l
      10
      jms1:~# netstat | grep streamer12 | wc -l
      12
      jms1:~# netstat | grep streamer12 | wc -l
      14

      So you can see the number of connections just keeps going up and up.

      This is the strack trace we eventually see in the server:

      [New I/O server boss #1 (channelId: 782092838, /10.101.1.73:5445)] 10:00:39,072 WARNING [org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink] Failed to accept a connection.
      java.io.IOException: Too many open files
      at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
      at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
      at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:227)
      at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
      at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:179)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java.lang.Thread.run(Thread.java:619)

      After which nothing can connect to the server anymore.

        Gliffy Diagrams

          Activity

          Hide
          AdrianWoodhead Adrian Woodhead added a comment -

          Here is what we see in the client logs at the same time:

          [2009-12-16 10:29:22] ERROR (fm.last.jms.JMSProducer:48) - Error connecting PRODUCER for /topic/connectionControl connected
          javax.jms.JMSException: Server and client versions incompatible
          at org.hornetq.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:268)
          at org.hornetq.core.client.impl.FailoverManagerImpl.createSession(FailoverManagerImpl.java:387)
          at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:1123)
          at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:846)
          at org.hornetq.jms.client.HornetQConnection.authorize(HornetQConnection.java:544)
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:600)
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:114)
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:109)
          at fm.last.jms.BaseJMSObject.connect(BaseJMSObject.java:89)
          at fm.last.jms.JMSProducer.connect(JMSProducer.java:41)
          at fm.last.jms.QueueingProducer.connect(QueueingProducer.java:79)
          at fm.last.jms.ConnectionMonitor.run(ConnectionMonitor.java:48)
          Caused by: HornetQException[errorCode=108 message=Server and client versions incompatible]
          ... 12 more
          [2009-12-16 10:29:22] WARN (fm.last.jms.ConnectionMonitor:53) - Connection down, trying again in 10000ms...
          [2009-12-16 10:29:23] INFO (fm.last.jms.JMSReceiver:26) - JMS RECEIVER for /topic/connectionControl connecting...
          [2009-12-16 10:29:23] ERROR (fm.last.jms.JMSReceiver:34) - Error connecting RECEIVER for /topic/connectionControl connected
          javax.jms.JMSException: Server and client versions incompatible
          at org.hornetq.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:268)
          at org.hornetq.core.client.impl.FailoverManagerImpl.createSession(FailoverManagerImpl.java:387)
          at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:1123)
          at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:846)
          at org.hornetq.jms.client.HornetQConnection.authorize(HornetQConnection.java:544)
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:600)
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:114)
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:109)
          at fm.last.jms.BaseJMSObject.connect(BaseJMSObject.java:89)
          at fm.last.jms.JMSReceiver.connect(JMSReceiver.java:27)
          at fm.last.jms.ConnectionMonitor.run(ConnectionMonitor.java:48)
          Caused by: HornetQException[errorCode=108 message=Server and client versions incompatible]
          ... 11 more

          Show
          AdrianWoodhead Adrian Woodhead added a comment - Here is what we see in the client logs at the same time: [2009-12-16 10:29:22] ERROR (fm.last.jms.JMSProducer:48) - Error connecting PRODUCER for /topic/connectionControl connected javax.jms.JMSException: Server and client versions incompatible at org.hornetq.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:268) at org.hornetq.core.client.impl.FailoverManagerImpl.createSession(FailoverManagerImpl.java:387) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:1123) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:846) at org.hornetq.jms.client.HornetQConnection.authorize(HornetQConnection.java:544) at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:600) at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:114) at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:109) at fm.last.jms.BaseJMSObject.connect(BaseJMSObject.java:89) at fm.last.jms.JMSProducer.connect(JMSProducer.java:41) at fm.last.jms.QueueingProducer.connect(QueueingProducer.java:79) at fm.last.jms.ConnectionMonitor.run(ConnectionMonitor.java:48) Caused by: HornetQException [errorCode=108 message=Server and client versions incompatible] ... 12 more [2009-12-16 10:29:22] WARN (fm.last.jms.ConnectionMonitor:53) - Connection down, trying again in 10000ms... [2009-12-16 10:29:23] INFO (fm.last.jms.JMSReceiver:26) - JMS RECEIVER for /topic/connectionControl connecting... [2009-12-16 10:29:23] ERROR (fm.last.jms.JMSReceiver:34) - Error connecting RECEIVER for /topic/connectionControl connected javax.jms.JMSException: Server and client versions incompatible at org.hornetq.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:268) at org.hornetq.core.client.impl.FailoverManagerImpl.createSession(FailoverManagerImpl.java:387) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:1123) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:846) at org.hornetq.jms.client.HornetQConnection.authorize(HornetQConnection.java:544) at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:600) at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:114) at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:109) at fm.last.jms.BaseJMSObject.connect(BaseJMSObject.java:89) at fm.last.jms.JMSReceiver.connect(JMSReceiver.java:27) at fm.last.jms.ConnectionMonitor.run(ConnectionMonitor.java:48) Caused by: HornetQException [errorCode=108 message=Server and client versions incompatible] ... 11 more
          Hide
          jmesnil Jeff Mesnil added a comment -

          fixed in the trunk (see changelog)

          Show
          jmesnil Jeff Mesnil added a comment - fixed in the trunk (see changelog)

            People

            • Assignee:
              jmesnil Jeff Mesnil
              Reporter:
              AdrianWoodhead Adrian Woodhead
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development