Uploaded image for project: 'HornetQ'
  1. HornetQ
  2. HORNETQ-252

Connections not closed if client/server version mismatch

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 2.0.0.CR2
    • Fix Version/s: 2.0.0.CR3
    • Component/s: None
    • Labels:
      None

      Description

      If the client and server versions don't match (e.g. during an upgrade where the server is upgraded first) the server logs an error but then leaves the connection open instead of closing it. In our case we have clients which try to reconnect every 10 seconds which means every 10 seconds another connection is left open. We have about 30 clients so this means that within minutes we hit the "too many open files" limit on linux. Here is output from grepping number of connections to one of our clients over time:

      jms1:~# netstat | grep streamer12 | wc -l
      8
      jms1:~# netstat | grep streamer12 | wc -l
      10
      jms1:~# netstat | grep streamer12 | wc -l
      12
      jms1:~# netstat | grep streamer12 | wc -l
      14

      So you can see the number of connections just keeps going up and up.

      This is the strack trace we eventually see in the server:

      [New I/O server boss #1 (channelId: 782092838, /10.101.1.73:5445)] 10:00:39,072 WARNING [org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink] Failed to accept a connection.
      java.io.IOException: Too many open files
      at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
      at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
      at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:227)
      at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
      at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:179)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java.lang.Thread.run(Thread.java:619)

      After which nothing can connect to the server anymore.

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                jmesnil Jeff Mesnil
                Reporter:
                AdrianWoodhead Adrian Woodhead
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: