Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-4610

Fabric gateway fails because of incorrect use of ShutdownTracker

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Done
    • Affects Version/s: jboss-fuse-6.2
    • Fix Version/s: jboss-fuse-6.3
    • Component/s: Fabric8 v1
    • Labels:
      None
    • Environment:

      JBoss Fuse 6.2.0

    • Sprint:
      Sprint 5 - towards ER2

      Description

      An installation has several fabric gateways distributing requests to ActiveMQ message brokers in fabric instances. There is a further, upstream load balancer ahead of the gateways.

      Under load, after some time, one of the gateways stops accepting further requests. The problem is first noticed in an exception handler, although the actual exception is never reported. The exception handler tries to call release() on the gateway's ShutdownHandler, but this call fails with the following exception:

      2015-12-01 23:10:54,165 | ERROR | entloop-thread-2 | DefaultContext                   | 266 - io.fabric8.fabric-vertx - 1.2.0.redhat-133 | Unhandled exception
      java.lang.IllegalStateException: Unbalanced calls to release detected.
      	at io.fabric8.common.util.ShutdownTracker.release(ShutdownTracker.java:78)[65:io.fabric8.common-util:1.2.0.redhat-133]
      	at io.fabric8.gateway.handlers.detecting.DetectingGateway.handleShutdown(DetectingGateway.java:430)[268:io.fabric8.gateway-core:1.2.0.redhat-133]
      	at io.fabric8.gateway.handlers.detecting.DetectingGateway.access$400(DetectingGateway.java:54)[268:io.fabric8.gateway-core:1.2.0.redhat-133]
      	at io.fabric8.gateway.handlers.detecting.DetectingGateway$6$2.handle(DetectingGateway.java:410)[268:io.fabric8.gateway-core:1.2.0.redhat-133]
      	at io.fabric8.gateway.handlers.detecting.DetectingGateway$6$2.handle(DetectingGateway.java:407)[268:io.fabric8.gateway-core:1.2.0.redhat-133]
      	at org.vertx.java.core.net.impl.ConnectionBase.handleException(ConnectionBase.java:126)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133]
      	at org.vertx.java.core.net.impl.VertxHandler$2.run(VertxHandler.java:104)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133]
      	at org.vertx.java.core.impl.DefaultContext$3.run(DefaultContext.java:175)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133]
      	at org.vertx.java.core.impl.DefaultContext.execute(DefaultContext.java:135)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133]
      	at org.vertx.java.core.net.impl.VertxHandler.exceptionCaught(VertxHandler.java:96)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:275)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:253)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:835)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:87)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)[275:io.netty.common:4.0.27.Final]
      	at java.lang.Thread.run(Thread.java:745)[:1.8.0_65]
      

      Thereafter the gateway is in a peculiar state, and all further requests on it fail, with the following exception:

      2015-12-01 23:10:54,413 | ERROR | entloop-thread-2 | DefaultContext                   | 266 - io.fabric8.fabric-vertx - 1.2.0.redhat-133 | Unhandled exception
      io.fabric8.common.util.ShutdownTracker$ShutdownException
      	at io.fabric8.common.util.ShutdownTracker.retain(ShutdownTracker.java:51)[65:io.fabric8.common-util:1.2.0.redhat-133]
      	at io.fabric8.gateway.handlers.detecting.DetectingGateway.handle(DetectingGateway.java:206)[268:io.fabric8.gateway-core:1.2.0.redhat-133]
      	at io.fabric8.gateway.handlers.detecting.DetectingGatewayNetSocketHandler.handle(DetectingGatewayNetSocketHandler.java:33)[268:io.fabric8.gateway-core:1.2.0.redhat-133]
      	at io.fabric8.gateway.handlers.detecting.DetectingGatewayNetSocketHandler.handle(DetectingGatewayNetSocketHandler.java:24)[268:io.fabric8.gateway-core:1.2.0.redhat-133]
      	at org.vertx.java.core.net.impl.DefaultNetServer$ServerHandler.doConnected(DefaultNetServer.java:572)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133]
      	at org.vertx.java.core.net.impl.DefaultNetServer$ServerHandler.access$1400(DefaultNetServer.java:525)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133]
      	at org.vertx.java.core.net.impl.DefaultNetServer$ServerHandler$2.run(DefaultNetServer.java:564)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133]
      	at org.vertx.java.core.impl.DefaultContext$3.run(DefaultContext.java:175)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133]
      	at org.vertx.java.core.impl.DefaultContext.execute(DefaultContext.java:135)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133]
      	at org.vertx.java.core.net.impl.DefaultNetServer$ServerHandler.connected(DefaultNetServer.java:562)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133]
      	at org.vertx.java.core.net.impl.DefaultNetServer$ServerHandler.channelActive(DefaultNetServer.java:557)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:212)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:818)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:454)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:418)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:60)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:48)[277:io.netty.transport:4.0.27.Final]
      	at org.vertx.java.core.net.impl.VertxEventLoopGroup.register(VertxEventLoopGroup.java:62)[266:io.fabric8.fabric-vertx:1.2.0.redhat-133]
      	at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:251)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[277:io.netty.transport:4.0.27.Final]
      	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)[275:io.netty.common:4.0.27.Final]
      	at java.lang.Thread.run(Thread.java:745)[:1.8.0_65]
      

      It seems that in some cases we end up in a situation where an operation that shut only be performed during shutdown is performed in response to an exception, and that leaves the gateway in an indeterminate state.

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                chirino Hiram Chirino
                Reporter:
                kboone Kevin Boone
                Tester:
                Andrej Vano
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: