-
Bug
-
Resolution: Done
-
Major
-
jboss-fuse-6.1, jboss-fuse-6.2
-
%
-
Under certain (random) circumstances during server restart, we are noticing that not all of our camel endpoints are re-registered in ZooKeeper. This leads to:
13:49:18,843 | INFO | p1607069631-2406 | FabricProxyRoute | 142 - org.apache.camel.camel-core - 2.12.0.redhat-611423 | header.CamelHttpUri : /services/ping 13:49:18,844 | INFO | p1607069631-2406 | LoggingEventNotifer | 256 - Fabric-Proxy - 3.0.1 | Took 0 millis to send to: Endpoint[fabric://Services] SSO/EMAIL/PORTAL=null/null/null 13:49:18,945 | INFO | p1607069631-2406 | LoggingEventNotifer | 256 - Fabric-Proxy - 3.0.1 | Took 0 millis to send to: Endpoint[fabric://Services] SSO/EMAIL/PORTAL=null/null/null 13:49:19,046 | INFO | p1607069631-2406 | LoggingEventNotifer | 256 - Fabric-Proxy - 3.0.1 | Took 1 millis to send to: Endpoint[fabric://Services] SSO/EMAIL/PORTAL=null/null/null 13:49:19,146 | INFO | p1607069631-2406 | LoggingEventNotifer | 256 - Fabric-Proxy - 3.0.1 | Took 0 millis to send to: Endpoint[fabric://Services] SSO/EMAIL/PORTAL=null/null/null 13:49:19,147 | ERROR | p1607069631-2406 | DefaultErrorHandler | 142 - org.apache.camel.camel-core - 2.12.0.redhat-611423 | Failed delivery for (MessageId: ID-ip-10-230-209-166-57863-1442961249079-0-18501 on ExchangeId: ID-ip-10-230-209-166-57863-1442961249079-0-18500). Exhausted after delivery attempt: 4 caught: java.lang.IllegalStateException: No processors found. Message History --------------------------------------------------------------------------------------------------------------------------------------- RouteId ProcessorId Processor Elapsed (ms) [FabricProxyRoute ] [FabricProxyRoute ] [http://0.0.0.0:8200/services ] [ 304] [FabricProxyRoute ] [log1 ] [log ] [ 0] [FabricProxyRoute ] [log4 ] [log ] [ 0] [FabricProxyRoute ] [log6 ] [log ] [ 0] [FabricProxyRoute ] [when3 ] [when[simple{'' != ''}] ] [ 0] [FabricProxyRoute ] [choice2 ] [when[simple{${header.CamelHttpUri} contains '/services/cname'}]choice[when[sim] [ 304] [FabricProxyRoute ] [log57 ] [log ] [ 0] [FabricProxyRoute ] [when48 ] [when[simple{${header.CamelHttpMethod} == 'POST'}] ] [ 0] [FabricProxyRoute ] [to37 ] [fabric:Services ] [ 303] Exchange --------------------------------------------------------------------------------------------------------------------------------------- Exchange[ Id ID-ip-10-230-209-166-57863-1442961249079-0-18500 ExchangePattern InOut Headers {Accept=*/*, breadcrumbId=ID-ip-10-230-209-166-57863-1442961249079-0-18501, CamelHttpMethod=HEAD, CamelHttpPath=/ping, CamelHttpQuery=null, CamelHttpServletRequest=(HEAD /services/ping)@1820517843 org.eclipse.jetty.server.Request@6c82e5d3, CamelHttpServletResponse=HTTP/1.1 200 ^M , CamelHttpUri=/services/ping, CamelHttpUrl=http://localhost:8200/services/ping, CamelRedelivered=true, CamelRedeliveryCounter=3, CamelRedeliveryMaxCounter=3, CamelServletContextPath=/services, CamelSkipWwwFormUrlEncoding=true, Content-Type=null, Host=localhost:8200, User-Agent=curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2} BodyType org.apache.camel.converter.stream.InputStreamCache Body [Body is instance of org.apache.camel.StreamCache] ] Stacktrace Stacktrace --------------------------------------------------------------------------------------------------------------------------------------- java.lang.IllegalStateException: No processors found. at io.fabric8.camel.DefaultLoadBalancerFactory$1.process(DefaultLoadBalancerFactory.java:37)[260:io.fabric8.fabric-camel:1.0.0.redhat-423] at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:105)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.loadbalancer.LoadBalancerSupport.process(LoadBalancerSupport.java:87)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at io.fabric8.camel.FabricLocatorEndpoint$1.process(FabricLocatorEndpoint.java:108)[260:io.fabric8.fabric-camel:1.0.0.redhat-423] at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:110)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.interceptor.HandleFaultInterceptor.process(HandleFaultInterceptor.java:41)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:91)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:111)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.interceptor.HandleFaultInterceptor.process(HandleFaultInterceptor.java:41)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:91)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)[142:org.apache.camel.camel-core:2.12.0.redhat-611423] at org.apache.camel.component.jetty.CamelContinuationServlet.service(CamelContinuationServlet.java:152)[234:org.apache.camel.camel-jetty:2.12.0.redhat-611423] at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)[91:org.apache.geronimo.specs.geronimo-servlet_3.0_spec:1.0] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.server.Server.handle(Server.java:370)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)[92:org.eclipse.jetty.aggregate.jetty-all-server:8.1.14.v20131031] at java.lang.Thread.run(Thread.java:745)[:1.7.0_79] 13:49:19,148 | ERROR | p1607069631-2406 | LoggingEventNotifer | 256 - Fabric-Proxy - 3.0.1 | Failed exchange FabricProxyRoute
Registered ZK services, one registered, while the other is missing.
JBossFuse:fuseadmin@mygeaportal-10.230.209.103> zk:list -r | grep -i endpoints
/fabric/registry/camel/endpoints
/fabric/registry/camel/endpoints/fabricServices
/fabric/registry/camel/endpoints/fabricServices/00000000015
/fabric/registry/camel/endpoints/Services
From the camel point of view, everything looks correct.
JBossFuse:fuseadmin@node> route-list Context Route Status ------- ----- ------ blueprintContext PocJob Started fabricMServiceCTX fabricMServiceRoute Started fabricServiceCTX fabricServiceRoute Started JBossFuse:fuseadmin@noce> endpoint-list Context Uri Status ------- --- ------ blueprintContext timer://POC?period=600000 Started fabricMServiceCTX fabric://Services:jetty:http://0.0.0.0:8310/services?bridgeEndpoint=true&continuationTimeout=600000&enableMulti Started fabricMServiceCTX http://0.0.0.0:8210/services Started fabriMServiceCTX http://1.2.3.4:8310/services Started fabricServiceCTX fabric://fabricServices:jetty:http://1.2.3.4:8302/services?bridgeEndpoint=true&matchOnUriPrefix=true&throwExcepti Started fabricServiceCTX http://0.0.0.0:8202/services Started fabricServiceCTX http://1.2.3.4:8302/services
