Details
-
Bug
-
Resolution: Done
-
Major
-
None
-
jboss-fuse-6.3
-
None
Description
Some containers that are running with fabric gateway shown as stopped in hawtio console despite being active and functioning properly. All containers are not affected by this, only a couple of them.
This is not reproducible (Or atleast I have not been able to do so). This has been observed in Fuse 6.3 R10.
Also, these containers were missing from the "/fabric/registry/containers/alive" registry.
If I manually create a node here with the name of the affected container, then it fixes this issue and so does restarting the container.
There are a few exception in the logs but the most relevant one is:
2019-08-05 12:20:28,699 | WARN | onStateManager-0 | DefaultPullPushPolicy | 72 - io.fabric8.fabric-git - 1.2.0.redhat-630371 | Pull failed during fetch because remote repository is not ready yet (pull will be retried): http://10.x.x.x:8181/git/fabric/: http://10.x.x.x:8181/git/fabric/info/refs?service=git-upload-pack not found 2019-08-05 12:20:28,763 | ERROR | MCF-1-thread-1 | KarafContainerRegistration | 80 - io.fabric8.runtime.fabric-runtime-container-karaf-registration - 1.2.0.redhat-630371 | Error while checking/setting container status. 2019-08-05 12:20:28,855 | ERROR | n-reg-5-thread-1 | FabricMBeanRegistrationListener | 70 - io.fabric8.fabric-core - 1.2.0.redhat-630371 | Error while updating the process id. org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /fabric/registry/containers/status/container-name/pid at org.apache.zookeeper.KeeperException.create(KeeperException.java:122)[77:io.fabric8.fabric-zookeeper:1.2.0.redhat-630371] at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)[77:io.fabric8.fabric-zookeeper:1.2.0.redhat-630371] at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792)[77:io.fabric8.fabric-zookeeper:1.2.0.redhat-630371] at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740)[77:io.fabric8.fabric-zookeeper:1.2.0.redhat-630371] at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)[77:io.fabric8.fabric-zookeeper:1.2.0.redhat-630371] 2019-08-05 12:20:28,977 | ERROR | MCF-1-thread-1 | FabricWebRegistrationHandler | 153 - io.fabric8.fabric-web - 1.2.0.redhat-630371 | Failed to register servlet /jolokia. org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /fabric/registry/clusters/servlets/io.fabric8.fabric-jolokia/1.2.0.redhat-630371/jolokia/container-name at org.apache.zookeeper.KeeperException.create(KeeperException.java:122)[77:io.fabric8.fabric-zookeeper:1.2.0.redhat-630371] at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)[77:io.fabric8.fabric-zookeeper:1.2.0.redhat-630371] at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792)[77:io.fabric8.fabric-zookeeper:1.2.0.redhat-630371] 2019-08-05 12:28:18,325 | WARN | MCF-1-thread-1 | KarafContainerRegistration | 80 - io.fabric8.runtime.fabric-runtime-container-karaf-registration - 1.2.0.redhat-630371 | Error updating Fabric Container information. This exception will be ignored. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for/fabric/registry/containers/alive/container-name at org.apache.zookeeper.KeeperException.create(KeeperException.java:114) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:882) at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:250) at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:244) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109) at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:241) at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:225) at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:35) at io.fabric8.zookeeper.utils.ZooKeeperUtils.delete(ZooKeeperUtils.java:282) at io.fabric8.runtime.container.karaf.KarafContainerRegistration.checkAlive(KarafContainerRegistration.java:252) at io.fabric8.runtime.container.karaf.KarafContainerRegistration.activateInternal(KarafContainerRegistration.java:158) at io.fabric8.runtime.container.karaf.KarafContainerRegistration.activate(KarafContainerRegistration.java:136) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)[:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)[:1.8.0_151] at java.lang.reflect.Method.invoke(Unknown Source)[:1.8.0_151] at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:231)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:39)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:624)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:508)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:149)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:315)[13:org.apache .felix.scr:1.8.4] at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:127)[13:org.apache.felix.scr: 1.8.4] at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:871)[13:org.apache.felix.scr:1.8.4 ] at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:838)[13:org.apache.felix.s cr:1.8.4] at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:127)[13:org.apache.felix.scr: 1.8.4] at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:871)[13:org.apache.felix.scr:1.8.4 ] at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:838)[13:org.apache.felix.s cr:1.8.4] at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:777)[13:org.apache.felix.scr:1.8.4 ] at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistrationImpl.java:320)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.framework.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:231)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.framework.ServiceRegistry.getService(ServiceRegistry.java:327)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.framework.Felix.getService(Felix.java:3576)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.framework.BundleContextImpl.getService(BundleContextImpl.java:468)[org.apache.felix.framework-4.4.1.jar:] at org.osgi.util.tracker.ServiceTracker.addingService(ServiceTracker.java:411)[8:org.apache.felix.configadmin:1.8.12] at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:932)[8:org.apache.felix.configadmin:1.8.12] at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:864)[8:org.apache.felix.configadmin:1.8.12] at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)[8:org.apache.felix.configadmin:1.8.12] at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:229)[8:org.apache.felix.configadmin:1.8.12] at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:894)[8:org.apache.felix.configadmin:1.8.12] at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:943)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:794)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:544)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4445)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.framework.Felix.registerService(Felix.java:3431)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:1003)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:992)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:134)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:1044)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:841)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:931)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:895)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1480)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1401)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:1210)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:1148)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1432)[13:org.apache.felix.scr:1.8.4] at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:943)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:794)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:544)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4445)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.framework.Felix.registerService(Felix.java:3431)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)[org.apache.felix.framework-4.4.1.jar:] at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:353)[org.apache.felix.framework-4.4.1.jar:] at io.fabric8.zookeeper.curator.ManagedCuratorFramework$State.stateChanged(ManagedCuratorFramework.java:163)[77:io.fabric8.fabric-zookeeper:1.2.0.redhat-630371] at org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:259)[77:io.fabric8.fabric-zookeeper:1.2.0.redhat-630371] at org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:255)[77:io.fabric8.fabric-zookeeper:1.2.0.redhat-630371] at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93)[77:io.fabric8.fabric-zookeeper:1.2.0.redhat-630371] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)[:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)[:1.8.0_151] at java.lang.Thread.run(Unknown Source)[:1.8.0_151]
