Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-2236

Fabric very instable when insights is deployed

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • jboss-fuse-6.2
    • jboss-fuse-6.2
    • Fabric8 v1
    • None
    • % %

    Description

      • What has been deployed under the Fabric Root container -->
      Here is what I have deployed in jboss-fuse-6.2.0.redhat-046
      
      JBossFuse:karaf@root> fabric:container-list
      [id]   [version]  [type]  [connected] [profiles]                       
      [provision status]
      root*  1.0        karaf   yes fabric                           success
                                              fabric-ensemble-0000-1
                                              jboss-fuse-full
                                              insight-camel
                                              insight-console
      insight-metrics.elasticsearch
                                              example-camel-twitter
                                              insight-logs.elasticsearch
      insight-elasticsearch.datastore
                                              insight-elasticsearch.node
      
      • Issue description :
      The behavior is really strange. After a certain period of time (I was 
      absent from my desk & machine for 2 hours), fabric is gone
      
      JBossFuse:karaf@root> fabric:
      fabric:archetype-generate     fabric:archetype-info 
      fabric:archetype-list         fabric:archetype-workspace 
      fabric:create                 fabric:crypt-algorithm-get 
      fabric:crypt-algorithm-set
      fabric:crypt-password-get     fabric:crypt-password-set 
      fabric:encrypt-message        fabric:join fabric:welcomeld
      
      and these errors appears in the log
      
      2014-12-01 16:06:22,298 | ERROR | pool-40-thread-1 | 
      MetricsCollector                 | ?                                   
      ? | 299 - io.fabric8.insight.insight-metrics - 1.2.0.redhat-046 | 
      Error sending metrics
      java.lang.SecurityException: Insufficient roles/credentials for operation
          at 
      org.apache.karaf.management.KarafMBeanServerGuard.handleInvoke(KarafMBeanServerGuard.java:301)[36:org.apache.karaf.management.server:2.4.0.redhat-620046]
          at 
      org.apache.karaf.management.KarafMBeanServerGuard.handleGetAttribute(KarafMBeanServerGuard.java:221)[36:org.apache.karaf.management.server:2.4.0.redhat-620046]
          at 
      org.apache.karaf.management.KarafMBeanServerGuard.handleGetAttributes(KarafMBeanServerGuard.java:227)[36:org.apache.karaf.management.server:2.4.0.redhat-620046]
          at 
      org.apache.karaf.management.KarafMBeanServerGuard.invoke(KarafMBeanServerGuard.java:91)[36:org.apache.karaf.management.server:2.4.0.redhat-620046]
          at 
      org.apache.karaf.management.boot.KarafMBeanServerBuilder$MBeanInvocationHandler.invoke(KarafMBeanServerBuilder.java:63)[karaf-jmx-boot.jar:]
          at com.sun.proxy.$Proxy0.getAttributes(Unknown Source)[:]
          at 
      io.fabric8.insight.metrics.service.support.JmxUtils.execute(JmxUtils.java:69)[299:io.fabric8.insight.insight-metrics:1.2.0.redhat-046]
          at 
      io.fabric8.insight.metrics.service.support.JmxUtils.execute(JmxUtils.java:39)[299:io.fabric8.insight.insight-metrics:1.2.0.redhat-046]
          at 
      io.fabric8.insight.metrics.service.support.JmxUtils.execute(JmxUtils.java:32)[299:io.fabric8.insight.insight-metrics:1.2.0.redhat-046]
          at 
      io.fabric8.insight.metrics.service.MetricsCollector$Task$1.run(MetricsCollector.java:355)[299:io.fabric8.insight.insight-metrics:1.2.0.redhat-046]
          at 
      io.fabric8.insight.metrics.service.MetricsCollector$Task$1.run(MetricsCollector.java:351)[299:io.fabric8.insight.insight-metrics:1.2.0.redhat-046]
          at java.security.AccessController.doPrivileged(Native 
      Method)[:1.7.0_51]
          at 
      javax.security.auth.Subject.doAsPrivileged(Subject.java:473)[:1.7.0_51]
          at 
      io.fabric8.insight.metrics.service.MetricsCollector$Task.run(MetricsCollector.java:351)[299:io.fabric8.insight.insight-metrics:1.2.0.redhat-046]
          at 
      java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)[:1.7.0_51]
          at 
      java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)[:1.7.0_51]
          at 
      java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)[:1.7.0_51]
          at 
      java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.7.0_51]
          at 
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_51]
          at 
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_51]
          at java.lang.Thread.run(Thread.java:744)[:1.7.0_51]
      
      
      2014-12-01 16:25:40,769 | ERROR | ad-1-EventThread | 
      ConnectionState                  | ?                                   
      ? | 90 - io.fabric8.fabric-zookeeper - 1.2.0.redhat-046 | Connection 
      timed out for connection string (192.168.1.80:2181) and timeout 
      (15000) / elapsed (15015)
      org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = 
      ConnectionLoss
          at 
      org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
          at 
      org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
          at 
      org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
          at 
      org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:474)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
          at 
      org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:302)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
          at 
      org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:291)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
          at 
      org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
          at 
      org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:287)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
          at 
      org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
          at 
      org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
          at 
      org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:192)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
          at 
      org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:42)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
          at 
      org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:58)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
          at 
      org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
          at 
      org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
          at 
      org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)[90:io.fabric8.fabric-zookeeper:1.2.0.redhat-046]
      
      

      If I restart the server, then Fabric is back & insight too but this
      message appears in the log now

      2014-12-01 16:35:12,162 | ERROR | pool-40-thread-5 | 
      MetricsCollector                 | ?                                   
      ? | 299 - io.fabric8.insight.insight-metrics - 1.2.0.redhat-046 | 
      Error sending metrics
      java.lang.SecurityException: Insufficient roles/credentials for operation
          at 
      org.apache.karaf.management.KarafMBeanServerGuard.handleInvoke(KarafMBeanServerGuard.java:301)[36:org.apache.karaf.management.server:2.4.0.redhat-620046]
          at 
      org.apache.karaf.management.KarafMBeanServerGuard.handleGetAttribute(KarafMBeanServerGuard.java:221)[36:org.apache.karaf.management.server:2.4.0.redhat-620046]
          at 
      org.apache.karaf.management.KarafMBeanServerGuard.handleGetAttributes(KarafMBeanServerGuard.java:227)[36:org.apache.karaf.management.server:2.4.0.redhat-620046]
          at 
      org.apache.karaf.management.KarafMBeanServerGuard.invoke(KarafMBeanServerGuard.java:91)[36:org.apache.karaf.management.server:2.4.0.redhat-620046]
          at 
      org.apache.karaf.management.boot.KarafMBeanServerBuilder$MBeanInvocationHandler.invoke(KarafMBeanServerBuilder.java:63)[karaf-jmx-boot.jar:]
          at com.sun.proxy.$Proxy0.getAttributes(Unknown Source)[:]
          at 
      io.fabric8.insight.metrics.service.support.JmxUtils.execute(JmxUtils.java:69)[299:io.fabric8.insight.insight-metrics:1.2.0.redhat-046]
          at 
      io.fabric8.insight.metrics.service.support.JmxUtils.execute(JmxUtils.java:39)[299:io.fabric8.insight.insight-metrics:1.2.0.redhat-046]
          at 
      io.fabric8.insight.metrics.service.support.JmxUtils.execute(JmxUtils.java:32)[299:io.fabric8.insight.insight-metrics:1.2.0.redhat-046]
          at 
      io.fabric8.insight.metrics.service.MetricsCollector$Task$1.run(MetricsCollector.java:355)[299:io.fabric8.insight.insight-metrics:1.2.0.redhat-046]
          at 
      io.fabric8.insight.metrics.service.MetricsCollector$Task$1.run(MetricsCollector.java:351)[299:io.fabric8.insight.insight-metrics:1.2.0.redhat-046]
          at java.security.AccessController.doPrivileged(Native 
      Method)[:1.7.0_51]
          at 
      javax.security.auth.Subject.doAsPrivileged(Subject.java:473)[:1.7.0_51]
          at 
      io.fabric8.insight.metrics.service.MetricsCollector$Task.run(MetricsCollector.java:351)[299:io.fabric8.insight.insight-metrics:1.2.0.redhat-046]
          at 
      java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)[:1.7.0_51]
          at 
      java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)[:1.7.0_51]
          at 
      java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)[:1.7.0_51]
          at 
      java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.7.0_51]
          at 
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_51]
          at 
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_51]
          at java.lang.Thread.run(Thread.java:744)[:1.7.0_51]
      2014-12-01 16:35:14,252 | WARN  | icMetricsStorage | 
      AbstractElasticsearchStorage     | ?                                   
      ? | 295 - io.fabric8.insight.insight-elasticsearch-factory - 
      1.2.0.redhat-046 | Error executing request: 
      UnavailableShardsException[[insight-2014.12.01][3] [2] shardIt, [0] 
      active : Timeout waiting for [1m], request: 
      org.elasticsearch.action.bulk.BulkShardRequest@2ee2ce3c]
      2014-12-01 16:35:14,252 | WARN  | icMetricsStorage | 
      AbstractElasticsearchStorage     | ?                                   
      ? | 295 - io.fabric8.insight.insight-elasticsearch-factory - 
      1.2.0.redhat-046 | Error executing request: 
      UnavailableShardsException[[insight-2014.12.01][1] [2] shardIt, [0] 
      active : Timeout waiting for [1m], request: 
      org.elasticsearch.action.bulk.BulkShardRequest@2c51dd30]
      2014-12-01 16:35:14,252 | WARN  | icMetricsStorage | 
      AbstractElasticsearchStorage     | ?                                   
      ? | 295 - io.fabric8.insight.insight-elasticsearch-factory - 
      1.2.0.redhat-046 | Error executing request: 
      UnavailableShardsException[[insight-2014.12.01][1] [2] shardIt, [0] 
      active : Timeout waiting for [1m], request: 
      org.elasticsearch.action.bulk.BulkShardRequest@2c51dd30]
      2014-12-01 16:35:14,252 | WARN  | icMetricsStorage | 
      AbstractElasticsearchStorage     | ?                                   
      ? | 295 - io.fabric8.insight.insight-elasticsearch-factory - 
      1.2.0.redhat-046 | Error executing request: 
      UnavailableShardsException[[insight-2014.12.01][3] [2] shardIt, [0] 
      active : Timeout waiting for [1m], request: 
      org.elasticsearch.action.bulk.BulkShardRequest@2ee2ce3c]
      2014-12-01 16:35:14,252 | WARN  | icMetricsStorage | 
      AbstractElasticsearchStorage     | ?                                   
      ? | 295 - io.fabric8.insight.insight-elasticsearch-factory - 
      1.2.0.redhat-046 | Error executing reques
      

      Attachments

        Activity

          People

            jdyson_jira Jimmi Dyson (Inactive)
            cmoullia Charles Moulliard
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: