Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-2755

[OSE] [6.1.R2P1] [AMQ] AMQ gear creation takes too long

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • jboss-fuse-6.2
    • jboss-fuse-6.1 on OSE
    • Fuse on Openshift
    • None
    • % %
    • Hide

      delete the last 14 lines of bin/upgrade

      if [ "$patch_applied" -eq 1 ]; then
  if ! is_fabric_provisioned; then
    client_message "${OPENSHIFT_APP_NAME} fabric needs to be restarted to complete installation of patch."
    restart
    exit 0
  fi
fi

if [ "$try_restart" -eq 1 ]; then
    client_message "${OPENSHIFT_APP_NAME} fabric needs to be restarted to complete provisioning."
    restart
    exit 0
fi
      Show
      delete the last 14 lines of bin/upgrade if [ "$patch_applied" -eq 1 ]; then if ! is_fabric_provisioned; then client_message "${OPENSHIFT_APP_NAME} fabric needs to be restarted to complete installation of patch." restart exit 0 fi fi if [ "$try_restart" -eq 1 ]; then client_message "${OPENSHIFT_APP_NAME} fabric needs to be restarted to complete provisioning." restart exit 0 fi
    • Hide
      1. create a fuse gear in a domain "dummy" (to generate a moderate load on the server to make it more reproducible)
      2. attempt to create an amq gear in a domain "test" (or any different domain than the first gear)
      3. delete the amq gear and repeat step 2 until it fails to be created
      Show
      create a fuse gear in a domain "dummy" (to generate a moderate load on the server to make it more reproducible) attempt to create an amq gear in a domain "test" (or any different domain than the first gear) delete the amq gear and repeat step 2 until it fails to be created

    Description

      The 6.1.R2P1 AMQ gear creation frequently fails due to a timeout even on a typical HW configuration.

      after adding additional logs to the upgrade script, it seems that the majority of time wasted is waiting in the upgrade script for "is_fabric_provisioned" after the patching has been done.

      Note that the last part of the upgrade script is not useful anyway when running the upgrade script as part of the first gear installation, for the following reasons:

      1. just created gears don't need to be restarted
      2. even if the amq creation failed in a way that would warrant a restart, it would timeout before the is_fabric_provisioned loop finishes
      March 03 08:47:34 INFO oo_spawn buffer(11/) CLIENT_RESULT: Application amq created successfully.  Please make note of these credentials:
CLIENT_RESULT:
March 03 08:47:34 INFO oo_spawn buffer(11/) CLIENT_RESULT:    Console URL:        https://amq-test.example.com
March 03 08:47:34 INFO oo_spawn buffer(11/) CLIENT_RESULT:    Console User:       [HIDDEN]
March 03 08:47:34 INFO oo_spawn buffer(11/) CLIENT_RESULT:    Console Password:   [HIDDEN]
March 03 08:47:34 INFO oo_spawn buffer(11/) CLIENT_RESULT:    Zookeeper URL:      amq-test.example.com:64931
March 03 08:47:34 INFO oo_spawn buffer(11/) CLIENT_RESULT:    Zookeeper Password: [HIDDEN]
March 03 08:47:34 INFO oo_spawn buffer(11/) CLIENT_RESULT:    SSL Certificate URL: https://amq-test.example.com/hawtio/index.html#/wiki/branch/1.0/view/fabric/profiles/default.profile/keystore.cert
March 03 08:47:34 INFO oo_spawn buffer(11/) CLIENT_RESULT:
March 03 08:47:34 INFO oo_spawn buffer(11/) CLIENT_RESULT:    The management console will be available shortly at: https://amq-test.example.com
March 03 08:47:34 INFO oo_spawn buffer(11/) CLIENT_RESULT:
March 03 08:48:23 INFO oo_spawn buffer(11/) CLIENT_RESULT:
March 03 08:48:35 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: amq already started
March 03 08:48:36 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: Waiting for patching service to become available...
March 03 08:48:38 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: Patching all profile versions with /var/lib/openshift/54f5bb546892df59da000a1a/amq//patches/jboss-fuse-6.1.0.redhat-379-r2-611423.zip
March 03 08:49:03 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX Patch /var/lib/openshift/54f5bb546892df59da000a1a/amq//patches/jboss-fuse-6.1.0.redhat-379-r2-611423.zip applied
March 03 08:49:05 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: Waiting for patching service to become available...
March 03 08:49:10 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: Patching all profile versions with /var/lib/openshift/54f5bb546892df59da000a1a/amq//patches/jboss-fuse-6.1.0.redhat-379-r2p1-611424.zip
March 03 08:49:23 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX Patch /var/lib/openshift/54f5bb546892df59da000a1a/amq//patches/jboss-fuse-6.1.0.redhat-379-r2p1-611424.zip applied
March 03 08:49:23 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX amq about to check is_fabric_provisioned
March 03 08:49:23 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=60 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:49:30 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=59 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:49:35 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=58 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:49:41 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=57 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:49:46 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=56 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:49:51 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=55 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:49:56 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=54 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:50:01 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=53 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:50:06 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=52 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:50:11 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=51 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:50:16 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=50 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:50:21 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=49 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:50:27 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=48 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:50:33 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=47 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:50:40 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=46 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:50:45 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=45 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:50:51 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=44 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:50:56 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=43 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:51:01 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=42 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:51:06 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=41 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:51:11 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=40 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:51:16 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=39 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:51:21 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=38 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:51:26 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=37 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:51:31 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=36 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:51:36 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=35 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:51:41 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=34 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:51:46 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=33 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:51:51 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=32 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:51:56 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=31 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq
March 03 08:52:01 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: XXX retries=30 curl -s http://admin:Xs9m3hT-RqSI@127.4.206.1:8080/jolokia/exec/io.fabric8:type=Fabric/getContainer(java.lang.String)/amq

      Attachments

        1. karaf-amq.log
          180 kB
        2. karaf-amq.log.1
          1.00 MB

        Issue Links

          relates to

          Bug - A problem that impairs or prevents the functions of the product. ENTESB-2858 [OSE] [6.1.R2P2] [patching] hawtio briefly online before HTTP 503 after installation

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed

          Activity

            People

              janstey@redhat.com Jonathan Anstey
              maschmid@redhat.com Marek Schmidt
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: