Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-1999

fuse gear creation sometimes exceeds OSE install timeout "exceeded timeout of"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Critical Critical
    • Future
    • jboss-fuse-6.1 on OSE
    • Fuse on Openshift
    • None
    • % %
    • Hide
      1. create some gears with rhc create-app fuseX fuse-1.0.0 --no-git
      2. deploy mq-amq on some, example-mq on others to generate some realistic load
      3. create some more gears
      4. remove some gears / repeat until it fails
      Show
      create some gears with rhc create-app fuseX fuse-1.0.0 --no-git deploy mq-amq on some, example-mq on others to generate some realistic load create some more gears remove some gears / repeat until it fails

      Fuse gear creation sometimes fail with the following error when the node is under load.

      This specific run was with a build containing patch, which may increase the time of the install script, although according to the logs it didn't make into the patching yet, so this is probably not patching-specific.

      September 24 07:15:29 INFO openshift-agent: request end: action=cartridge_do, requestid=e07118b9ae195307b22dba57de1b123e, senderid=broker.hosts.sample.com, statuscode=1, data={:time=>nil, :output=>"Shell command '/sbin/runuser -s /bin/sh 5422a6b58db227512300014d -c \"exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c5,c315' /bin/sh -c \\\"/var/lib/openshift/5422a6b58db227512300014d/fuse/bin/install --version 1.0.0\\\"\"' exceeded timeout of 225\n\n", :exitcode=>157, :addtl_params=>nil}

      Looking at /var/log/openshift/node/platform-trace.log it seems it just didn't manage to setup in time for the default 360s openshift timeout.

      September 24 07:11:44 INFO oo_spawn running /sbin/runuser -s /bin/sh 5422a6b58db227512300014d -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c5,c315' /bin/sh -c \"/var/lib/openshift/5422a6b58db22
7512300014d/fuse/bin/install --version 1.0.0\"": {:unsetenv_others=>true, :close_others=>true, :in=>"/dev/null", :chdir=>"/var/lib/openshift/5422a6b58db227512300014d/fuse", :out=>#<IO:fd 12>, :err=>#<IO:fd 8>}
September 24 07:11:44 INFO oo_spawn buffer(11/) CLIENT_RESULT: Using provided password from domain

September 24 07:11:44 INFO oo_spawn buffer(11/) CLIENT_RESULT: Using provided url from domain

September 24 07:11:44 INFO oo_spawn buffer(11/) CLIENT_RESULT: 

September 24 07:11:44 INFO oo_spawn buffer(11/) CLIENT_RESULT: Application fuse9 created successfully.  Please make note of these credentials:

September 24 07:11:44 INFO oo_spawn buffer(11/) CLIENT_RESULT: 

September 24 07:11:44 INFO oo_spawn buffer(11/) CLIENT_RESULT:    Console URL:        https://fuse-foobar.apps.sample.com

September 24 07:11:44 INFO oo_spawn buffer(11/) CLIENT_RESULT:    Console User:       [HIDDEN]
CLIENT_RESULT:    Console Password:   [HIDDEN]

September 24 07:11:44 INFO oo_spawn buffer(11/) CLIENT_RESULT:    Zookeeper URL:      fuse-foobar.apps.sample.com:56081

September 24 07:11:44 INFO oo_spawn buffer(11/) CLIENT_RESULT:    Zookeeper Password: [HIDDEN]

September 24 07:11:44 INFO oo_spawn buffer(11/) CART_PROPERTIES: username=[HIDDEN]

September 24 07:11:44 INFO oo_spawn buffer(11/) CART_PROPERTIES: password=[HIDDEN]

September 24 07:11:47 INFO oo_spawn buffer(11/) CLIENT_RESULT: 

September 24 07:11:47 INFO oo_spawn buffer(11/) client: JAVA_HOME not set; results may vary

September 24 07:11:56 INFO oo_spawn buffer(11/) retrying (attempt 1) ...
September 24 07:11:56 INFO oo_spawn buffer(11/) 
September 24 07:12:01 INFO oo_spawn buffer(11/) retrying (attempt 2) ...

September 24 07:12:06 INFO oo_spawn buffer(11/) retrying (attempt 3) ...

September 24 07:12:11 INFO oo_spawn buffer(11/) retrying (attempt 4) ...

September 24 07:12:16 INFO oo_spawn buffer(11/) retrying (attempt 5) ...
September 24 07:12:16 INFO oo_spawn buffer(11/) 

September 24 07:12:16 INFO oo_spawn buffer(6/) Failed to get the session.

September 24 07:12:26 INFO oo_spawn buffer(11/) client: JAVA_HOME not set; results may vary

September 24 07:12:33 INFO oo_spawn buffer(11/) retrying (attempt 1) ...

September 24 07:12:38 INFO oo_spawn buffer(11/) retrying (attempt 2) ...

September 24 07:12:43 INFO oo_spawn buffer(11/) retrying (attempt 3) ...

September 24 07:12:48 INFO oo_spawn buffer(11/) retrying (attempt 4) ...

September 24 07:12:53 INFO oo_spawn buffer(11/) retrying (attempt 5) ...

September 24 07:12:53 INFO oo_spawn buffer(6/) Failed to get the session.

September 24 07:13:03 INFO oo_spawn buffer(11/) client: JAVA_HOME not set; results may vary

September 24 07:13:12 INFO oo_spawn buffer(11/) retrying (attempt 1) ...

September 24 07:13:17 INFO oo_spawn buffer(11/) retrying (attempt 2) ...

September 24 07:13:22 INFO oo_spawn buffer(11/) retrying (attempt 3) ...

September 24 07:13:27 INFO oo_spawn buffer(6/) 21836 [pool-2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [/127.10.147.129:23101, DSA, e8:d4:98:4d:b4:8f:a8:84:15:8a:2f:67:21:e8:7d:02] presented unverified {} key: {}

September 24 07:13:37 INFO oo_spawn buffer(11/) true

September 24 07:13:37 INFO oo_spawn buffer(11/) client: JAVA_HOME not set; results may vary

September 24 07:13:39 INFO oo_spawn buffer(6/) 1319 [pool-2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [/127.10.147.129:23101, DSA, e8:d4:98:4d:b4:8f:a8:84:15:8a:2f:67:21:e8:7d:02] presented unverified {} key: {}

September 24 07:14:14 INFO oo_spawn buffer(11/) CLIENT_RESULT: 

September 24 07:14:14 INFO oo_spawn buffer(11/) CLIENT_MESSAGE: Waiting for container to be provisioned...
September 24 07:14:14 INFO oo_spawn buffer(11/) client: JAVA_HOME not set; results may vary

September 24 07:14:17 INFO oo_spawn buffer(6/) 1313 [pool-2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [/127.10.147.129:23101, DSA, e8:d4:98:4d:b4:8f:a8:84:15:8a:2f:67:21:e8:7d:02] presented unverified {} key: {}

September 24 07:14:17 INFO oo_spawn buffer(11/) true

September 24 07:14:17 INFO oo_spawn buffer(11/) client: JAVA_HOME not set; results may vary

September 24 07:14:18 INFO oo_spawn buffer(6/) 606 [pool-2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [/127.10.147.129:23101, DSA, e8:d4:98:4d:b4:8f:a8:84:15:8a:2f:67:21:e8:7d:02] presented unverified {} key: {}

September 24 07:15:29 INFO oo_spawn running quota --always-resolve -w 5422a6b58db227512300014d: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 12>, :err=>#<IO:fd 8>}
September 24 07:15:29 INFO oo_spawn buffer(11/) Disk quotas for user 5422a6b58db227512300014d (uid 5415): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
      /dev/vda1  114568       0 5242880            1525       0   80000        

September 24 07:15:31 INFO oo_spawn running /usr/bin/pkill -9 -u 5415: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 13>, :err=>#<IO:fd 11>}
September 24 07:15:31 INFO oo_spawn running /usr/bin/pgrep -u 5415: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 13>, :err=>#<IO:fd 11>}
September 24 07:15:31 INFO oo_spawn running /usr/bin/pkill -9 -u 5415: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 13>, :err=>#<IO:fd 11>}
September 24 07:15:31 INFO oo_spawn running /usr/bin/pgrep -u 5415: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 13>, :err=>#<IO:fd 11>}
September 24 07:15:31 INFO oo_spawn running rm /var/lib/openshift/.last_access/5422a6b58db227512300014d: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 13>, :err=>#<IO:fd 11>}
September 24 07:15:32 INFO oo_spawn buffer(8/) rm: cannot remove `/var/lib/openshift/.last_access/5422a6b58db227512300014d': No such file or directory

September 24 07:15:32 INFO oo_spawn running /usr/bin/pkill -9 -u 5415: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 13>, :err=>#<IO:fd 11>}
September 24 07:15:32 INFO oo_spawn running /usr/bin/pgrep -u 5415: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 13>, :err=>#<IO:fd 11>}
September 24 07:15:32 INFO oo_spawn running /usr/bin/ipcs -c -m 2> /dev/null: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 13>, :err=>#<IO:fd 11>}
September 24 07:15:32 INFO oo_spawn buffer(12/)

            janstey@redhat.com Jonathan Anstey
            maschmid@redhat.com Marek Schmidt
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: