Creating multiple openshift containers at once overcommits nodes / exceeds node capacity

Marek Schmidt 2014-08-11 03:09:37 EDT

Description of problem:

Creating new openshift containers via Fuse console allows creation of multiple containers at once. This will create multiple node creation requests and the broker will attempt to assign all of these requests to the same node. This will fail with

2014-08-11 02:50:03.643 [ERROR] Gear creation failed. The server node2.hosts.sample.com that was chosen to run your gear has already exceeded its capacity. Try your request again.

Since Fuse does not appear to understand this message, the container creation will not be attempted again and the container is not created.

Version-Release number of selected component (if applicable):
OpenShift Enterprise 2.1.3
openshift-origin-cartridge-fuse-6.1.0.redhat.387-6.el6op.noarch.rpm

How reproducible:
always

Steps to Reproduce:
1. Setup two OSE nodes, edit their /etc/openshift/resource_limits.conf

max_active_gears=2
no_overcommit_active=true

2. Create a fuse gear

3. Login into the fuse console, attempt to create 3 new openshift containers

Actual results:
Only 1 or 2 of the three requested containers are created. And the "
Gear creation failed. The server .* that was chosen to run your gear has already exceeded its capacity. Try your request again. " message can be seen in the broker logs.

Expected results:
3 new containers should be created.

Additional info:

I am not sure whether this counts as a broker issue (should try to evenly distribute the new gear creation requests), or the Fuse cartridge (fuse not understanding the "Try your request again." error.

Note that if "no_overcommit_active=false", the node will turn out to be overcommited in this case (which for a fuse cart is problematic, as it uses most of the assigned memory, so it will likely fail with out of memory issues)

Marek Schmidt 2014-08-11 03:11:43 EDT
Summary: Creating multiple openshift containers at once overcommits nodes / exceeds node capacity → [JBossFuse] Creating multiple openshift containers at once overcommits nodes / exceeds node capacity