Uploaded image for project: 'WildFly Core'
  1. WildFly Core
  2. WFCORE-1560

Cli calls leak resources in Host Controller when repeatedly calling jboss-cli.sh

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 3.0.0.Beta17
    • 2.0.8.Final
    • Management
    • None
    • Hide

      Setup a wildfly-10.0.0.Final cluster with a dedicated domain controller and two slaves.
      Run management commands on the cluster in a loop.
      In my case the following commands are run in a loop:

      • get all hosts connected
      • iterate over hosts
      • get all deployments of a single host and select a specific deployment
      • get the number of sessions for the selected deployment

      hosts=/opt/wildfly-10.0.0.Final-DC/bin/./jboss-cli.sh --connect --user="username" --password="password" --command=":read-children-names(child-type=host)"
      for i in $hosts
      do
      host=$(echo $i | cut -d "\"" -f 2)
      deployment100=$(/opt/wildfly-10.0.0.Final-DC/bin/./jboss-cli.sh --connect --user="username" --password="password" --command="/host=$host/server=$host-100 :read-children-names(child-type=deployment)" | grep deploymentname | cut -d "\"" -f 4)
      sessions=$(/opt/wildfly-10.0.0.Final-DC/bin/./jboss-cli.sh --connect --user="username" --password="password" --command="/host=$host/server=$host-100/deployment=$deployment100/subsystem=undertow :read-attribute(name=active-sessions)")
      done

      Show
      Setup a wildfly-10.0.0.Final cluster with a dedicated domain controller and two slaves. Run management commands on the cluster in a loop. In my case the following commands are run in a loop: get all hosts connected iterate over hosts get all deployments of a single host and select a specific deployment get the number of sessions for the selected deployment hosts=/opt/wildfly-10.0.0.Final-DC/bin/./jboss-cli.sh --connect --user="username" --password="password" --command=":read-children-names(child-type=host)" for i in $hosts do host=$(echo $i | cut -d "\"" -f 2) deployment100=$(/opt/wildfly-10.0.0.Final-DC/bin/./jboss-cli.sh --connect --user="username" --password="password" --command="/host=$host/server=$host-100 :read-children-names(child-type=deployment)" | grep deploymentname | cut -d "\"" -f 4) sessions=$(/opt/wildfly-10.0.0.Final-DC/bin/./jboss-cli.sh --connect --user="username" --password="password" --command="/host=$host/server=$host-100/deployment=$deployment100/subsystem=undertow :read-attribute(name=active-sessions)") done
    • User Experience
    • Medium

    Description

      When executing management commands using jboss-cli.sh against the domain controller of a cluster repeatedly the host controller uses up more and more memory in oldgen. After several thousands of runs of jboss-cli the host controller eventually becomes unresponsive (see attached picture for memory consumption, dc became entirely unresponsive at roughly 6:30am):

      [root@dc broken]# /opt/wildfly-10.0.0.Final-DC/bin/./jboss-cli.sh --connect --user="username" --password="password" --command=":read-children-names(child-type=host)"
      Failed to connect to the controller: The controller is not available at xx.xx.xx.xx:9993: java.net.ConnectException: WFLYPRT0023: Could not connect to https-remoting://xx.xx.xx.xx:9993. The connection timed out: WFLYPRT0023: Could not connect to https-remoting://xx.xx.xx.xx:9993. The connection timed out

      I discovered the issue when testing whether https://issues.jboss.org/browse/WFCORE-974 was actually resolved in wildfly-10.0.0.Final as advertised. I can confirm that the issue is different, since no OOM-Exceptions are thrown. However the DC still becomes useless, since it won't accept any connections anymore. I will check whether the work-around from WFCORE-974 applies to this issue as well. However the work-around from WFCORE-974 doesn't fix this issue.

      Please note that the attached logs are UTC, while the monitoring is UTC+2. Also the collection values are misleading since I haven't adapted my monitoring to the new output of jstat in JDK8. PU and PC are thus MU and MC.

      Attachments

        1. console-dc.log
          12 kB
        2. host-controller.log
          10 kB
        3. JVM-DC.png
          JVM-DC.png
          192 kB
        4. process-controller.log
          0.2 kB

        Activity

          People

            kwills@redhat.com Ken Wills
            michael.noack Michael Noack (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: