Uploaded image for project: 'mod_cluster'
  1. mod_cluster
  2. MODCLUSTER-322

Using AverageSystemLoadMetric can improperly cause a Load Factor of 0

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: MOD_CLUSTER_1_0_10_GA_CP04, 1.2.1.Final
    • Component/s: None
    • Labels:
      None
    • Environment:

      *JBoss Enterprise Application Platform 5
      *Apache httpd
      *mod_cluster 1.0.10.GA_CP02 or 1.2.1

      Description

      It looks like AverageSystemLoadMetric is not properly implemented. When using it, mod_cluster may always report a load factor of 0, thus making the JBoss node unreachable from Apache. We've tested with a simple web app that checks the underlying MXBean SystemLoadAverage:

      Double.class.cast(server.getAttribute(ObjectName.getInstance(ManagementFactory.OPERATING_SYSTEM_MXBEAN_NAME), "SystemLoadAverage")).doubleValue()

      This info is grabbed pretty much the same way mod_cluster does, but these calls appeared to work just fine outside of mod_cluster as it returns the following values from my test app:

      12:08:26,519 INFO [STDOUT] From MBeanServer: 1.81640625

      So the issue does not appear to be necessarily with the underlying JDK/MXBean call but with how mod_cluster is handling the data grabbed from it. But the root cause here appears to be that a value above 1 is being returned, and it looks like mod_cluster is expecting metrics to return a 0-1 percentile based range.

      The way the load is determined allows the AverageSystemLoadMetric to improperly exceed its weight. For example if it were weighted as 2 and another metric was weighted at 1 (say RequestCountLoadMetric with a capacity of 1000), then AverageSystemLoadMetric should only be able to account for 67% of the load. But here we can see AverageSystemLoadMetric can out run its weight and really account for 100% of the load. So let's say AverageSystemLoadMetric is the above seen 1.81640625 value with 100 requests/second, putting RequestCountLoadMetric at .1 load, so DynamicLoadBalanceFactorProvider would calculate the load factor like so:

      int load = (int) Math.round(100 * totalWeightedLoad / totalWeight);
      int load = (int) Math.round(100 * (1.81640625 * 2 + 0.1) / 3);
      load = 124.4

      And that gets truncated down to 100 so AverageSystemLoadMetric comes to represent really all of the load. But if a 2 weight metric is at its max and a 1 weight metric is at .1%, then their total load should just be ~70-71%.

      S is mod_cluster assuming that the SystemLoadAverage will always be between 0 and 1? Does it look like mod_cluster is not properly scaling this metric? Do we know a definite max return to expect from OperatingSystemMXBean.getSystemLoadAverage() so that this metric can be scaled more in line with the other percentile based ones? Or should a user definable max capacity be implemented into this metric as it is with others?

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  pferraro Paul Ferraro
                  Reporter:
                  aogburn Aaron Ogburn
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: