Uploaded image for project: 'mod_cluster'
  1. mod_cluster
  2. MODCLUSTER-288

SystemMemoryUsageLoadMetric is not correct on Linux/Unix

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 1.0.10, 1.1.0.Final, 1.1.1.Final, 1.2.0.Final, 1.3.0.Alpha1
    • Fix Version/s: 1.3.0.Final
    • Component/s: None
    • Labels:
      None
    • Environment:

      Linux/Unix operating systems with standard memory management

      Description

      I am convinced that the SystemMemoryUsageLoadMetric.java is of no use to anyone.

      I have tested it by allocating some memory outside the JVM control with NativeMemoryStress.c.

      The results from Mod_cluster are such that the system is almost fully loaded all the time despite the fact that I work with 8GB RAM box with almost nothing running on it.
      This picture displays time on X axis and (100-(Load from Mod_cluster)) on Y axis. As you may observe, all 4 nodes appear to be under full load all the time and there is a spike on each of them. This spike represents a period of time when the aforementioned NativeMemoryStress.c was called with the following setting:

         megabytes    500;
         milliseconds 60000;
      

      What had happened is that kernel has cleared some cache and garbage data from the RAM in order to accommodate new demands, so by the time I freed the memory, there was all of sudden actually more free RAM then before.

      The actual problem with the current SystemMemoryUsageLoadMetric.java implementation using OperatingSystemMXBean is that this code:

      +++
        public static final String FREE_MEMORY = "FreePhysicalMemorySize";
        public static final String TOTAL_MEMORY = "TotalPhysicalMemorySize";
      +++
        double free = ((Number) this.server.getAttribute(name, FREE_MEMORY)).doubleValue();
        double total = ((Number) this.server.getAttribute(name, TOTAL_MEMORY)).doubleValue();
        return (total - free) / total;
      +++
      

      Is somewhat similar to this (on 8GB RAM machine with almost nothing running on at the moment):

      [karm@killerbox karm]$ free -m
                   total       used       free     shared    buffers     cached
      Mem:          7933       6755       1177          0         44        141
      -/+ buffers/cache:       6569       1363
      Swap:        16383         58      16325
      

      Obviously, this does not mean that I could get only 1177MB of RAM if I want to. The "free" actually means "wasted". The majority of RAM is being used as cache. If I ask for e.g. 2GB, I am gonna get them, because kernel will simply sacrifice some cache, clears some garbage from RAM and free up space so as to accommodate my demands.

      So, according to my opinion, on a sane, well-behaving caching operating system, the SystemMemoryUsageLoadMetric.java will always show ~ 98% RAM usage.

      Due to this fact, this metric is unlikely to be of some use to anyone on Linux/UNIX. What do you think?

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  rhusar Radoslav Husar
                  Reporter:
                  mbabacek Michal Karm Babacek
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: