Uploaded image for project: 'OptaPlanner'
  1. OptaPlanner
  2. PLANNER-395

Benchmark report: performance statistic with aggregation of reports run on different hardware should have a warning

XMLWordPrintable

    • NEW
    • NEW

      Unfair benchmark comparisons should stick out like a sore thumb.

      • Reuse/copy the warning system that is already in place in MemoryUseProblemStatistic#fillWarningList(). It shows up above a memory use problem statistic if you enable that statistic and run a parallel count of at least 2 (or if auto resolves that way). Try that first.
        • This will be a summary statistic warning, not a problem statistic warning. So it's maybe not possible to generalize all code.
        • Of both types there will further additional warnings in the future, for example if they use a terrible SA parameter and it can detect it in some statistic, it will warn about that.
        • Reuse the same warning style, css class etc, as far as possible.
      • This hardware warning shows up if the hardware description is different. For example, if you run one report on "Pentium 4 300Gz" and another on "Pentium 3 500Gz" or even "Pentium 4 233Gz" . But not if they have the same hardware description but are different machines. For example: running them on a room of identical classroom computers, should not lead to a warning in the aggregation
      • PlannerBenchmarkResult has an operatingSystem field, but no hardwareDescription field yet... I am not even sure if it's possible to get that in a pure Java way that works on JDK 6 (and doesn't import com.sun stuff). Do some research. If not possible, reject this issue.
        • Look at PlannerBenchmarkResult#createMergeSingleton(). It will set a field if the 2 reports being merged have the same value for it, otherwise it sets the field as null, to signal that it doesn't know. Maybe this null suffices (I hope it does, as that's less code to maintain), maybe it's needed to even hold a set of used values in the original reports...

            Unassigned Unassigned
            gdesmet@redhat.com Geoffrey De Smet (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: