Uploaded image for project: 'OptaPlanner'
  1. OptaPlanner
  2. PLANNER-58

Planner: Benchmark report should also show percentage versus worst dataset (and an average of that percentage)

XMLWordPrintable

      Managers like to know much better a result it in relative terms.
      This often boils down to comparing the results of the construction heuristic with the results of 1 or more full-fledged configuration (construction heuristic + metaheuristic).

      So for each dataset for each solver configuration, we can calculate a percentage: ((- Worst solver's score) - (- this solver's score)) / (- Worst solver's score). For the Worst's solver's score this will be 0%.
      If the worst solvers score is 0, then it should be NaN or empty.

      Then, for each solver configuration, we can calculate an "average percentage". Note that is is calculated based on the percentages, not the average of the scores! If any datasets have NaN or empty, that result should be discarded (which means the entire column gets discarded).

      Classes to look into:

      • SingleBenchmark should have the betterThanWorstScorePercentage. TODO: find a better name then that
      • SolverBenchmark should have the averageBetterThanWorstScorePercentage TODO: find a better name then that
      • PlannerStatistic should output a graph of it. Copy paste from writeBestScoreSummaryChart. (add it after the WinningScoreDifference code)
      • index.html.ftl should write the numbers in the report (add it after WinningScoreDifference)

            gdesmet@redhat.com Geoffrey De Smet (Inactive)
            gdesmet@redhat.com Geoffrey De Smet (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: