Uploaded image for project: 'ModeShape'
  1. ModeShape
  2. MODE-1323

Performance benchmark should graph results from all modules

    Details

    • Type: Enhancement
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Done
    • Affects Version/s: 3.0.0.Alpha1
    • Fix Version/s: 3.0.0.Alpha1
    • Component/s: Performance
    • Labels:
      None
    • Similar Issues:
      Show 10 results 

      Description

      The module-specific charts are good (though it'd be nice if the barchart.html page showed all of the charts there (with titles).

      But the real value is going to be a single page that has one graph for each test, where each graph contained the results from all of the tested modules. The comparison between tested modules is the valuable part of this. The utils/modeshape-connector-benchmark had such an output, and it's very valuable. It has a single HTML template with a $DIV$ section that was replaced with the chart data in the code.

      Best of all, Google charts now has support for box charts, which is exactly what we want! The question really becomes: what's the best way to produce the charts within the performance test framework? Since each test outputs the data (in the 'perf-test.txt' file), perhaps the aggregate HTML page can be generated by simply reading those files (whichever ones are there).

      (BTW, I don't think we need to put the outliers on our charts. There's no easy way to do that with Google Charts, since you need a separate data series for each pair of outliers for each candlestick.)

        Gliffy Diagrams

          Activity

          rhauch Randall Hauch created issue -
          rhauch Randall Hauch made changes -
          Field Original Value New Value
          Description The module-specific charts are good (though it'd be nice if the {{barchart.html}} page showed all of the charts there (with titles).

          But the real value is going to be a single page that has one graph for each test, where each graph contained the results from all of the tested modules. The comparison between tested modules is the valuable part of this.

          I know the Google graph library doesn't have support for error-bars or 5-number summaries, but there may be alternatives. For example [D3.js|http://mbostock.github.com/d3/ex/box.html] library does. The HTML is a quite a bit more complicated, but it actually just reads the raw data from a CSV file and then computes the statistics and plots the values.

          Generating a single page with multiple graphs may be more complicated, but I think it'd be worth it. We'll probably want to publish the results, and these graphs will be the best way to do this.

          Perhaps there are even better options than a bar-chart. Consider [bullet charts|http://mbostock.github.com/d3/ex/bullet.html] from D3.js. Again, if we created a separate bullet chart for each test, we could show multiple bullets lines for the different tested modules. The data actually is pretty straightforward; see the [example data file|http://mbostock.github.com/d3/ex/bullets.json].

          Thoughts??
          The module-specific charts are good (though it'd be nice if the {{barchart.html}} page showed all of the charts there (with titles).

          But the real value is going to be a single page that has one graph for each test, where each graph contained the results from all of the tested modules. The comparison between tested modules is the valuable part of this. The {{utils/modeshape-connector-benchmark}} had such an output, and it's very valuable. It has a single [HTML template|] with a {{$DIV$}} section that was [replaced with the chart data in the code|https://github.com/ModeShape/modeshape/blob/master/utils/modeshape-connector-benchmark/src/test/java/org/modeshape/test/benchmark/ConnectorBenchmarkTest.java#L164].

          Best of all, Google charts now has support for box charts, which is exactly what we want. The question really becomes: what's the best way to produce the charts? Since each test outputs the data (in the 'perf-test.txt' file), perhaps the aggregate HTML page can be generated by simply reading those files (whichever ones are there).
          rhauch Randall Hauch made changes -
          Comment [ Although the bullet charts are nice looking, I think the box-charts are the most accurate and will be most familiar to people, and thus may be the best way. The biggest challenge is probably going to be finding a charting library that requires no local install; that's why I focused on JavaScript libraries so far.

          Google charts now have [error bars|http://code.google.com/apis/chart/image/docs/gallery/bar_charts.html#markerplacementtarget] and [candlestick bars|http://code.google.com/apis/chart/image/docs/gallery/bar_charts.html#gcharts_candlestick_markers]. Candlesticks might be perfect if combined with a [circle or horizontal line|http://code.google.com/apis/chart/image/docs/gallery/bar_charts.html#gcharts_shape_markers] or even a [simple line|] to denote the median value.

          ]
          rhauch Randall Hauch made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          Description The module-specific charts are good (though it'd be nice if the {{barchart.html}} page showed all of the charts there (with titles).

          But the real value is going to be a single page that has one graph for each test, where each graph contained the results from all of the tested modules. The comparison between tested modules is the valuable part of this. The {{utils/modeshape-connector-benchmark}} had such an output, and it's very valuable. It has a single [HTML template|] with a {{$DIV$}} section that was [replaced with the chart data in the code|https://github.com/ModeShape/modeshape/blob/master/utils/modeshape-connector-benchmark/src/test/java/org/modeshape/test/benchmark/ConnectorBenchmarkTest.java#L164].

          Best of all, Google charts now has support for box charts, which is exactly what we want. The question really becomes: what's the best way to produce the charts? Since each test outputs the data (in the 'perf-test.txt' file), perhaps the aggregate HTML page can be generated by simply reading those files (whichever ones are there).
          The module-specific charts are good (though it'd be nice if the {{barchart.html}} page showed all of the charts there (with titles).

          But the real value is going to be a single page that has one graph for each test, where each graph contained the results from all of the tested modules. The comparison between tested modules is the valuable part of this. The {{utils/modeshape-connector-benchmark}} had such an output, and it's very valuable. It has a single [HTML template|https://github.com/ModeShape/modeshape/blob/master/utils/modeshape-connector-benchmark/src/test/resources/performance/benchmark.html] with a {{$DIV$}} section that was [replaced with the chart data in the code|https://github.com/ModeShape/modeshape/blob/master/utils/modeshape-connector-benchmark/src/test/java/org/modeshape/test/benchmark/ConnectorBenchmarkTest.java#L164].

          Best of all, Google charts now has support for [box charts|http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts], which is exactly what we want! The question really becomes: what's the best way to produce the charts within the performance test framework? Since each test outputs the data (in the 'perf-test.txt' file), perhaps the aggregate HTML page can be generated by simply reading those files (whichever ones are there).

          (BTW, I don't think we need to put the outliers on our charts. There's no easy way to do that with Google Charts, since you need a separate data series for each pair of outliers for each candlestick.)
          Hide
          hchiorean Horia Chiorean added a comment - - edited

          So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable.

          Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure

          Show
          hchiorean Horia Chiorean added a comment - - edited So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable. Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure
          Hide
          hchiorean Horia Chiorean added a comment -

          I'm no chart wiz unfortunately, but from the example (http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ?

          Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1.

          Show
          hchiorean Horia Chiorean added a comment - I'm no chart wiz unfortunately, but from the example ( http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts ), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ? Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1.
          Hide
          rhauch Randall Hauch added a comment -

          So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable.

          Yes, so there'd be a chart for the "OpenAnonymousSessionTestSuite" (maybe called "Open Anonymous Session") that has boxes for the results from ModeShape 2.7, ModeShape 3.0, Jackrabbit 2.3.2 (and any other versions/impls that are added). There'd also be additional charts for the other test suites. (IIRC the connector benchmark results places the charts in a table, with 2 or 3 columns and as many rows as necessary.)

          Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure

          I agree that this may require a dedicated reporting module that always runs last. One alternative, though, maybe be to try adding the chart generation logic (not sure what form that takes) to the end of the 'test' phase or in the assembly phase.

          Show
          rhauch Randall Hauch added a comment - So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable. Yes, so there'd be a chart for the "OpenAnonymousSessionTestSuite" (maybe called "Open Anonymous Session") that has boxes for the results from ModeShape 2.7, ModeShape 3.0, Jackrabbit 2.3.2 (and any other versions/impls that are added). There'd also be additional charts for the other test suites. (IIRC the connector benchmark results places the charts in a table, with 2 or 3 columns and as many rows as necessary.) Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure I agree that this may require a dedicated reporting module that always runs last. One alternative, though, maybe be to try adding the chart generation logic (not sure what form that takes) to the end of the 'test' phase or in the assembly phase.
          Hide
          rhauch Randall Hauch added a comment -

          I'm no chart wiz unfortunately, but from the example (http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ?

          Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1.

          I think that's controlled with the range: for example "?chxr=0,10,260" (for one chart axis).

          Here's an example: http://chart.apis.google.com/chart?cht=bvs&chs=500x400&chxr=0,1,15%7C1,0,50&chxt=y,x&chd=t0:-1,5,10,7,12,-1%7C-1,25,30,27,24,-1%7C-1,40,45,47,39,-1%7C-1,55,63,59,80,-1%7C-1,30,40,35,30,-1%7C-1,-1,5,70,90,-1%7C-1,-1,-1,80,5,-1&chm=F,FF9900,0,1:4,40%7CH,0CBF0B,0,1:4,1:20%7CH,000000,4,1:4,1:40%7CH,0000FF,3,1:4,1:20

          But I don't see how varying the range actually changes the data; seems to just change the axis values.

          Show
          rhauch Randall Hauch added a comment - I'm no chart wiz unfortunately, but from the example ( http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts ), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ? Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1. I think that's controlled with the range: for example " ?chxr=0,10,260 " (for one chart axis). Here's an example: http://chart.apis.google.com/chart?cht=bvs&chs=500x400&chxr=0,1,15%7C1,0,50&chxt=y,x&chd=t0:-1,5,10,7,12,-1%7C-1,25,30,27,24,-1%7C-1,40,45,47,39,-1%7C-1,55,63,59,80,-1%7C-1,30,40,35,30,-1%7C-1,-1,5,70,90,-1%7C-1,-1,-1,80,5,-1&chm=F,FF9900,0,1:4,40%7CH,0CBF0B,0,1:4,1:20%7CH,000000,4,1:4,1:40%7CH,0000FF,3,1:4,1:20 But I don't see how varying the range actually changes the data; seems to just change the axis values.
          Hide
          hchiorean Horia Chiorean added a comment - - edited
          • added new Maven module - perf-tests-report which aggregates the data from the individual modules
          • added a new report which generates Google box chart links for each of the tests ran, comparing the performance of each repository. They will be located in perf-tests-report/target/test-classes/google
          Show
          hchiorean Horia Chiorean added a comment - - edited added new Maven module - perf-tests-report which aggregates the data from the individual modules added a new report which generates Google box chart links for each of the tests ran, comparing the performance of each repository. They will be located in perf-tests-report/target/test-classes/google added a new box-plot report generator ( http://informationandvisualization.de/blog/box-plot ) which generates 1 report per each ran test, comparing each of the repositories. They will be located at perf-tests-report/target/test-classes/boxplot
          hchiorean Horia Chiorean made changes -
          Status Open [ 1 ] Pull Request Sent [ 10011 ]
          Git Pull Request https://github.com/ModeShape/modeshape-performance/pull/5
          Hide
          hchiorean Horia Chiorean added a comment - - edited

          Report generated by boxplot - ignore the html, you need the zip file

          Show
          hchiorean Horia Chiorean added a comment - - edited Report generated by boxplot - ignore the html, you need the zip file
          hchiorean Horia Chiorean made changes -
          Attachment SmallFileWriteTestSuite.html [ 12349685 ]
          Hide
          hchiorean Horia Chiorean added a comment - - edited

          Correct boxplot report (needs js and css as well)

          As a comparison, the same view using Google charts (unfortunately scaling is poor in G Charts): http://tinyurl.com/7kazns6

          Show
          hchiorean Horia Chiorean added a comment - - edited Correct boxplot report (needs js and css as well) As a comparison, the same view using Google charts (unfortunately scaling is poor in G Charts): http://tinyurl.com/7kazns6
          hchiorean Horia Chiorean made changes -
          Attachment boxplot.zip [ 12349687 ]
          rhauch Randall Hauch made changes -
          Attachment SmallFileWriteTestSuite.html [ 12349685 ]
          Hide
          hchiorean Horia Chiorean added a comment - - edited

          Attached sample report with D3.

          There is still a problem (and I'm tending to be believe that this problem cannot be solved with box charts): if the values from the different series are wide-enough apart, the "lower" series will not be displayed properly

          In the attached CreateChildNodesTestSuite.html, the series have the following values:

          • [146, 117, 127, 109, 111, 149, 176, ...] (rounded)
          • [20, 21, 22, 26, ...] (rounded)
          • [4.0, 4.4, 4.24, 4.28, ...]

          the last 2 series and especially the last one, will not be displayed clearly enough on a [0, 176] scale.

          Show
          hchiorean Horia Chiorean added a comment - - edited Attached sample report with D3. There is still a problem (and I'm tending to be believe that this problem cannot be solved with box charts): if the values from the different series are wide-enough apart, the "lower" series will not be displayed properly In the attached CreateChildNodesTestSuite.html, the series have the following values: [146, 117, 127, 109, 111, 149, 176, ...] (rounded) [20, 21, 22, 26, ...] (rounded) [4.0, 4.4, 4.24, 4.28, ...] the last 2 series and especially the last one, will not be displayed clearly enough on a [0, 176] scale.
          hchiorean Horia Chiorean made changes -
          Attachment d3_report.zip [ 12349739 ]
          Hide
          hchiorean Horia Chiorean added a comment -

          Merged into master

          Show
          hchiorean Horia Chiorean added a comment - Merged into master
          hchiorean Horia Chiorean made changes -
          Status Pull Request Sent [ 10011 ] Resolved [ 5 ]
          Resolution Done [ 1 ]
          rhauch Randall Hauch made changes -
          Fix Version/s 3.0.0.Alpha1 [ 12318454 ]
          Fix Version/s 3.0.0.Alpha2 [ 12318455 ]
          Git Pull Request https://github.com/ModeShape/modeshape-performance/pull/5 https://github.com/ModeShape/modeshape-performance/pull/5
          rhauch Randall Hauch made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              hchiorean Horia Chiorean
              Reporter:
              rhauch Randall Hauch
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development