ModeShape
  1. ModeShape
  2. MODE-1323

Performance benchmark should graph results from all modules

    Details

    • Type: Enhancement Enhancement
    • Status: Closed Closed (View Workflow)
    • Priority: Critical Critical
    • Resolution: Done
    • Affects Version/s: 3.0.0.Alpha1
    • Fix Version/s: 3.0.0.Alpha1
    • Component/s: Performance
    • Labels:
      None
    • Similar Issues:
      Show 10 results 

      Description

      The module-specific charts are good (though it'd be nice if the barchart.html page showed all of the charts there (with titles).

      But the real value is going to be a single page that has one graph for each test, where each graph contained the results from all of the tested modules. The comparison between tested modules is the valuable part of this. The utils/modeshape-connector-benchmark had such an output, and it's very valuable. It has a single HTML template with a $DIV$ section that was replaced with the chart data in the code.

      Best of all, Google charts now has support for box charts, which is exactly what we want! The question really becomes: what's the best way to produce the charts within the performance test framework? Since each test outputs the data (in the 'perf-test.txt' file), perhaps the aggregate HTML page can be generated by simply reading those files (whichever ones are there).

      (BTW, I don't think we need to put the outliers on our charts. There's no easy way to do that with Google Charts, since you need a separate data series for each pair of outliers for each candlestick.)

        Activity

        Randall Hauch
        made changes -
        Field Original Value New Value
        Description The module-specific charts are good (though it'd be nice if the {{barchart.html}} page showed all of the charts there (with titles).

        But the real value is going to be a single page that has one graph for each test, where each graph contained the results from all of the tested modules. The comparison between tested modules is the valuable part of this.

        I know the Google graph library doesn't have support for error-bars or 5-number summaries, but there may be alternatives. For example [D3.js|http://mbostock.github.com/d3/ex/box.html] library does. The HTML is a quite a bit more complicated, but it actually just reads the raw data from a CSV file and then computes the statistics and plots the values.

        Generating a single page with multiple graphs may be more complicated, but I think it'd be worth it. We'll probably want to publish the results, and these graphs will be the best way to do this.

        Perhaps there are even better options than a bar-chart. Consider [bullet charts|http://mbostock.github.com/d3/ex/bullet.html] from D3.js. Again, if we created a separate bullet chart for each test, we could show multiple bullets lines for the different tested modules. The data actually is pretty straightforward; see the [example data file|http://mbostock.github.com/d3/ex/bullets.json].

        Thoughts??
        The module-specific charts are good (though it'd be nice if the {{barchart.html}} page showed all of the charts there (with titles).

        But the real value is going to be a single page that has one graph for each test, where each graph contained the results from all of the tested modules. The comparison between tested modules is the valuable part of this. The {{utils/modeshape-connector-benchmark}} had such an output, and it's very valuable. It has a single [HTML template|] with a {{$DIV$}} section that was [replaced with the chart data in the code|https://github.com/ModeShape/modeshape/blob/master/utils/modeshape-connector-benchmark/src/test/java/org/modeshape/test/benchmark/ConnectorBenchmarkTest.java#L164].

        Best of all, Google charts now has support for box charts, which is exactly what we want. The question really becomes: what's the best way to produce the charts? Since each test outputs the data (in the 'perf-test.txt' file), perhaps the aggregate HTML page can be generated by simply reading those files (whichever ones are there).
        Randall Hauch
        made changes -
        Comment [ Although the bullet charts are nice looking, I think the box-charts are the most accurate and will be most familiar to people, and thus may be the best way. The biggest challenge is probably going to be finding a charting library that requires no local install; that's why I focused on JavaScript libraries so far.

        Google charts now have [error bars|http://code.google.com/apis/chart/image/docs/gallery/bar_charts.html#markerplacementtarget] and [candlestick bars|http://code.google.com/apis/chart/image/docs/gallery/bar_charts.html#gcharts_candlestick_markers]. Candlesticks might be perfect if combined with a [circle or horizontal line|http://code.google.com/apis/chart/image/docs/gallery/bar_charts.html#gcharts_shape_markers] or even a [simple line|] to denote the median value.

        ]
        Randall Hauch
        made changes -
        Priority Major [ 3 ] Critical [ 2 ]
        Description The module-specific charts are good (though it'd be nice if the {{barchart.html}} page showed all of the charts there (with titles).

        But the real value is going to be a single page that has one graph for each test, where each graph contained the results from all of the tested modules. The comparison between tested modules is the valuable part of this. The {{utils/modeshape-connector-benchmark}} had such an output, and it's very valuable. It has a single [HTML template|] with a {{$DIV$}} section that was [replaced with the chart data in the code|https://github.com/ModeShape/modeshape/blob/master/utils/modeshape-connector-benchmark/src/test/java/org/modeshape/test/benchmark/ConnectorBenchmarkTest.java#L164].

        Best of all, Google charts now has support for box charts, which is exactly what we want. The question really becomes: what's the best way to produce the charts? Since each test outputs the data (in the 'perf-test.txt' file), perhaps the aggregate HTML page can be generated by simply reading those files (whichever ones are there).
        The module-specific charts are good (though it'd be nice if the {{barchart.html}} page showed all of the charts there (with titles).

        But the real value is going to be a single page that has one graph for each test, where each graph contained the results from all of the tested modules. The comparison between tested modules is the valuable part of this. The {{utils/modeshape-connector-benchmark}} had such an output, and it's very valuable. It has a single [HTML template|https://github.com/ModeShape/modeshape/blob/master/utils/modeshape-connector-benchmark/src/test/resources/performance/benchmark.html] with a {{$DIV$}} section that was [replaced with the chart data in the code|https://github.com/ModeShape/modeshape/blob/master/utils/modeshape-connector-benchmark/src/test/java/org/modeshape/test/benchmark/ConnectorBenchmarkTest.java#L164].

        Best of all, Google charts now has support for [box charts|http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts], which is exactly what we want! The question really becomes: what's the best way to produce the charts within the performance test framework? Since each test outputs the data (in the 'perf-test.txt' file), perhaps the aggregate HTML page can be generated by simply reading those files (whichever ones are there).

        (BTW, I don't think we need to put the outliers on our charts. There's no easy way to do that with Google Charts, since you need a separate data series for each pair of outliers for each candlestick.)
        Hide
        Horia Chiorean
        added a comment - - edited

        So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable.

        Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure

        Show
        Horia Chiorean
        added a comment - - edited So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable. Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure
        Hide
        Horia Chiorean
        added a comment -

        I'm no chart wiz unfortunately, but from the example (http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ?

        Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1.

        Show
        Horia Chiorean
        added a comment - I'm no chart wiz unfortunately, but from the example ( http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts ), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ? Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1.
        Hide
        Randall Hauch
        added a comment -

        So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable.

        Yes, so there'd be a chart for the "OpenAnonymousSessionTestSuite" (maybe called "Open Anonymous Session") that has boxes for the results from ModeShape 2.7, ModeShape 3.0, Jackrabbit 2.3.2 (and any other versions/impls that are added). There'd also be additional charts for the other test suites. (IIRC the connector benchmark results places the charts in a table, with 2 or 3 columns and as many rows as necessary.)

        Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure

        I agree that this may require a dedicated reporting module that always runs last. One alternative, though, maybe be to try adding the chart generation logic (not sure what form that takes) to the end of the 'test' phase or in the assembly phase.

        Show
        Randall Hauch
        added a comment - So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable. Yes, so there'd be a chart for the "OpenAnonymousSessionTestSuite" (maybe called "Open Anonymous Session") that has boxes for the results from ModeShape 2.7, ModeShape 3.0, Jackrabbit 2.3.2 (and any other versions/impls that are added). There'd also be additional charts for the other test suites. (IIRC the connector benchmark results places the charts in a table, with 2 or 3 columns and as many rows as necessary.) Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure I agree that this may require a dedicated reporting module that always runs last. One alternative, though, maybe be to try adding the chart generation logic (not sure what form that takes) to the end of the 'test' phase or in the assembly phase.
        Hide
        Randall Hauch
        added a comment -

        I'm no chart wiz unfortunately, but from the example (http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ?

        Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1.

        I think that's controlled with the range: for example "?chxr=0,10,260" (for one chart axis).

        Here's an example: http://chart.apis.google.com/chart?cht=bvs&chs=500x400&chxr=0,1,15%7C1,0,50&chxt=y,x&chd=t0:-1,5,10,7,12,-1%7C-1,25,30,27,24,-1%7C-1,40,45,47,39,-1%7C-1,55,63,59,80,-1%7C-1,30,40,35,30,-1%7C-1,-1,5,70,90,-1%7C-1,-1,-1,80,5,-1&chm=F,FF9900,0,1:4,40%7CH,0CBF0B,0,1:4,1:20%7CH,000000,4,1:4,1:40%7CH,0000FF,3,1:4,1:20

        But I don't see how varying the range actually changes the data; seems to just change the axis values.

        Show
        Randall Hauch
        added a comment - I'm no chart wiz unfortunately, but from the example ( http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts ), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ? Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1. I think that's controlled with the range: for example " ?chxr=0,10,260 " (for one chart axis). Here's an example: http://chart.apis.google.com/chart?cht=bvs&chs=500x400&chxr=0,1,15%7C1,0,50&chxt=y,x&chd=t0:-1,5,10,7,12,-1%7C-1,25,30,27,24,-1%7C-1,40,45,47,39,-1%7C-1,55,63,59,80,-1%7C-1,30,40,35,30,-1%7C-1,-1,5,70,90,-1%7C-1,-1,-1,80,5,-1&chm=F,FF9900,0,1:4,40%7CH,0CBF0B,0,1:4,1:20%7CH,000000,4,1:4,1:40%7CH,0000FF,3,1:4,1:20 But I don't see how varying the range actually changes the data; seems to just change the axis values.
        Horia Chiorean
        07/Dec/11 7:06 AM
        View full commit
        MODE-1323 added Google box chart report for aggregate results -created a new module used for aggregating the results produced by the other modules -implemented a Google charts based report which displays all the repositories side-by-side
        Hide
        Horia Chiorean
        added a comment - - edited
        • added new Maven module - perf-tests-report which aggregates the data from the individual modules
        • added a new report which generates Google box chart links for each of the tests ran, comparing the performance of each repository. They will be located in perf-tests-report/target/test-classes/google
        Show
        Horia Chiorean
        added a comment - - edited added new Maven module - perf-tests-report which aggregates the data from the individual modules added a new report which generates Google box chart links for each of the tests ran, comparing the performance of each repository. They will be located in perf-tests-report/target/test-classes/google added a new box-plot report generator ( http://informationandvisualization.de/blog/box-plot ) which generates 1 report per each ran test, comparing each of the repositories. They will be located at perf-tests-report/target/test-classes/boxplot
        Horia Chiorean
        made changes -
        Status Open [ 1 ] Pull Request Sent [ 10011 ]
        Git Pull Request https://github.com/ModeShape/modeshape-performance/pull/5
        Hide
        Horia Chiorean
        added a comment - - edited

        Report generated by boxplot - ignore the html, you need the zip file

        Show
        Horia Chiorean
        added a comment - - edited Report generated by boxplot - ignore the html, you need the zip file
        Horia Chiorean
        made changes -
        Attachment SmallFileWriteTestSuite.html [ 12349685 ]
        Hide
        Horia Chiorean
        added a comment - - edited

        Correct boxplot report (needs js and css as well)

        As a comparison, the same view using Google charts (unfortunately scaling is poor in G Charts): http://tinyurl.com/7kazns6

        Show
        Horia Chiorean
        added a comment - - edited Correct boxplot report (needs js and css as well) As a comparison, the same view using Google charts (unfortunately scaling is poor in G Charts): http://tinyurl.com/7kazns6
        Horia Chiorean
        made changes -
        Attachment boxplot.zip [ 12349687 ]
        Randall Hauch
        made changes -
        Attachment SmallFileWriteTestSuite.html [ 12349685 ]
        Hide
        Horia Chiorean
        added a comment - - edited

        Attached sample report with D3.

        There is still a problem (and I'm tending to be believe that this problem cannot be solved with box charts): if the values from the different series are wide-enough apart, the "lower" series will not be displayed properly

        In the attached CreateChildNodesTestSuite.html, the series have the following values:

        • [146, 117, 127, 109, 111, 149, 176, ...] (rounded)
        • [20, 21, 22, 26, ...] (rounded)
        • [4.0, 4.4, 4.24, 4.28, ...]

        the last 2 series and especially the last one, will not be displayed clearly enough on a [0, 176] scale.

        Show
        Horia Chiorean
        added a comment - - edited Attached sample report with D3. There is still a problem (and I'm tending to be believe that this problem cannot be solved with box charts): if the values from the different series are wide-enough apart, the "lower" series will not be displayed properly In the attached CreateChildNodesTestSuite.html, the series have the following values: [146, 117, 127, 109, 111, 149, 176, ...] (rounded) [20, 21, 22, 26, ...] (rounded) [4.0, 4.4, 4.24, 4.28, ...] the last 2 series and especially the last one, will not be displayed clearly enough on a [0, 176] scale.
        Horia Chiorean
        made changes -
        Attachment d3_report.zip [ 12349739 ]
        Hide
        Horia Chiorean
        added a comment -

        Merged into master

        Show
        Horia Chiorean
        added a comment - Merged into master
        Horia Chiorean
        made changes -
        Status Pull Request Sent [ 10011 ] Resolved [ 5 ]
        Resolution Done [ 1 ]
        Randall Hauch
        made changes -
        Fix Version/s 3.0.0.Alpha1 [ 12318454 ]
        Fix Version/s 3.0.0.Alpha2 [ 12318455 ]
        Git Pull Request https://github.com/ModeShape/modeshape-performance/pull/5 https://github.com/ModeShape/modeshape-performance/pull/5
        Randall Hauch
        made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Horia Chiorean
            Reporter:
            Randall Hauch
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: