ModeShape
  1. ModeShape
  2. MODE-1323

Performance benchmark should graph results from all modules

    Details

    • Type: Enhancement Enhancement
    • Status: Closed Closed (View Workflow)
    • Priority: Critical Critical
    • Resolution: Done
    • Affects Version/s: 3.0.0.Alpha1
    • Fix Version/s: 3.0.0.Alpha1
    • Component/s: Performance
    • Labels:
      None
    • Similar Issues:
      Show 10 results 

      Description

      The module-specific charts are good (though it'd be nice if the barchart.html page showed all of the charts there (with titles).

      But the real value is going to be a single page that has one graph for each test, where each graph contained the results from all of the tested modules. The comparison between tested modules is the valuable part of this. The utils/modeshape-connector-benchmark had such an output, and it's very valuable. It has a single HTML template with a $DIV$ section that was replaced with the chart data in the code.

      Best of all, Google charts now has support for box charts, which is exactly what we want! The question really becomes: what's the best way to produce the charts within the performance test framework? Since each test outputs the data (in the 'perf-test.txt' file), perhaps the aggregate HTML page can be generated by simply reading those files (whichever ones are there).

      (BTW, I don't think we need to put the outliers on our charts. There's no easy way to do that with Google Charts, since you need a separate data series for each pair of outliers for each candlestick.)

        Activity

        Hide
        Horia Chiorean
        added a comment - - edited

        So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable.

        Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure

        Show
        Horia Chiorean
        added a comment - - edited So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable. Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure
        Hide
        Horia Chiorean
        added a comment -

        I'm no chart wiz unfortunately, but from the example (http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ?

        Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1.

        Show
        Horia Chiorean
        added a comment - I'm no chart wiz unfortunately, but from the example ( http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts ), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ? Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1.
        Hide
        Randall Hauch
        added a comment -

        So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable.

        Yes, so there'd be a chart for the "OpenAnonymousSessionTestSuite" (maybe called "Open Anonymous Session") that has boxes for the results from ModeShape 2.7, ModeShape 3.0, Jackrabbit 2.3.2 (and any other versions/impls that are added). There'd also be additional charts for the other test suites. (IIRC the connector benchmark results places the charts in a table, with 2 or 3 columns and as many rows as necessary.)

        Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure

        I agree that this may require a dedicated reporting module that always runs last. One alternative, though, maybe be to try adding the chart generation logic (not sure what form that takes) to the end of the 'test' phase or in the assembly phase.

        Show
        Randall Hauch
        added a comment - So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable. Yes, so there'd be a chart for the "OpenAnonymousSessionTestSuite" (maybe called "Open Anonymous Session") that has boxes for the results from ModeShape 2.7, ModeShape 3.0, Jackrabbit 2.3.2 (and any other versions/impls that are added). There'd also be additional charts for the other test suites. (IIRC the connector benchmark results places the charts in a table, with 2 or 3 columns and as many rows as necessary.) Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure I agree that this may require a dedicated reporting module that always runs last. One alternative, though, maybe be to try adding the chart generation logic (not sure what form that takes) to the end of the 'test' phase or in the assembly phase.
        Hide
        Randall Hauch
        added a comment -

        I'm no chart wiz unfortunately, but from the example (http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ?

        Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1.

        I think that's controlled with the range: for example "?chxr=0,10,260" (for one chart axis).

        Here's an example: http://chart.apis.google.com/chart?cht=bvs&chs=500x400&chxr=0,1,15%7C1,0,50&chxt=y,x&chd=t0:-1,5,10,7,12,-1%7C-1,25,30,27,24,-1%7C-1,40,45,47,39,-1%7C-1,55,63,59,80,-1%7C-1,30,40,35,30,-1%7C-1,-1,5,70,90,-1%7C-1,-1,-1,80,5,-1&chm=F,FF9900,0,1:4,40%7CH,0CBF0B,0,1:4,1:20%7CH,000000,4,1:4,1:40%7CH,0000FF,3,1:4,1:20

        But I don't see how varying the range actually changes the data; seems to just change the axis values.

        Show
        Randall Hauch
        added a comment - I'm no chart wiz unfortunately, but from the example ( http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts ), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ? Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1. I think that's controlled with the range: for example " ?chxr=0,10,260 " (for one chart axis). Here's an example: http://chart.apis.google.com/chart?cht=bvs&chs=500x400&chxr=0,1,15%7C1,0,50&chxt=y,x&chd=t0:-1,5,10,7,12,-1%7C-1,25,30,27,24,-1%7C-1,40,45,47,39,-1%7C-1,55,63,59,80,-1%7C-1,30,40,35,30,-1%7C-1,-1,5,70,90,-1%7C-1,-1,-1,80,5,-1&chm=F,FF9900,0,1:4,40%7CH,0CBF0B,0,1:4,1:20%7CH,000000,4,1:4,1:40%7CH,0000FF,3,1:4,1:20 But I don't see how varying the range actually changes the data; seems to just change the axis values.
        Hide
        Horia Chiorean
        added a comment - - edited
        • added new Maven module - perf-tests-report which aggregates the data from the individual modules
        • added a new report which generates Google box chart links for each of the tests ran, comparing the performance of each repository. They will be located in perf-tests-report/target/test-classes/google
        Show
        Horia Chiorean
        added a comment - - edited added new Maven module - perf-tests-report which aggregates the data from the individual modules added a new report which generates Google box chart links for each of the tests ran, comparing the performance of each repository. They will be located in perf-tests-report/target/test-classes/google added a new box-plot report generator ( http://informationandvisualization.de/blog/box-plot ) which generates 1 report per each ran test, comparing each of the repositories. They will be located at perf-tests-report/target/test-classes/boxplot
        Hide
        Horia Chiorean
        added a comment - - edited

        Report generated by boxplot - ignore the html, you need the zip file

        Show
        Horia Chiorean
        added a comment - - edited Report generated by boxplot - ignore the html, you need the zip file
        Hide
        Horia Chiorean
        added a comment - - edited

        Correct boxplot report (needs js and css as well)

        As a comparison, the same view using Google charts (unfortunately scaling is poor in G Charts): http://tinyurl.com/7kazns6

        Show
        Horia Chiorean
        added a comment - - edited Correct boxplot report (needs js and css as well) As a comparison, the same view using Google charts (unfortunately scaling is poor in G Charts): http://tinyurl.com/7kazns6
        Hide
        Horia Chiorean
        added a comment - - edited

        Attached sample report with D3.

        There is still a problem (and I'm tending to be believe that this problem cannot be solved with box charts): if the values from the different series are wide-enough apart, the "lower" series will not be displayed properly

        In the attached CreateChildNodesTestSuite.html, the series have the following values:

        • [146, 117, 127, 109, 111, 149, 176, ...] (rounded)
        • [20, 21, 22, 26, ...] (rounded)
        • [4.0, 4.4, 4.24, 4.28, ...]

        the last 2 series and especially the last one, will not be displayed clearly enough on a [0, 176] scale.

        Show
        Horia Chiorean
        added a comment - - edited Attached sample report with D3. There is still a problem (and I'm tending to be believe that this problem cannot be solved with box charts): if the values from the different series are wide-enough apart, the "lower" series will not be displayed properly In the attached CreateChildNodesTestSuite.html, the series have the following values: [146, 117, 127, 109, 111, 149, 176, ...] (rounded) [20, 21, 22, 26, ...] (rounded) [4.0, 4.4, 4.24, 4.28, ...] the last 2 series and especially the last one, will not be displayed clearly enough on a [0, 176] scale.
        Hide
        Horia Chiorean
        added a comment -

        Merged into master

        Show
        Horia Chiorean
        added a comment - Merged into master

          People

          • Assignee:
            Horia Chiorean
            Reporter:
            Randall Hauch
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: