Uploaded image for project: 'ModeShape'
  1. ModeShape
  2. MODE-1323

Performance benchmark should graph results from all modules

    Details

    • Type: Enhancement
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Done
    • Affects Version/s: 3.0.0.Alpha1
    • Fix Version/s: 3.0.0.Alpha1
    • Component/s: Performance
    • Labels:
      None

      Description

      The module-specific charts are good (though it'd be nice if the barchart.html page showed all of the charts there (with titles).

      But the real value is going to be a single page that has one graph for each test, where each graph contained the results from all of the tested modules. The comparison between tested modules is the valuable part of this. The utils/modeshape-connector-benchmark had such an output, and it's very valuable. It has a single HTML template with a $DIV$ section that was replaced with the chart data in the code.

      Best of all, Google charts now has support for box charts, which is exactly what we want! The question really becomes: what's the best way to produce the charts within the performance test framework? Since each test outputs the data (in the 'perf-test.txt' file), perhaps the aggregate HTML page can be generated by simply reading those files (whichever ones are there).

      (BTW, I don't think we need to put the outliers on our charts. There's no easy way to do that with Google Charts, since you need a separate data series for each pair of outliers for each candlestick.)

        Gliffy Diagrams

          Activity

          Hide
          hchiorean Horia Chiorean added a comment - - edited

          So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable.

          Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure

          Show
          hchiorean Horia Chiorean added a comment - - edited So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable. Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure
          Hide
          hchiorean Horia Chiorean added a comment -

          I'm no chart wiz unfortunately, but from the example (http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ?

          Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1.

          Show
          hchiorean Horia Chiorean added a comment - I'm no chart wiz unfortunately, but from the example ( http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts ), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ? Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1.
          Hide
          rhauch Randall Hauch added a comment -

          So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable.

          Yes, so there'd be a chart for the "OpenAnonymousSessionTestSuite" (maybe called "Open Anonymous Session") that has boxes for the results from ModeShape 2.7, ModeShape 3.0, Jackrabbit 2.3.2 (and any other versions/impls that are added). There'd also be additional charts for the other test suites. (IIRC the connector benchmark results places the charts in a table, with 2 or 3 columns and as many rows as necessary.)

          Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure

          I agree that this may require a dedicated reporting module that always runs last. One alternative, though, maybe be to try adding the chart generation logic (not sure what form that takes) to the end of the 'test' phase or in the assembly phase.

          Show
          rhauch Randall Hauch added a comment - So basically for each kind of test, we would produce 1 box chart, with all the values from all the modules against which the test was run ? I'm not sure whether an overlayed box chart would be readable. Yes, so there'd be a chart for the "OpenAnonymousSessionTestSuite" (maybe called "Open Anonymous Session") that has boxes for the results from ModeShape 2.7, ModeShape 3.0, Jackrabbit 2.3.2 (and any other versions/impls that are added). There'd also be additional charts for the other test suites. (IIRC the connector benchmark results places the charts in a table, with 2 or 3 columns and as many rows as necessary.) Anyway, the main challenge is how to aggregate test results from each module (as they are atm, they are all independent and each runs with it's own runner). Maybe having a dedicated reporting module...not sure I agree that this may require a dedicated reporting module that always runs last. One alternative, though, maybe be to try adding the chart generation logic (not sure what form that takes) to the end of the 'test' phase or in the assembly phase.
          Hide
          rhauch Randall Hauch added a comment -

          I'm no chart wiz unfortunately, but from the example (http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ?

          Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1.

          I think that's controlled with the range: for example "?chxr=0,10,260" (for one chart axis).

          Here's an example: http://chart.apis.google.com/chart?cht=bvs&chs=500x400&chxr=0,1,15%7C1,0,50&chxt=y,x&chd=t0:-1,5,10,7,12,-1%7C-1,25,30,27,24,-1%7C-1,40,45,47,39,-1%7C-1,55,63,59,80,-1%7C-1,30,40,35,30,-1%7C-1,-1,5,70,90,-1%7C-1,-1,-1,80,5,-1&chm=F,FF9900,0,1:4,40%7CH,0CBF0B,0,1:4,1:20%7CH,000000,4,1:4,1:40%7CH,0000FF,3,1:4,1:20

          But I don't see how varying the range actually changes the data; seems to just change the axis values.

          Show
          rhauch Randall Hauch added a comment - I'm no chart wiz unfortunately, but from the example ( http://code.google.com/apis/chart/image/docs/gallery/compound_charts.html#box_charts ), if I change any of values in the max series to be either above 100 or below 1, nothing gets displyed - any idea why ? Since the test data is recorded in ns, it can be converted to any unit we want, which is going to yield values either greater than 100 or less than 1. I think that's controlled with the range: for example " ?chxr=0,10,260 " (for one chart axis). Here's an example: http://chart.apis.google.com/chart?cht=bvs&chs=500x400&chxr=0,1,15%7C1,0,50&chxt=y,x&chd=t0:-1,5,10,7,12,-1%7C-1,25,30,27,24,-1%7C-1,40,45,47,39,-1%7C-1,55,63,59,80,-1%7C-1,30,40,35,30,-1%7C-1,-1,5,70,90,-1%7C-1,-1,-1,80,5,-1&chm=F,FF9900,0,1:4,40%7CH,0CBF0B,0,1:4,1:20%7CH,000000,4,1:4,1:40%7CH,0000FF,3,1:4,1:20 But I don't see how varying the range actually changes the data; seems to just change the axis values.
          Hide
          hchiorean Horia Chiorean added a comment - - edited
          • added new Maven module - perf-tests-report which aggregates the data from the individual modules
          • added a new report which generates Google box chart links for each of the tests ran, comparing the performance of each repository. They will be located in perf-tests-report/target/test-classes/google
          Show
          hchiorean Horia Chiorean added a comment - - edited added new Maven module - perf-tests-report which aggregates the data from the individual modules added a new report which generates Google box chart links for each of the tests ran, comparing the performance of each repository. They will be located in perf-tests-report/target/test-classes/google added a new box-plot report generator ( http://informationandvisualization.de/blog/box-plot ) which generates 1 report per each ran test, comparing each of the repositories. They will be located at perf-tests-report/target/test-classes/boxplot
          Hide
          hchiorean Horia Chiorean added a comment - - edited

          Report generated by boxplot - ignore the html, you need the zip file

          Show
          hchiorean Horia Chiorean added a comment - - edited Report generated by boxplot - ignore the html, you need the zip file
          Hide
          hchiorean Horia Chiorean added a comment - - edited

          Correct boxplot report (needs js and css as well)

          As a comparison, the same view using Google charts (unfortunately scaling is poor in G Charts): http://tinyurl.com/7kazns6

          Show
          hchiorean Horia Chiorean added a comment - - edited Correct boxplot report (needs js and css as well) As a comparison, the same view using Google charts (unfortunately scaling is poor in G Charts): http://tinyurl.com/7kazns6
          Hide
          hchiorean Horia Chiorean added a comment - - edited

          Attached sample report with D3.

          There is still a problem (and I'm tending to be believe that this problem cannot be solved with box charts): if the values from the different series are wide-enough apart, the "lower" series will not be displayed properly

          In the attached CreateChildNodesTestSuite.html, the series have the following values:

          • [146, 117, 127, 109, 111, 149, 176, ...] (rounded)
          • [20, 21, 22, 26, ...] (rounded)
          • [4.0, 4.4, 4.24, 4.28, ...]

          the last 2 series and especially the last one, will not be displayed clearly enough on a [0, 176] scale.

          Show
          hchiorean Horia Chiorean added a comment - - edited Attached sample report with D3. There is still a problem (and I'm tending to be believe that this problem cannot be solved with box charts): if the values from the different series are wide-enough apart, the "lower" series will not be displayed properly In the attached CreateChildNodesTestSuite.html, the series have the following values: [146, 117, 127, 109, 111, 149, 176, ...] (rounded) [20, 21, 22, 26, ...] (rounded) [4.0, 4.4, 4.24, 4.28, ...] the last 2 series and especially the last one, will not be displayed clearly enough on a [0, 176] scale.
          Hide
          hchiorean Horia Chiorean added a comment -

          Merged into master

          Show
          hchiorean Horia Chiorean added a comment - Merged into master

            People

            • Assignee:
              hchiorean Horia Chiorean
              Reporter:
              rhauch Randall Hauch
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development