I want to make some performance measures (mainly runtime) for my Java code, a single-threaded, local, complex algorithm. (So I do not want a macro-benchmark to measure a JVM implementation.)
With the tool, I would like to
- see how my code scales for a parameter n (the search depth). (I already have a junit test parameterized in n.)
- do some trend analysis to get warned if some change to the code base makes the code slower.
For this, I would like to use a tool or framework that
- does the statistics, optimally computing the mean value, standard deviation and confidence intervals. This is very important.
- can be parameterized (see parameter n above). This is also very important.
- is able to produce a fancy plot would be nice, but is not required
- can be used in an automated (junit-)test to warn me if my program slows done, but this is also not required, just a plus.
So far, I have found the following solutions:
- jmeter, but I'm not sure if it can do all the statistics. I just have a single threaded, local algorithm, so wouldn't a more lightweight solution be better?
- perf4j, for which I have the same questions as for jmeter. Additionally, how strongly is it intertwined with log4j? I'm using java.util.logging.
- JAMon, but I'm not sure whether there is an add-on for the statistics. Again, is there a more lightweight solution?
- TPTP, but I've heard that this is a huge, complex plugin that slows eclipse down. Can it do the statistics?
- the benchmarking framework http://www.ellipticgroup.com/misc/projectLibrary.zip (see e.g. Create quick/reliable benchmark with java?). This framework does the statistics very well, but uses a lot of techniques that are only necessary for micro-benchmarking. If nobody convinces me otherwise, I'll probably use this.
What tool/framework do you think is suited best for my situation? Which tool/framework do you use for similar situations, and why?
Below is the list of all the mentioned tools. Relevant aspects are:
- easily parameterizable
- whether it is a Java library or at least easily integratable into your Java program
- whether it can handle JVM micro benchmarking, e.g. use a warmup phase
- whether it can plot the results visually
- whether it can store the measured values persistently
- whether it can do trend analysis to warn that a new commit caused a slow down
- whether it does some statistics (at least max, min, average and standard deviation)
the benchmarking framework http://www.ellipticgroup.com/misc/projectLibrary.zip
not parameterizable; Java library; JVM micro benchmarking; no plotting; no persistence; no trend analysis; statistics.
Does the statistics extremely well: besides average, max, min and standard deviation, it also computes the 95% confidence interval (via bootstrapping) and serial correlation (e.g. to warn about oscillating execution times, which can occur if your program behaves nondeterministically, e.g. because you use HashSets). It decides how often the program has to be iterated to get accurate measurements and interprets these for reporting and warnings (e.g. about outliers and serial correlation).
Also does the micro-benchmarking extremely well (see Create quick/reliable benchmark with java? for details).
Unfortunately, the framework comes in a util-package bundled together with a lot of other helper-classes. The benchmark classes depend on JSci (A science API for Java) and Mersenne Twister (http://www.cs.gmu.edu/~sean/research/). If the author, Brent Boyer, finds time, he will boil the library down and add a simpler grapher so that the user can visually inspect the measurements, e.g. for correlations and outliers.
parameterizable; Java library; JVM micro benchmarking; plotting; persistence; no trend analysis; statistics.
Relatively new project, tailored towards Android apps. Looks young but promising. Depends on Google Guava :(
parameterizable; Java library; JVM micro benchmarking; plotting; persistence (using CONSOLE, XML or database H2); graphical trend analysis; statistics (max, min, average, standard deviation; but not easily extensible for further statistics).
Simply add a junit-4-rule to your junit tests :)
Sources are included, but the project is not open source.
not parameterizable; Java library; JVM micro benchmarking; plotting; persistence; no trend analysis; no statistics.
Nice lightweight monitoring tool, no dependencies :) Does not offer sufficient statistics (no standard deviation), and extending the plugIn correspondingly looks quite difficult (Aggregators and Aggregates only have fixed getters for min, max and average).
parameterizable; Jython library; no JVM micro benchmarking; plotting; persistence; no trend analysis; no good statistics, but easily extensible.
Depends on Jython, HTTPClient, JEditSyntax, ApacheXMLBeans, PicoContainer.
parameterizable!?; java library; no JVM micro benchmarking!?; plotting; persistence; trend analysis!?; statistics!?.
Good monitoring library that is tailored towards load testing web applications.
not parameterizable; Java library; no JVM micro benchmarking; plotting; persistence via JMX; trend analysis via a log4j appender; statistics.
Builds upon a logging framework, can use AOP.
not parameterizable; Java library; no JVM micro benchmarking; plotting, persistence and trend analysis with additional tools (Jarep or JMX); statistics.
Good monitoring, intertwined with log4j, data can also be programmatically accessed or queried and your program can take actions on the results.
not parameterizable!?; Java library; no JVM micro benchmarking; plotting only with Jarep; persistence only with JMX; no trend analysis; no statistics!?.
Competitor of Jamon, supports a hierarchy of monitors.
parameterizable!?; Java library; no JVM micro benchmarking; plotting; persistence; trend analysis!?; statistics!?.
Tool is tailored towards monitoring in large clusters.
not parameterizable!?; Java library; no JVM micro benchmarking!?; plotting; persistence through a servlet; no trend analysis!?; no statistics!?.
Supports AOP instrumentation.
Very general concept: monitors observe predefined conditions and specify how to react when they are met.
parameterizable!?; Java tool platform; no JVM micro benchmarking!?; plotting; persistence; graphical trend analysis; no statistics!?
The Test & Performance Tools Platform is a huge generic and extensible tool platform (based on Eclipse and four EMF models). Hence it is powerful but quite complex, can slow Eclipse down, and extending it for your own needs (e.g. with statistics so that they influence the number of iterations) seems to be very difficult.
parameterizable; Perl library; no JVM micro benchmarking; plotting; persistence; trend analysis!?; good statistics (run a given test until results stabilize; highlight outliers).
Try using http://labs.carrotsearch.com/junit-benchmarks.html. This is an extention to JUni4, features:
Records execution time average and standard deviation.
Garbage collector activity recording.
Per-benchmark JVM warm-up phase.
Per-run and historical chart generation.
Optional results persistence in the H2 SQL database (advanced querying, historical analysis).