Almost five years after I initially released criterion, I'm delighted to announce a major release with a large number of appealing new features.
As always, you can install the latest goodness using
cabal install criterion, or fetch the source from github.
Please let me know if you find criterion useful!
All of the documentation lives in the github repo, so if you'd like to see something improved, please send a bug report or pull request.
New execution engine
Criterion's model of execution has evolved, becoming vastly more reliable and accurate. It can now measure events that take just a few hundred picoseconds.
benchmarking return () time 512.9 ps (512.8 ps .. 513.1 ps)
While almost all of the core types have changed, criterion should remain API-compatible with the vast majority of your benchmarking code.
In addition to wall-clock time, criterion can now measure and regress on the following metrics:
- CPU time
- CPU cycles
- bytes allocated
- number of garbage collections
- number of bytes copied during GC
- wall-clock time spent in mutator threads
- CPU time spent running mutator threads
- wall-clock time spent doing GC
- CPU time spent doing GC
Criterion now supports linear regression of a number of metrics.
Here's a regression conducted using
cycles: 1.000 R² (1.000 R² .. 1.000 R²) iters 47.718 (47.657 .. 47.805)
The first line of the output is the R² goodness-of-fit measure for this regression, and the second is the number of CPU cycles (measured using the
rdtsc instruction) to execute the operation in question (integer division).
This next regression uses
--regress allocated:iters to measure the number of bytes allocated while constructing an
IntMap of 40,000 values.
allocated: 1.000 R² (1.000 R² .. 1.000 R²) iters 4.382e7 (4.379e7 .. 4.384e7)
(That's a little under 42 megabytes.)
While its support for active HTML has improved, criterion can also now output JSON and JUnit XML files.
Criterion has received its first spring cleaning, and is much easier to understand as a result.
I was inspired into some of this work by the efforts of the authors of the OCaml Core_bench package.