criterion performance measurements

overview

want to understand this report?

Int/IntMap/sorted

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 8.819502838635092e-3 8.937630044753272e-3 9.11687814107912e-3
Standard deviation 2.961206051453886e-4 3.8811768961642034e-4 5.209891461490966e-4

Outlying measurements have moderate (0.1783968758696835%) effect on estimated standard deviation.

Int/IntMap/random

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.8542157501053912e-2 2.900743929111707e-2 2.958125243996035e-2
Standard deviation 8.657166917576245e-4 1.0721292520478878e-3 1.363882252593622e-3

Outlying measurements have moderate (0.10833474193374858%) effect on estimated standard deviation.

Int/IntMap/revsorted

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 9.112766790888372e-3 9.362502704743527e-3 9.835610852580709e-3
Standard deviation 5.352201403618318e-4 8.624425557441545e-4 1.2370142915980008e-3

Outlying measurements have severe (0.5003056810937044%) effect on estimated standard deviation.

Int/Map/sorted

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.074059258295756e-2 3.211011289529789e-2 3.527622608186352e-2
Standard deviation 1.864156077219858e-3 4.2796965648887655e-3 7.382183902703018e-3

Outlying measurements have severe (0.5703320814503217%) effect on estimated standard deviation.

Int/Map/random

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.256863169781285e-2 5.423701345317711e-2 5.75263665915807e-2
Standard deviation 1.8884040663696014e-3 4.018496861800647e-3 6.914612452206933e-3

Outlying measurements have moderate (0.2191092792162958%) effect on estimated standard deviation.

Int/Map/revsorted

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.893885738311336e-2 2.9700954738630478e-2 3.0814787788813195e-2
Standard deviation 1.2367503170643201e-3 1.9677833953441904e-3 3.1655852527274785e-3

Outlying measurements have moderate (0.2237015542046663%) effect on estimated standard deviation.

Int/HashMap/sorted

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.4908814983483784e-2 3.609431558970996e-2 3.836960118282639e-2
Standard deviation 1.8750127434045843e-3 3.3717688192625266e-3 5.852156692756491e-3

Outlying measurements have moderate (0.3592629030261099%) effect on estimated standard deviation.

Int/HashMap/random

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.484697359895852e-2 3.704957245731288e-2 4.314680605467057e-2
Standard deviation 2.109599986174092e-3 6.49252372752462e-3 1.0967241520954466e-2

Outlying measurements have severe (0.6515906864355713%) effect on estimated standard deviation.

Int/HashMap/revsorted

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.438757486779517e-2 3.6006875840224564e-2 3.8605071212944604e-2
Standard deviation 2.3146055359431917e-3 3.86867675559429e-3 5.443213496982174e-3

Outlying measurements have moderate (0.42130793821828655%) effect on estimated standard deviation.

ByteString/Map/sorted

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 8.07748582599183e-3 8.32904866130215e-3 8.806077418534286e-3
Standard deviation 5.211793934928915e-4 9.312595677147134e-4 1.4390411593630463e-3

Outlying measurements have severe (0.6210257293950999%) effect on estimated standard deviation.

ByteString/Map/random

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.0110133435932062e-2 1.0387725441922272e-2 1.0842328111211384e-2
Standard deviation 5.911127933420502e-4 9.711546553142731e-4 1.592031821668909e-3

Outlying measurements have severe (0.5163836979954514%) effect on estimated standard deviation.

ByteString/Map/revsorted

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 8.028422157830457e-3 8.286984315863834e-3 8.759854503977735e-3
Standard deviation 5.086836504735383e-4 9.26637169575568e-4 1.7093393975807325e-3

Outlying measurements have severe (0.6210284027587997%) effect on estimated standard deviation.

ByteString/HashMap/sorted

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.59492287832012e-3 5.715304733736597e-3 5.854873905308174e-3
Standard deviation 2.897995995948311e-4 3.6805494396342846e-4 5.351977853987202e-4

Outlying measurements have moderate (0.3703243985993333%) effect on estimated standard deviation.

ByteString/HashMap/random

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.719008056297191e-3 6.011378247359856e-3 6.4127589615828925e-3
Standard deviation 6.413363439360069e-4 9.996367294972932e-4 1.4127482949420139e-3

Outlying measurements have severe (0.8007711032151971%) effect on estimated standard deviation.

ByteString/HashMap/revsorted

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.624547446996474e-3 5.687942787530952e-3 5.770744183837958e-3
Standard deviation 1.6570136451117353e-4 2.341919953104207e-4 3.860014545750548e-4

Outlying measurements have moderate (0.19518394390669644%) effect on estimated standard deviation.

ByteString/CritBit/sorted

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.230125738946211e-3 4.303448916215421e-3 4.397927873894102e-3
Standard deviation 2.0581719155502942e-4 2.643838981947279e-4 3.476473011416311e-4

Outlying measurements have moderate (0.3933449069031089%) effect on estimated standard deviation.

ByteString/CritBit/random

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 9.041657318955057e-3 9.351051711024927e-3 9.996201118714297e-3
Standard deviation 6.022697620875256e-4 1.1937269103816997e-3 2.142053000129604e-3

Outlying measurements have severe (0.6632260188850402%) effect on estimated standard deviation.

ByteString/CritBit/revsorted

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.291530142015521e-3 4.520433177048569e-3 4.95343113631778e-3
Standard deviation 6.692430182547295e-4 1.0286480378572111e-3 1.695181647792266e-3

Outlying measurements have severe (0.9034231878411371%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.