A Python benchmarking framework and command-line tool: bench run
for ad-hoc shell commands, and Python API for repeatable suites.
$ bench run --runs 5 'sleep 0.01' 'sleep 0.02'
run (elapsed)
matrix mean ± σ min … max
sleep 0.01 11.03 ± 0.36 ms (10.66 … 11.49) (5 runs)
sleep 0.02 20.76 ± 0.16 ms (20.58 … 20.99) (5 runs)
Comparison - run (elapsed)
sleep 0.01 was
1.91 ± 0.07× better than sleep 0.02 (5 runs)# compare.py
from bench import Time, bench, run, suite
s = suite(
"sleep",
bench("sleep")
.with_matrix(seconds=["0.01", "0.02"])
.with_command(lambda ctx: ["sleep", ctx.data.seconds])
.with_metric(Time())
.with_runs(5),
)
if __name__ == "__main__":
run(s)$ python compare.py --json out.json
$ python compare.py show out.jsonSee examples/ for one runnable script per capability.
A suite holds benchmarks or other sub-suites.
Settings set on an app, suite or benchmark are inherited downwards.
Each benchmark expands into one variant per point of
its matrix (.with_matrix(**dims)), and each variant is run until its stopping
policy is satisfied (typically fixed number of runs).
Every run is an Execution in a Report.
Benchmarks run with an empty environment unless .with_env(...) or
.with_inherit_env() is used.
Variants can be dropped with .add_matrix_skip(...) or .with_filter(...),
runs spaced with .with_cooldown(s), and a suite's order randomized with
.with_shuffle(seed).
| Component | Setter | |
|---|---|---|
| Metric | .with_metric(...) |
Turns a finished run into samples |
| Stopping policy | .with_runs(...), |
Decides when a variant has run enough times |
.with_warmup(...) |
Same as runs, but marks runs as warmups | |
| Runner | .with_runner(...) |
Schedules variants: sequential, parallel or dry |
| Reporter | .with_reporter(...) |
Shows progress reports to setup output (JSON, CSV, ...) |
| Summary | .with_summary(...) |
Renders statistics after the run |
| Hooks | .with_hook(...) |
Runs setup/teardown code around each execution |
| Outlier detection | .with_outlier_detection(...) |
Flags outlying samples |
Outputs: live progress, the summary, and optionally --json, --csv and
--dir. bench show report.json re-renders a saved report.
bench compare a.json b.json ... compares saved reports, the first being the baseline.
Noise: bench doctor prints the machine fingerprint and flags noise
sources, exiting non-zero on a high-severity one. On Linux as root, bench denoise minimize|restore|status quiets those knobs and reverts them.
bench run --check-environment / --denoise do the same around a run;
scripts use bench_app(probe=SystemProbe(), denoise=True).
The linter, typechecker, formatter and tests all run uppon running
make checkTo format the project use
make formatTo fix auto-fixable Ruff errors use
make fix-lintMuch of the bench has been inspired by these great tools: