Point it at any simulation that prints numbers. One command finds the runs where it misbehaves, shrinks each to the smallest case, and tells you what the failure depends on. No adapter to write, no key, no model needed for the scan. With a model (Muse Spark), an agent explains a failure and the engine accepts the explanation only if every step and value it cites matches the recorded run. Built for the Meta Global AI Developer Hackathon (Indie Developer track, "Push the Frontier") in C++.
Demo video (2.5 minutes): https://youtu.be/rHbK1fGoPsQ
cmake -S . -B build && cmake --build build -j
./build/rc_scan --cmd "./your_simulator --gain {gain} --count {count} --mode {mode} --steps {steps}" --param gain:0.1:8:log --param count:1:30:int --choice mode:fast,safe --out scan_out
Point rc_scan at any command-line program that prints a table (a header of signal names, one numeric row per step; no adapter to write), or at a program that speaks the JSON-lines protocol (docs/PROTOCOL.md). It learns what normal looks like from the program's own runs, hunts ordinary and boundary scenarios, shrinks each finding, says what the finding depends on, and writes report.md and a plot page report.html (docs/SCAN.md; how to read a report: docs/SCAN_GUIDE.md).
- Real, reproducible findings on code we did not write (standalone repros, none reported upstream yet, either may already be known): a hang in REBOUND's default IAS15 integrator on a near-symmetric three-body start (
docs/EXTERNAL_ENGINE.md), and a hang in Chipmunk2D's stock Release build, which adds-ffast-math(docs/SCAN.md). - Would it notice a bug if there were one? Measured on Box2D 2.4.1, an engine we did not write, with changes planted mechanically by a tool (not by me or a model): the scan noticed 16 of 29 changed versions; on a fresh set of 40, 18 of 33, and 25 of 33 after adding mirror and shift symmetry checks (those need two small hooks in the adapter, so the adapter-free command mode gets the 18 of 33 rate; the count was 24 of 32 until the slowest scan finished and was added). It crashes and blow-ups reliably and misses subtle changes that leave the simulation looking plausible (
docs/SCAN.md). - No bug claimed in Box2D v3, Box2D 2.4.1, SUNDIALS CVODE, ReactPhysics3D, Bullet, Open Dynamics Engine, GSL or LAMMPS (documented limits and a few odd behaviors, each written up; many false alarms to sort by hand). The generic scanner missed the REBOUND hang with random starts, and re-found it (18 of 600 scenarios) once the scenario family contained a symmetric start, a family I added knowing the hang, so that is re-discovery, not discovery (
docs/SCAN.md). - The oracle gap: across two model-planted blind sets (39 and 118 variants, locked by hash before any check ran) the hand-written rules fire on only 41% of the variants that change behavior; a reference written by a model from the description alone recovers 58-69% of the rest, with 0 false alarms (
docs/BLIND.md,docs/REFERENCE_ORACLE.md). - Model-driven hunting beats random on planted bugs: triage localized 53/88 causes versus 37/88 for a no-model localizer (p=0.02); on five hard seeded bugs the boundary-enabled agent found 45/100 versus 17/300 random runs (without the dominant bug, 25/80 versus 17/240), while a model-free boundary sampler alone found 62/200.
- Real-engine truth checks: against a precise truth, a REBOUND TRACE fix is closer in 37 of 47 differing scenarios; two Box2D restitution commits move in opposite directions (
docs/REAL_PROJECTS.md).
Limits, plainly: every planted bug was written by me or by a model of the same family; one model family; small samples; no person has planted bugs or used the tool yet.
Full methods, caveats, negative findings, and generated tables follow below. scripts/gen_results.sh writes docs/RESULTS.md.
model (Muse Spark, tool calls, vision) deterministic, no model
┌───────────────────────────────────────────┐ ┌──────────────────────────────────────┐
│ hypothesis ─► run_scenario ─► inspect ─► │──►│ simulator ─► declared rules checked │
│ shrink ─► counterfactual ─► submit_report │◄──│ at every step ─► violation or not │
└───────────────────────────────────────────┘ └──────────────────────────────────────┘
report accepted only if every cited step/value matches the recorded run,
and (v2) a one-change counterfactual that removes the failure really passes
All numbers are generated by scripts from files in this repository (scripts/gen_results.sh writes docs/RESULTS.md).
Seeded bugs are planted by me; see the caveats before reading anything as "the model finds bugs".
| Question | Answer | Where |
|---|---|---|
| Given a failure, does the model find the cause better than a program? | Yes, and it replicated. On 83 failures found by random testing the agent localized the cause in 52 (63%) against 35 (42%) for a no-model localizer (paired sign test p = 0.006). A pre-registered replication on 88 new failures gave 53 (60%) against 37 (42%), p = 0.02, and the agent also beats a stronger no-model baseline (p = 0.009). 166 of 171 reports passed the engine's checks. It points the same way on two older models (p = 0.03, 0.047) and on the open-weight Muse Spark Open (44 of 83, p = 0.18, not significant alone). Same 22 bugs (planted by me); new failing scenarios. | docs/TRIAGE.md |
| Does the model find failures better than random testing? | Not with the first agent, yes with the boundary-case agent (v5), on the seeded bugs where random is weakest. Pre-registered, 20 fresh seeds per bug: agent v4 found the bug in 5 of 100 runs against 17 of 300 for random (sign test p = 0.875, no gain). After adding a boundary-value option to the sampler (docs/HYBRID.md), agent v5 found it in 45 of 100 runs (b02 7/20, b06 12/20, b09 6/20, b12 20/20, c02 0/20) against 17 of 300 for random (pooled one-sided Fisher p = 3e-18; 25/80 against 17/240 without b12, p = 3e-7), each with a validated report (44 of 45). Limits: the five bugs are the ones I seeded and picked as hardest for random testing, and they tend to sit at parameter boundaries; c02 is still not found; on the 160 sampled mutants there is no headroom to win. The boundary sampler alone, with no model, gets 62 of 200 on the same bugs. The open-weight Muse Spark Open model gives the same pattern (38 of 100, p = 7e-14 against random; 18/80 against 17/240 without b12). |
docs/HYBRID.md, docs/RESULTS.md |
| Does a smarter search (margin-guided hill climbing) beat random? | No. Tested with no model at equal step budget; it found each bug no more often than random in all 15 rows. So guided search was not built into the agent. Boundary-value sampling does (model-free, pre-declared, 40 fresh seeds per bug on the five bugs where random testing is weakest): random plus extreme-value scenarios found the bug in 62 of 200 runs against 17 of 200 for plain random (one-sided Fisher p = 8e-9). Honest breakdown: b12 0/40 -> 39/40 and b06 5 -> 15 drive it, b09 got worse (11 -> 5), b02 tied, c02 0 -> 2; without b12 it is 23 vs 17 of 160 (p = 0.20, not significant), and these seeded bugs trigger at parameter boundaries, so a boundary sampler is the favorable test. | docs/GUIDED_SEARCH.md, docs/HYBRID.md |
| Can a model write the rules from a description? | Partly. Model-written invariants hit 9 of 22 seeded bugs against 19 of 22 for hand-written rules; they find conservation laws but write loose bounds. Invariants mined from the clean simulator with no model hit 10 of 22; both together 11 of 22. A second attempt with duration forms hit 0 of 3 on traffic. | docs/SPEC_RULES.md |
| Is the v2 agent (checked counterfactual, frames in the loop) better than v1? | Not clearly. 20/30 against 19/30 valid and attributable, overlapping intervals; cause localization is better with frames (20/30 against 15/30). | docs/RESULTS.md |
| What about bugs I did not plant? | Mutation benchmark, 160 mechanically sampled bugs: random testing finds 50 of the 53 rule-catchable mutants in every step-budget window, so search is not the bottleneck; 69 of 122 behavior-changing mutants violate no declared rule, so the oracle is. Hand-written rules catch 43% of the changed mutants, model-written invariants 30%, metamorphic relations 20%; all three together 48%. Two model-planted blind sets (39 and 116 variants, different planters, locked by hash first, docs/BLIND.md) agree: hand-written rules fire on 41% and 42% of the variants that change behavior. |
docs/MUTATION.md |
| Does it work on code I did not write? | Yes, as a harness. Unmodified Box2D 3.x and REBOUND 5.2.2 run through the same rules and a 14-scenario literature corpus (all pass). That run also surfaced a reproducible hang in REBOUND's default IAS15 mode on the figure-eight initial condition (not reported upstream; not claimed as a confirmed bug). Real historical fixes were also tried (REBOUND TRACE, Box2D tunneling and restitution commits, 8 REBOUND releases): the declared rules cannot tell a fixed build from a buggy one, differential testing only says that something changed, and two of five fixes are never reached by my scenarios. The exception is a reference oracle: against a high-precision integrator, the REBOUND TRACE fix is closer to truth in 37 of 47 differing scenarios (10 the other way, p = 1e-4, pre-registered confirmation on 600 new scenarios), for one fix and one scenario family, with the reference supplied by me. The same holds for Box2D's restitution commits against an analytic ball-drop truth: the August commit moved away from the truth in 932 of 1000 scenarios, the September one moved back toward it in 969 (declared first; the truth is a simplified bounce I wrote). | docs/EXTERNAL_ENGINE.md, docs/REAL_PROJECTS.md |
| Can a model repair a simulator once a failure is found, and is "the rules pass" enough to accept the repair? | It restores most of them, and passing the rules is not enough. 96 of 105 repaired mutants reproduced the original bit for bit (four domains, one and two simultaneous changes, with and without a region hint; model-written code is compiled in a sandbox). The other 9 passed every declared rule and invariant yet were wrong, 8 of them on traffic, where the declared rules are weakest. | docs/REPAIR.md |
| Can rules be mined from the simulator itself, with no model and no hand-writing? | Not for the bugs that matter. The miner has 0 false positives on 4000 clean scenarios and catches 15 of 122 changed mutants, all already caught by other oracles, and 0 of the 64 that none catch. On outside engines it is a precise, low-recall screen. | docs/MINING.md |
| Can the oracle gap be closed? | Mostly, where a specification exists. A model writes an independent reference from the written description only; a program compares it to the recorded run (sandboxed, pre-registered gates, 0 false alarms on the clean simulators). It catches 38 of the 55 mutants that nothing else sees (orbit 2 of 5, cart-pole 13 of 19, traffic 23 of 31) and 12 of 16 on the held-out planted sets, which takes the share of behavior-changing mutants that some check catches from 48% to 79% (pre-registered bar: 70%). Caveats: the traffic reference needs a precise written spec (it checks conformance, not unspecified behavior), one traffic draw failed its clean gate and the spec was reworded (disclosed), orbit and cart-pole replicate exactly on a second independent draw, and rigid-body is not covered (its uncaught mutants are mostly in the controller, whose law the description does not specify). | docs/REFERENCE_ORACLE.md |
| Does an accepted report prove the explanation is right? | No. A red team of 8 contradicting forgeries on 47 real failures: 0 accepted. But a true set of numbers with a false story about the cause is accepted (4 uncheckable kinds quantified). An opt-in strict mode closes most of that (wrong-field stories and valueless citations 0 accepted, invented numbers 92 to 95% refused) but rejects about half of honest reports, so it is an audit tier, not a gate. | docs/VALIDATOR.md |
| Does it flag clean simulators? | Controls repeatedly caught weaknesses in my own simulators (rigid-body v1, the cart-pole scenario envelope); all are reported. | docs/CARTPOLE_DOMAIN.md |
Costs: about $0.04 to $0.09 per agent run; about $35 of API spend in total, capped per run, per evaluation, and by a lifetime ledger.
| Piece | What it does |
|---|---|
| Any simulator | a JSON model card + a program that prints named signals per step: the hunt, shrinker, agent and spec-to-rules run on it with no C++ domain (docs/GENERIC_DOMAIN.md) |
| Four domains | rigid-body physics (13 seeded bugs), orbits (3), cart-pole control (3), a traffic-signal state machine (3); same tools, rules, shrinker and agent |
| Tester agent | v1 (frozen) and v2 (counterfactual-checked reports, frames in the loop); a triage mode that explains a failure found by a fuzzer |
| Deterministic tools | run_scenario, check_rules, inspect_state, render_frames, shrink, replay; also as a local HTTP API and an MCP server, no model needed (docs/API.md) |
| Repair loop | the model proposes a fix for a failing simulator; the engine compiles it in a sandbox and accepts it only if every trace matches the unmodified simulator (docs/REPAIR.md) |
| Spec to rules | the model proposes invariants from a description; the engine keeps those that hold on clean runs (docs/SPEC_RULES.md) |
| Outside engines | unmodified Box2D and REBOUND, plus any program speaking the JSON-lines protocol, with a per-request deadline (docs/PROTOCOL.md) |
| Differential testing, regression tests | rc_diff compares two engines; rc_repro turns a failure into scenario.json + test.sh with JUnit output; a GitHub Action re-runs bundles on pull requests |
| Viewer and reports | local web UI (./build/rc_demo): Scan (random-test a built-in simulator, with or without a hidden bug; each failing rule gets a shrunk reproduction that opens in the replay), Oracle gap (which checks see which bugs, from the stored results), replay with checked citations, live agent runs against the real API, a scoreboard, and a single-file offline report per finding (scripts/share_report.sh) |
| Evaluation | locked sets, random baseline, controls, Wilson intervals, paired tests, an ablation, an external scenario corpus (data/external_scenarios/), CI on Linux and macOS |
- Seeded bugs were planted by the author; they are controlled test cases, not discoveries. The only claims about real software are in
docs/EXTERNAL_ENGINE.mdand they are modest: Box2D shows soft-contact penetration and, with continuous collision off, tunneling; REBOUND's fixed-step integrators drift at coarse steps as expected; one hang in a default mode is reproduced but not confirmed as a defect. - Every bug in the triage and spec evaluations had been seen in earlier work, so none of those is fresh held-out.
- Sample sizes are small (5 to 10 seeds per bug). Seeds label independent, nondeterministic runs; the model is not seeded.
- "Localized" in the triage evaluation means the counterfactual changed a hidden trigger field. It does not grade the written explanation, and a
hand audit found the older keyword scorer overstates (
docs/SCORER_AUDIT.md). - Controls on a clean simulator are mostly clean; the exception (rigid-body heavy-light squeeze, 2 of 9 control runs in the ablation) is in
docs/RESULTS.md. - Attribution ("the failure is due to the seeded bug") is checked post hoc by replaying on the clean simulator.
- Determinism is per platform: a run is bit-for-bit reproducible on the same machine type, but libm differences change simulated traces on Linux compared
with macOS. CI found this; pinned trace hashes are checked on macOS only. The model-free evaluations (random baselines, red team, corpus) were re-run on Linux and match the macOS tables exactly; the model-dependent ones were not re-run on Linux, but the 244 recorded model reports re-validate identically there (
docs/ROBUSTNESS.md). - A validated report establishes that its cited numbers and the violated rule are real; it does not establish that the written cause is true (
docs/VALIDATOR.md). - No human-planted blind bug set exists yet (
docs/BLIND.mdis the mechanism and needs a second person). A model-planted set of 39 variants, locked by hash before any oracle ran, gives the same picture as the author-sampled mutants: hand-written rules fire on 14 of the 34 that change behavior, the miner on none of the other 20. - Differential testing works when two engines are numerically close; for contact dynamics it mostly reports chaos.
- The wall-clock budget of the random baseline is not applied to the agent; the step budgets are identical.
cmake -S . -B build && cmake --build build -j && ./build/rc_tests
Optional, for the Box2D engine: brew install box2d before configuring. Everything else is plain C++17, no third-party dependencies, no network
unless --live is passed. CI builds and tests on Ubuntu and macOS.
./build/rc help # one front door: rc hunt | agent | triage | spec | repro | diff | eval | corpus | api | serve | spend | doctor
./build/rc doctor # check the installation, the key, the spend ledger
./build/rc_demo --open # viewer at http://127.0.0.1:8080 (recorded runs; Live needs MODEL_API_KEY)
./build/rc_hunt --engine builtin:o02 --domain orbit --scenarios 400 --bundles out/ # random hunt + regression bundles
./build/rc_diff --a builtin --b 'exec:./build/naive_orbit 0.03' --domain orbit --tol 0.1 --horizon 1
./build/rc_repro check --repro out/orbit_momentum_conserved --engine builtin:o02 # exit 1: failure reproduces
./build/rc_agent --live --domain orbit --bugs o02 --agent-version 2 --out runs/x # needs MODEL_API_KEY
./build/rc_api_server --mcp # the deterministic tools as an MCP server (no key)
./build/rc_corpus # run the 14 externally sourced scenarios
The API key lives in your environment (MODEL_API_KEY), never in the repo. The model client shells out to curl.
src/sim simulators and scenarios, src/rules declared rules and margins, src/tools the six deterministic tools, src/shrink shrinker,
src/agent the tester agent, src/model API client, replay/record/mock backends and budgets, src/external engine adapters, protocol,
hunt/repro/diff tools, src/spec spec-to-rules, src/api the API/MCP server and corpus runner, src/eval evaluation harness,
src/viewer + src/demo + web the viewer, data hidden ground truth, locked evaluation sets and external scenarios (harness only), results
generated outputs, docs write-ups, examples/external sample simulators and adapters. WHAT_WE_BUILT.md lists what was created during the hackathon.
License: MIT. REBOUND, used only as an external engine, is GPLv3 and is not included in this repository.