Repository navigation
M8: optional Jev/System One escalation (--escalate jev) - #68
Conversation
|
A4 live + bonus fix (2026-10-19) — |
Two-axis review remediation — all findings addressed in
|
c09633d to
ca18792
Compare
Port of aui-cli/src/typesafe.rs onto our ActionSpace: Instinct decides first; JevEscalation runs only on Abstain behind EscalatingPolicy. One blocking System One call per abstain — an `operation` head plus one `<kind>_target` head per element-bearing kind, keyed by id region so the composed reply re-enters the same closed-menu validator as M7. Fail-closed throughout: off-menu choices, uncovered/extra probability keys, bad sums, out-of-range or non-argmax answers, transport errors, and missing TYPESAFE_API_KEY all surface as escalation:* → Blocked. Requests go uncompressed so scripted endpoints stay std-only. typesafe-sdk 0.3 is an optional dep; default/instinct builds carry no typesafe in cargo tree. --escalate conflicts with --escalate-cmd; the text tier stays command-only. A4 bench arm added; not run (no key). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- 40/40 jev.rs mutants caught (was 13 missed): keys_for / element_kind_of / model_non_empty coverage, range-check boundary matrix (inclusive 0/1, negative-vs-sum isolation, NaN), criterion label text, and a std-only in-crate System One mock asserting request model/state/questions. - PROTOCOL.md: snapshot since/compact/fingerprints, revision op, TreeDiff shape, expect_fingerprint gate, hello.features. - INTEGRATING.md: host-side expect_fingerprint contract + checklist. - PERF.md: M2–M8 numbers. RECIPES.md: instinct resolve. README.md: run --goal, locate/act MCP, feature flags. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A4 live data: on T08 (two identical "Buy milk" rows, refuse task) Jev picked one of the identical delete targets — a coin flip that counts as wrong_action. No endpoint can resolve identical (kind, label) contenders; the menu itself is ambiguous. EscalatingPolicy now returns the local abstain without calling the fallback — the model faces the same identical options, so paying for its guess violates abstain-over-guess. Same fix protects --escalate-cmd. Also fix bench LLM harness for hosts that ship the v2 challenge handshake (HMAC auth, never the raw token), pydantic tool-use serialization in the byte counter, and explicit api_key for anthropic>=0.75 credentials-file discovery. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
All previously blocked gates now have measured data: - A0 baseline (never recorded before): 65/80, 13,409 in_tok p50, $0.00609/passed. T14 = driver quits the app on a refuse task. - GATE-T PASS: A1 65/80 wrong=0 at 4,550 in_tok p50 = 34% of A0. - GATE-E PASS: A3 80/80 wrong=0, 61ms p50, 15 escalations total, $0.0000306/passed (0.5% of A0). - A4: 75/80 — Jev coin-flipped T08 identical-row menu (5 wrong, pre-guard binary); with the ambiguity guard T08 is 5/5 wrong=0. - escalate.rs mutation gate: 38 mutants, 28 caught, 0 missed. New tests: ambiguity guard (2), subprocess round-trip via /bin/sh stubs (3, unix-gated), choose_request wire shape, text boundary 256. bench report skips non-benchmark side files and tolerates sparse rows (harness-error rows lack wall_ms). Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…rage
Standards fixes:
- ci.yml: test/clippy under instinct + jev features; bench job runs r1 and
a2 arms so the M2-M8 test surface actually compiles in CI
- bench/run.py: drop raw-token fallback on challenge-less hosts (PROTOCOL
forbids tokens on the wire)
- runner.rs: INSTINCT_VERSION_REQ const + manifest pin-test (was "0.1" literal)
- run_cmd.rs: remove dead json!(receipt) + unused import
- SECURITY.md: document --escalate-cmd env inheritance (GPUI_AGENT_TOKEN,
TYPESAFE_API_KEY reach the child endpoint)
- lib.rs: carve jev (blocking network + f64 reply validation) out of the
crate-level "no floats, no network, no model" contract wording
Spec fixes:
- OfferedAction gains NodeState {value,checked,states}; choose/jev request
actions now carry state per PRD M7.2 (wire + regression tests)
- act replies conform to M4.5: {"done":{op,id,text?},"rev":N,"diff":{...}}
via post-dispatch snapshot since=<pre-rev>; abstains and blocked/wait
controls emit {"abstain":{reason,candidates}}
- tests/props.rs: proptest — a Choice always names an offered action and
never a dangerous one (M2.5 abstain-never-dispatches); committed
regression seed guards the duplicate-id shrink
- runner stale bound aligned to two re-decides (M5.3 "at most twice")
- text.rs: multi-literal no-label-match returns TextError::Ambiguous
instead of the untyped[0] fallback (spec-strict)
- mcp.rs: declare k in the act tool schema
- PRD §11 addendum documents every deliberate deviation
- RESULTS.md: A2 re-run at n=20 — 320/320, wrong=0, wall p50 ~49ms
Verified: fmt + clippy -D warnings x3 configs, workspace tests x3
(0 failures), escalate.rs mutation gate 38 mutants / 0 missed, live
spot-checks a3 10/10 + a4 6/6 wrong=0.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
395f0a5 to
66716b8
Compare
Summary
JevEscalationingpui-agent-policybehind new optionaljevfeature — port ofaui-cli/src/typesafe.rsonto ourActionSpace. Oneblocking::Client::system_onecall per abstain: anoperationchoice head plus one<kind>_targethead per element-bearing kind, keyed by id region so composed replies are valid offeredActionIds.EscalatingPolicy: local Instinct decides first; Jev runs only onAbstain— abstain never becomes "top candidate wins". The text tier stays command-escalation-only.--escalate jev(conflicts with--escalate-cmd). MissingTYPESAFE_API_KEYis a clear config error; a binary built withoutjevnames the missing feature.TYPESAFE_MODEL/TYPESAFE_BASE_URL/TYPESAFE_DEFAULT_MODELhandled by the SDK.escalation:→Blocked: off-menu choice, uncovered/extra probability keys, sum ≠ 1, out-of-range/non-finite probabilities, non-argmax choice, transport errors, empty operation menu.typesafe-sdk 0.3optional —cargo treeshows notypesafein default orinstinctbuilds (extends the M1.7 guard).Compression::Noneso scripted endpoints stay std-only.bench/arms.tomlgains arma4(--escalate jev). A4 ran live (jev-latest): 75/80 → ambiguity guard →a4+guardwrong=0; recorded in RESULTS.md.Test plan
jev.rsunit tests —build_questionscoverage (heads, id-region keys, criterion label text, controls-only space),validate_choice(off-menu, uncovered, extra keys, bad sums, out-of-range incl. boundary matrix + NaN, non-argmax, argmax accept),keys_for,element_kind_of,model_non_empty, plus a std-only in-crate mock System One server asserting request model/state/questionsrun_jev.rsintegration tests over a std-only mock — menu pick dispatches, Instinct choice never calls Jev, non-argmax/bad-sum/uncovered/off-menu → blocked, missing key → clear error; request assertsmodelfromTYPESAFE_MODELcargo test --workspacegreen in default,instinct, andjevconfigs (25 suites under jev)cargo clippy --workspace --all-targets --features jev -- -D warningsclean; also default + instinct configs;cargo docclean under-D warningscargo treeunder default/instinct has notypesafecargo mutants -p gpui-agent-policy --features jev -f jev.rs→ 40 caught / 0 missed / 8 unviablejev-latest, n=5 × 16): 75/80 — Jev coin-flipped T08's identical-row menu (5 wrong) → ambiguity guard (7cdfa37): identical(kind,label)contenders short-circuit before any endpoint →a4+guard5/5, wrong=0. RESULTS.md §M8 + §Final summary.Generated with Devin