[ifrit98]

Implements PRD-002 (plan): can a reader work out a result that an explanation never states?

What's in it

• 30 rederivation targets in evals/artifact/questions.yaml. Each has a question the explanation was never asked to answer, 3–6 expected steps (building blocks plus derived steps), and a result.
  • A fresh agent checked every target (explainer eval targets, results in evals/artifact/rederive-checks.json).
  • All 30 were frozen in 6b61cdf, before any reader or grader code existed. A test pins them.
• The rederivation stage (explainer_kit/rederive.py).
  • A reader works the target out from the renderings, naming the part of the explanation each step rests on.
  • A blind grader marks each expected step as supported, unsupported, missing, or stated.
  • Every run makes three passes. The separation rule was fixed before the first run.
  • It runs with explainer eval rederive --builds <run>, and eval artifact now runs it too.
• A human check: explainer eval human plus evals/artifact/human-check.md.
• A dev baseline on the saved v0.8 builds, with no new builds: evals/artifact/results/2026-10-20-dev-rederive/. The v0.8 record is unchanged.

What it found (docs/evals.md)

• The stage is near its ceiling. 90–100% of steps are supported in every condition.
• One pair separates under the rule: none > published.
  • The gap is about 2 steps out of 38, all from one subject (two-tides).
  • There, the published explanation gives the Sun-to-Moon tide ratio but never says when the two tides add. The quiz already caught that gap.
  • The order roughly follows length.
• Reported alongside:
  • one target that overclaims (float-sum);
  • derived steps stated outright in 3 of 10 subjects;
  • compound building blocks graded inconsistently when only half present.
• Conclusion: this is not yet a measure of understanding beyond the quiz. Fixing the instrument comes next, as a separate PRD.

Review

A final whole-branch review led to these fixes, each with a test that failed first:

• A failed build is now forced to score nothing, and the report shows it.
• A grader reply that doesn't fit is retried once, and its raw marks are kept.
• The test-split targets are pinned to the freeze.
• Two doc corrections: the gap behind the separation is half a building block, not a whole one, and the float-sum consistency claim was overstated.

Test plan

• uv run pytest -q: 264 passed, offline.
• uv run explainer check passes.
• git diff main -- plugin/skills explainers evals/artifact/results/2026-10-04-dev* is empty.
• Results scanned for paths, usernames, emails, and keys: none.
• CI green.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DuVfjP13T9TDZwFJdP5uui