PRD-002: a rederivation eval, and what its dev baseline found

AGENTS.md

@@ -19,6 +19,7 @@ uv run explainer coldread <slug> --run        # first-viewing read by a fresh ag
 uv run explainer findings --stats             # how many review findings authors adopt, per tool
 uv run explainer eval chat                    # chat answers under each system prompt, graded blind
 uv run explainer eval artifact --split dev    # held-out subjects built under each condition, read and graded blind
+uv run explainer eval rederive --builds <run> # a fresh reader works out a result each explanation never states, graded blind
 uv run explainer render <slug> --draft        # fast silent layout pass
 uv run pytest -q                              # toolkit tests
 ```

CHANGELOG.md

@@ -1,5 +1,14 @@
 # Changelog
 
+## Unreleased
+
+Rederivation: can a reader work out what an explanation never states?
+
+- **A rederivation target for every subject** in `evals/artifact/questions.yaml`: a question the explanation was never asked to answer, the expected steps (building blocks a good explanation gives, and derived steps), and the result. A fresh agent checked each one (`explainer eval targets`, `evals/artifact/rederive-checks.json`), and all 30 were frozen before the first run.
+- **The rederivation stage** (`explainer eval rederive --builds <run>`; `eval artifact` runs it too). A fresh reader works the target out from the renderings, naming the part of the explanation each step rests on; a grader marks each expected step supported, unsupported, missing, or stated, blind, in three passes. Stated steps are left out of the score. [docs/evals.md](docs/evals.md#rederivation).
+- [What it found](docs/evals.md#what-the-dev-baseline-found) on the v0.8 dev builds: it is also near its ceiling (90–100% of steps supported). Under the rule fixed before the run, one pair separates, none > published, by about two steps of 38, from one subject where the published explanation lacks part of a building block (when the Sun's and Moon's tides add), a gap the quiz also found. Reported with it: one target that overclaims (float-sum), explanations that state the derived steps in 3 of 10 subjects, and a partial building block graded inconsistently.
+- `explainer eval human` and `evals/artifact/human-check.md`: a person works out a target, graded with the same rubric.
+
 ## 0.8.0 — 2026-10-19
 
 Measure the toolkit, not the examples.

CLAUDE.md

@@ -19,6 +19,7 @@ uv run explainer coldread <slug> --run        # first-viewing read by a fresh ag
 uv run explainer findings --stats             # how many review findings authors adopt, per tool
 uv run explainer eval chat                    # chat answers under each system prompt, graded blind (needs the claude CLI)
 uv run explainer eval artifact --split dev     # held-out subjects built under each condition, read and graded blind
+uv run explainer eval rederive --builds <run>  # a fresh reader works out a result each explanation never states, graded blind
 uv run explainer render <slug> --draft        # fast silent layout pass
 uv run explainer render <slug> --review       # 1080p60, voice, captions, chapters, review sheet
 uv run pytest -q                              # toolkit tests

ROADMAP.md

@@ -221,6 +221,7 @@ Rounds 2 and 3 also produced general rules: names match pictures (call it an L o
 - The examples are frozen as reference implementations and regression fixtures: `explainer check` must pass; no more review rounds.
 - `evals/artifact/questions.yaml`: 30 held-out subjects (10 dev, 20 test), none an example or a chat-eval question, each with a fixed quiz of four questions; the last is a prediction about a case the explanation need not mention.
 - `explainer eval artifact`: per subject and condition, a fresh agent builds the explainer (none, the principles, or the explain skill at the quick or published tier), a second reads only the renderings and answers the quiz with a quote for each answer (an answer whose quote is not in the renderings scores 0), and a third grades blind. [docs/evals.md](docs/evals.md#the-artifact-eval).
+- Rederivation (PRD-002): each subject also has a frozen, model-checked target, a result its explanation was never asked to state. A fresh reader works it out from the renderings, naming the part each step rests on; a grader marks each expected step blind, in three passes. `eval artifact` runs it; `eval rederive --builds <run>` runs it on saved builds. [docs/evals.md](docs/evals.md#rederivation).
 
 **Rules.**
 
@@ -233,12 +234,14 @@ Rounds 2 and 3 also produced general rules: names match pictures (call it an L o
 - The quiz is at its ceiling (94–100% in every condition, with a strong reader and with Haiku): a model reader cannot measure the difference. The same holds for the published tier's blind test.
 - The principles alone cut length 40% and rank first on effort in every grading pass, at the lowest cost. The quick tier is shortest but ranks below them and makes the most errors; the published tier costs 13× as much for no measured gain.
 - The first run was invalid: builders inside the repo loaded its CLAUDE.md in every condition. Builds now run outside every project.
+- Rederivation is also near its ceiling (90–100% of steps supported). One pair separates under the rule fixed before the run, none > published, by about two of 38 steps, from one subject (two-tides) where the published explanation lacks part of a building block (when the Sun's and Moon's tides add), a gap the quiz also found. Two target defects and three subjects whose explanations state the derived steps are reported with it ([docs/evals.md](docs/evals.md#what-the-dev-baseline-found)).
 
 **Next.**
 
 - A person ranks a sample (principles, quick, published on three subjects); the grader's ranking is used only if it agrees.
 - If it agrees: decide what the skill keeps. Candidates: the model file and the reviews only for the published tier, or the reviews dropped where they do not change the result.
 - A quiz that discriminates: harder subjects, or a reader limited in another way (time, words read), since a weaker model was not enough.
+- Open: whether PRD-003 adopts rederivation as its measure. With a strong model reader it mostly re-detects missing building blocks. Options: a weaker or constrained reader, harder targets with more derived steps (and a checker that tests "not stated" against real explanations), or human readers (`evals/artifact/human-check.md`). Recommendation: get two or three human checks before adopting it.
 - `plugin/skills/explain/references/html.md` says "the seven understanding questions from principles.md §7", but §7 has eight. Fix it with the principles rewrite, because skill text is measured.
 - The blind test (`explainer_kit/review.py`) does not ask "Could anything be cut with no loss?" Decide whether it should when the objective becomes least *wasted* effort.
 

docs/cli.md

@@ -22,7 +22,10 @@ With the plugin installed, run `explainer <command>` from any project. In the Ex
 | `explainer say "<text>" [--voice V] [--speed S]` | Synthesize one line and play it. Use it to choose a voice or test a lexicon spelling. |
 | `explainer frames <slug> [--draft] [--every N]` | Rebuild the contact sheet from an existing render. |
 | `explainer eval chat [--conditions name=source ...]` | Answer the questions in `evals/chat/questions.yaml` under several system prompts with fresh `claude -p` calls, grade them blind, and write a report. See [Evals](evals.md). |
-| `explainer eval artifact [--split dev\|test\|all] [--only id ...] [--conditions name=source ...]` | Build an explainer for each held-out subject in `evals/artifact/questions.yaml` under each condition (`none`, `principles`, `git:<rev>`, a file, `quick`, `published`), have a fresh reader answer its quiz from the renderings alone, grade blind, and write a report. A rerun into the same `--out` resumes. See [Evals](evals.md#the-artifact-eval). |
+| `explainer eval artifact [--split dev\|test\|all] [--only id ...] [--conditions name=source ...]` | Build an explainer for each held-out subject in `evals/artifact/questions.yaml` under each condition (`none`, `principles`, `git:<rev>`, a file, `quick`, `published`), have a fresh reader answer its quiz from the renderings alone, grade blind, and write a report; then run the rederivation stage on the same builds. A rerun into the same `--out` resumes. See [Evals](evals.md#the-artifact-eval). |
+| `explainer eval rederive --builds <run> [--only id ...] [--out folder]` | The rederivation stage alone, on the builds of an earlier artifact run (only read, never changed): in three passes, a fresh reader works out each subject's target question from the renderings and names the part each step rests on, and a grader marks each expected step blind. Writes `rederive.md` and `rederive-scores.json`. See [Evals](evals.md#rederivation). |
+| `explainer eval targets [--only id ...]` | Check each rederivation target in `evals/artifact/questions.yaml` with a fresh agent, and record the verdicts in `evals/artifact/rederive-checks.json`. Only targets changed since their last check are checked. |
+| `explainer eval human --builds <run> --chain <file>` | Grade a person's working on a rederivation target with the same rubric as the model readers. See [`evals/artifact/human-check.md`](../evals/artifact/human-check.md). |
 
 ## `check`
 

docs/evals.md

@@ -46,6 +46,55 @@ Effort rank: the grader's ranking of each subject's four explanations by the rea
 
 Two faults in the first run, both fixed: builders ran inside this repo, so Claude Code loaded its CLAUDE.md (and the principles) in every condition, including `none`; and the grader read Markdown with inline code removed, so formulas looked like blanks. Builds now run in a temp folder outside every project, with a guard.
 
+## Rederivation
+
+The quiz checks what an explanation says. A reader who has grokked a subject can do more: from its building blocks, work out a result nobody told them. An explanation of entropy as "the average length of the best code for a source" never mentions KL divergence, yet a reader who holds that idea can derive it: messages from p sent with a code built for q cost −log q each instead of −log p, so the average extra cost is Σ p log(p/q), never negative, because no code beats the one built for the true source. The rederivation stage tests that, on the same held-out subjects.
+
+**Targets.** Each subject in `evals/artifact/questions.yaml` has a `rederive` target: a question the explanation was never asked to answer, 3–6 expected steps, and the result. Each step is a building block (an idea a good explanation gives) or a derived step (worked out from the steps before it). A fresh agent checked every target (`explainer eval targets`, verdicts in `evals/artifact/rederive-checks.json`) against five tests: every step and the result are true; every building block comes from the subject's question, the quiz's expected answers, or background this reader already has; every derived step follows from earlier steps plus that background and brings in no new fact about the subject; each step is one any valid derivation needs; and a good answer to the subject's question would not state the result. The second and third tests matter most: a target that needs an idea a focused explanation may leave out would reward length, not understanding. All 30 passed (two after a revision) and were frozen in one commit before the first run; a test fails if a results folder's targets differ from the file.
+
+**Reader and grader.** The reader gets the renderings and the target's question, never the steps or the result. It works the question out step by step, and for each step names the part of the explanation it rests on (a heading, paragraph, figure, or a control on a page) and says what that part gives it. Reader and grader get the same rule: background this reader already has may be used without a citation; every fact about the subject must come from a part the reader names. The grader sees each subject's explanations and the readers' working under shuffled labels, and marks each expected step, credited in any order, merged, split, or by an equivalent route:
+
+- **supported:** made, and the cited parts carry the ideas it rests on (or it uses only background on top of earlier steps);
+- **unsupported:** made from the reader's own knowledge, or from a part that does not carry the idea;
+- **missing;**
+- **stated:** a derived step the explanation states outright, so the reader looked it up. A stated step, or a stated result, is left out of the score and counted, so a long explanation is neither rewarded nor penalized for spelling out what the reader was meant to derive.
+
+The headline is the **supported** share of expected steps; **raw** is the share made at all; **result right** the share of results that match. A run makes three **passes**: each is a fresh reader run over every build, graded once under its own label shuffle, so the spread covers both the reader's and the grader's variation. **The separation rule**, fixed before the first run: condition A beats condition B when A's supported score is higher than B's in all three passes.
+
+```bash
+uv run explainer eval rederive --builds evals/artifact/results/2026-10-04-dev   # the stage alone, on saved builds
+uv run explainer eval artifact --split dev                                     # builds, quiz, then this stage
+```
+
+### What the dev baseline found
+
+Dev baseline, 2026-10-20: the 40 v0.8 builds (10 subjects × 4 conditions), read and graded three times by the `claude` CLI's default model; no new builds. [Report](../evals/artifact/results/2026-10-20-dev-rederive/rederive.md).
+
+| condition | supported (3 passes) | result right | stated steps | words | quiz (grounded) |
+|---|---|---|---|---|---|
+| none | 100%, 100%, 100% | 100%, 100%, 100% | 2, 2, 3 | 1,689 | 100% |
+| principles | 95%, 100%, 97% | 89%, 89%, 100% | 4, 4, 5 | 1,006 | 99% |
+| quick (skill) | 92%, 100%, 90% | 100%, 100%, 89% | 2, 2, 2 | 606 | 95% |
+| published (skill) | 95%, 95%, 95% | 89%, 78%, 89% | 3, 3, 3 | 1,016 | 95% |
+
+**Under the separation rule, one pair separates: none > published** (100% against 95% in every pass). No other pair does.
+
+**What that result is made of.** About 38 expected steps count per condition, so 95% is two steps short. The gap comes from one subject, two-tides, and from half of one building block. Block 2 bundles two ideas: the Sun's tide is about half the Moon's, and at new and full Moon the two line up and add. The published explanation gives the ratio but never says when the tides add (no spring or neap tides), so the grader marked the block unsupported in all three passes. That is the gap the quiz already found (published scored 6 of 8 on two-tides, 0 on the spring-tide question). The quick explanation also gives the ratio, and only mentions in a "not covered" line that the Sun's bulges add to or partly cancel the Moon's; the grader marked its block unsupported, supported, and missing in the three passes, so it does not separate.
+
+**How much a single subject moves between passes.** The same quick two-tides build scored 3/4, 4/4, and 1/4 supported steps in the three passes. A difference of a step or two in one subject is within what one fresh reader and one grading change on their own.
+
+**Known defects, found after the run** (the targets are frozen, so they stay and are reported):
+
+- **The float-sum target overclaims.** Its last derived step argues that adding 0.1 ten times cannot land exactly on 1.0 because each addition rounds. Rounding errors can cancel (`0.1` added five times does give exactly `0.5`), so the honest answer from the building blocks is "not guaranteed", which is what the principles and published readers said. They were marked wrong in passes 1 and 2; in pass 3 the published reader gave the same hedged answer and was marked right, so the grader was not consistent either. This lowers principles' and published's "result right" (published's also falls on two-tides in passes 2 and 3) and cost each one supported step in pass 1. The separation does not depend on it: two-tides alone carries none > published.
+- **Real explanations stated derived steps outright in 3 of 10 subjects,** more often than the target checker predicted: the gps-relativity clock argument (three conditions), the greenhouse no-lapse-rate case (both derived steps, in principles, quick, and published), and the two-tides right-angle step (none and principles). Those steps are left out of the score, so for greenhouse three conditions were scored on building blocks alone.
+- **Compound building blocks are graded inconsistently when half present.** The quorum block "any two majorities share a server, so the next group sees every committed write" is half present in the principles and quick explanations (they give the overlap, for conflicting writes, not for reads); the grader marked it unsupported in two of three passes. The two-tides block above is the same kind, and it carries the only separation.
+
+**Does the order follow length?** Roughly. On the mean supported share the order is none (100%) > principles (97%) > published (95%) > quick (94%); by words it is none (1,689) > published (1,016) ≈ principles (1,006) > quick (606). The one separation goes in length's direction. The word count also leaves out what the grader reads in pages' scripts, and only none made pages (5 of 10).
+
+**What we take from it (interpretation, not the rule).** With a strong model reader, rederivation mostly re-detects missing building blocks, which the quiz already catches; it does not yet show a difference in understanding between conditions. The one separation is small, comes from half of one building block in one subject, and favors the longest explanation. Whether PRD-003 adopts this stage, and what to change first (a weaker or constrained reader, harder targets with more derived steps, or human readers), is open: see ROADMAP §15.
+
+**Human check:** none yet. A brief for testers is in [`evals/artifact/human-check.md`](../evals/artifact/human-check.md); results will be recorded here.
+
 ## The chat eval
 
 `evals/chat/questions.yaml` holds six questions about phenomena (why ice floats, why TCP starts slowly, how a wing makes lift, why a hash table is fast, why we see one side of the Moon, why salt melts ice). For each one it lists: