From fcec7714ade455bf9caa7eeced45a439cc7721e5 Mon Sep 17 00:00:00 2001 From: ifrit98 Date: Tue, 20 Oct 2026 05:00:56 -0400 Subject: [PATCH 01/13] PRD-002: a rederivation eval, and its plan Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DuVfjP13T9TDZwFJdP5uui --- docs/prds/PRD-002-rederivation-eval.md | 183 ++++++++ .../plans/2026-10-20-rederivation-eval.md | 440 ++++++++++++++++++ 2 files changed, 623 insertions(+) create mode 100644 docs/prds/PRD-002-rederivation-eval.md create mode 100644 docs/superpowers/plans/2026-10-20-rederivation-eval.md diff --git a/docs/prds/PRD-002-rederivation-eval.md b/docs/prds/PRD-002-rederivation-eval.md new file mode 100644 index 0000000..f55894b --- /dev/null +++ b/docs/prds/PRD-002-rederivation-eval.md @@ -0,0 +1,183 @@ +# PRD-002: A rederivation eval + +Status: approved, 2026-10-20 · Part 2 of 3 (PRD-001: ship v0.8; PRD-003: Olah-compass principles) + +## Why + +The v0.8 quiz is at its ceiling. Every condition scores 94–100%, because a model reader answers from any competent explanation (`docs/evals.md`). So the quiz cannot tell PRD-003 whether a change to the principles helps. + +An explanation here aims for the reader to grok the subject: once they hold its building blocks, they can work out results nobody told them. Rederivation tests exactly that. + +An example from outside the 30 subjects: an explanation of entropy as "the average length of the best code for a source" never mentions KL divergence, but a reader who holds that idea can derive it. + +- Messages from p sent with a code built for q cost −log q each instead of −log p. +- So the average extra cost is Σ p log(p/q). +- Its properties follow without being taught: + - it is never negative, because no code beats the one built for the real source; + - it is zero only when p = q; + - it is not symmetric; + - it is infinite when q gives zero probability to something p produces. + +Each quiz already ends with a one-step "predict" item. This PRD adds the several-step version and grades the chain of reasoning, not only the answer. + +The reader is a model that already knows most results, so a right answer alone proves nothing. The fix: every step must rest on a part of the explanation that the reader names, and the grader checks that the part carries the idea. + +## Goals + +### 1. A rederivation target for every subject, frozen before any run + +Each of the 30 subjects in `evals/artifact/questions.yaml` gets a target with three parts: + +- **the question** the reader works out; +- **the expected steps:** 3–6 of them, one sentence each. Each step is an idea that any valid derivation needs, not the bookkeeping of one route. Each is marked as one of: + - a **building block**, which a good explanation gives; + - a **derived step**, which the reader works out from earlier steps. +- **the result:** the expected conclusion, specific enough to be right or wrong. + +Each target passes three tests: + +1. **Allowed building blocks only.** Every step uses one of: + - an idea from the subject's question; + - an idea from the quiz's expected answers; + - background at the reader's level. That level is set by the `reader` line in `questions.yaml`: school mathematics and science, and basic programming. + + A target that needs an idea a focused explanation can leave out rewards length, not understanding. The conditions average from 606 to 1,689 words, so that bias would be large. PRD-003 would then tune the principles toward length, the opposite of their aim. +2. **Several steps:** at least three. +3. **Not stated:** a good answer to the subject's question would not state the result. + +Claude drafts the targets. A fresh `claude -p` checker sees the subject's question, its quiz with the expected answers, and the target, then applies the three tests and returns pass or fail with a reason. + +- A failed target is rewritten and checked again, at most twice. +- A subject whose target still fails gets no target, and the report lists it. +- The checker's verdicts are saved with the targets. + +All 30 targets, test subjects included, are committed in one commit before the first rederivation run. Writing them now means the PRD-003 changes cannot shape them. The owner may skim them; nothing waits on that. + +### 2. A rederive stage in the artifact eval + +**Reader.** The reader gets the renderings only, as today, and the target's question. It does not see the expected steps or the result. It works the question out step by step. For each step it gives: + +- the idea the step uses; +- the part of the explanation that idea comes from: a heading, a paragraph, a figure or diagram, or a control on a page; +- that idea in its own words. + +If the explanation lacks an idea the reader needs, it says so and stops there. A gap is evidence. + +**Outside-knowledge rule.** The same text goes in the reader prompt and the grader prompt: + +- background at the reader's level may be used without citing anything; +- any fact about the subject must come from a part of the explanation that the reader names. + +**Grader.** The grader works blind, under shuffled labels, as today. For each explanation it sees the text, the reader's chain, and the target's expected steps and result. The expected steps are a checklist of ideas, not a script. A step counts when the reader's chain establishes the same idea in any order, whether merged with another step, split in two, or reached by an equivalent route. The grader marks each expected step as one of: + +- **supported:** the reader made the step, and either the cited parts carry the ideas it rests on, or it uses only reader-level background on top of earlier steps; +- **unsupported:** the reader made the step from its own knowledge, or from a part that does not carry the idea; +- **missing;** +- **stated:** a derived step that the explanation states outright, so the reader looked it up instead of working it out. + +The grader also marks whether the result is right, and whether the explanation states the result outright. + +**Scores.** Each condition gets three scores: + +- **supported:** the share of expected steps that are supported; +- **raw:** the share made at all; +- **result right.** + +These mirror the quiz's grounded and raw scores. + +A stated derived step, or a stated result, is left out of that explanation's score: out of both the count and the total. The report counts these per condition. That way a long explanation is neither rewarded nor penalized for spelling out what the reader was meant to derive. + +**Passes.** Every run of the stage makes three passes. In each pass, a fresh reader works through every build, and the grader grades it once under its own label shuffle. The spread across passes then covers both the reader's variation and the grader's. + +The stage runs in every `eval artifact` run from now on, because PRD-003 needs it on new builds. It can also run alone on the saved builds of an earlier run. + +### 3. A dev baseline on the saved v0.8 builds + +- **No new builds.** The results go in a new folder that reuses the 40 builds saved in `evals/artifact/results/2026-10-04-dev/` (10 subjects × 4 conditions). +- **Models.** The reader and the grader use the `claude` CLI's default model, as the v0.8 baseline did. +- **Three passes,** as defined in Goal 2, each reported. The v0.8 ranges came from three different gradings, one of them the faulty first run, so they are no model to copy. +- **The report** gives, per condition, the three scores (with their range across passes), next to words and the v0.8 quiz score, plus a table per subject. + +### 4. The result is written up, whatever it is + +`docs/evals.md` gets a "Rederivation" section. It says what the eval measures and gives the three target tests. Then it reports what the dev baseline found, judged by the rule below. + +**The separation rule, fixed before the first run.** Condition A beats condition B when A's supported score is higher than B's in all three passes. + +The section says: + +- which pairs of conditions separate; +- whether the order follows word count, which would mean the score measures length; +- if no pair separates, that the eval does not yet rank the conditions, and that PRD-003 should not adopt it as its measure without changes. + +The "Next" list in ROADMAP §15 and the "Unreleased" section of `CHANGELOG.md` record the stage and its result. + +### 5. A human check that does not block + +`evals/artifact/human-check.md` is a one-page brief for a person who reads one explanation and works out its target on paper, before seeing the expected steps. Their chain is graded with the same rubric. `docs/evals.md` then records whether the person's score and the model reader's score on that explanation agree. + +PRD-002 is done without any human results; they are added when they come in. + +## Non-goals + +- **No change to the principles, the skills, or other measured text.** That is PRD-003. +- **No new builds and no run on the test split.** A test run needs new builds for all 20 test subjects. It runs when PRD-003 confirms a change. +- **No changed-world variants and no reader run without an explanation.** They were cut to keep this simple. +- **No change to the quiz, its scoring, or the v0.8 reports.** + +## Anti-goals + +- Changing a target, its steps, or the separation rule after seeing any rederivation score. +- A target that rewards length because it needs an idea outside the allowed building blocks. +- Claiming a separation the rule does not support, or presenting a 10-subject dev run as more than that. + +## Constraints + +- **The repo is public (MIT).** Targets and docs contain no personal or private-project context. +- **The shipped v0.8 record stays as it is.** `evals/artifact/results/2026-10-04-dev/` and `2026-10-04-dev-haiku-reader/` stay byte-identical: `git diff --exit-code` on them is clean. +- **The builder never sees a target.** Build prompts are unchanged. +- **The reader never sees** the expected steps or the result. +- **Grading stays blind** under shuffled labels. +- **Every stage saves its output,** and a rerun skips what exists, as today. +- **Tests run offline,** without the `claude` CLI, like the existing artifact eval tests. + +## Decided + +All decided by the owner on 2026-10-19. + +- **Rederivation is the test of grokking.** Once the reader has the building blocks, they can rederive the results and the rest of the theory from first principles. +- **A citation names a part of the explanation and restates its idea.** Word-for-word quotes are too strict, and an idea carried by a figure cannot be quoted. +- **The owner's time is not required.** Claude drafts the targets, a model checks them, and the owner may skim. +- **Simpler beats complete.** No changed-world variants and no control run without an explanation. +- **Human calibration happens when a tester is free.** It blocks nothing. +- **Targets are frozen before the first run.** + +## Calls made by Claude (check these) + +Approving this PRD approves these calls. Nothing else waits on the owner. + +- **The allowed-building-blocks test:** the question, the quiz's expected answers, and reader-level background. This keeps the score from rewarding length. +- **Targets for all 30 subjects now;** only dev runs in this PRD. +- **The separation rule:** higher in all three passes. +- **A pass is a fresh reader run, graded once under its own shuffle.** This triples the reader calls in every run. Reads are cheap next to builds, and only this design tests whether a difference survives the reader's own variation. +- **Credit for other routes.** Expected steps are a checklist of ideas, credited in any order or form. Each step must be one that any valid derivation needs. +- **Stated derived steps and stated results are left out of the score and counted.** They are neither rewarded nor penalized. +- **Scores:** supported, raw and result-right, mirroring the quiz. +- **The rederive stage joins every artifact eval run** from now on. +- **No release is decided here.** The owner decides at the end whether PRD-002 ships as v0.9.0 or waits for PRD-003. + +## Left to the builder + +- Field names in `questions.yaml`, and the prompt wording beyond the outside-knowledge rule. +- The CLI form for running the stage on an earlier run's builds. +- How the new folder reuses the builds, by copy or by reference, provided the v0.8 folders stay unchanged. +- Whether the grader sees a subject's four explanations in one call, as today, or one at a time, provided the labels stay shuffled. + +## Done when + +- Each of the 30 subjects has a target whose checker verdict is pass, or is listed as having none. The targets were committed before any rederivation result. +- The dev baseline is in a new results folder with three passes. Its report shows the scores next to word count and the quiz. +- `docs/evals.md` states the result under the separation rule, including "no separation" if that is the outcome. +- The v0.8 results folders are unchanged. +- `evals/artifact/human-check.md` exists. +- Tests pass offline, `explainer check` passes, and CI is green. diff --git a/docs/superpowers/plans/2026-10-20-rederivation-eval.md b/docs/superpowers/plans/2026-10-20-rederivation-eval.md new file mode 100644 index 0000000..8b983b2 --- /dev/null +++ b/docs/superpowers/plans/2026-10-20-rederivation-eval.md @@ -0,0 +1,440 @@ +# Rederivation Eval Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add a rederivation stage to the artifact eval, give all 30 subjects a frozen, model-checked target, and report a three-pass dev baseline on the saved v0.8 builds, whatever it shows. + +**Architecture:** +- One new module, `explainer_kit/rederive.py`, holds the targets (validation, hashing, checker), the reader, the grader, scoring, the run, the report, and the grading of a human chain. +- `artifact_eval.py` changes only where the reader's file copies are made: that code is extracted into `copy_for_reader` so both stages share it. +- The CLI gains three `eval` suites (`targets`, `rederive`, `human`), and `eval artifact` runs the rederive stage after it grades. +- The stage reads builds from a `--builds` folder and writes only to `--out`. That is how the baseline reuses the v0.8 builds without touching them. +- Work happens on branch `prd-002-rederivation` from `main`. + +**Tech Stack:** Python 3 / uv, pytest, PyYAML, the `claude` CLI through `explainer_kit.agent.ask`. + +**Spec:** `docs/prds/PRD-002-rederivation-eval.md` + +## Global Constraints + +- **Measured text does not change:** `git diff --stat main -- plugin/skills explainers` is empty at every commit. +- **The builder never sees a target.** In `artifact_eval.py`, `TASK`, `SKILL_TIER` and `build()` are unchanged: `git diff main -- explainer_kit/artifact_eval.py` touches only `read()` and the new `copy_for_reader`. +- **The v0.8 record stays byte-identical:** `git status --porcelain -- evals/artifact/results/2026-10-04-dev evals/artifact/results/2026-10-04-dev-haiku-reader` is empty at every commit. +- **Targets are frozen at the Task 2 commit.** Afterwards, `rederive` blocks in `evals/artifact/questions.yaml` never change. A test (Task 5) fails if a results folder's snapshot differs from the file. +- **The repo is public.** No committed file holds an absolute path, a username, an email address, a key, or private-project context. +- **The reader prompt and the grader prompt contain the same constant, `OUTSIDE_RULE`:** + > Background that this reader already has (see the reader above) may be used without a citation. Every fact about the subject must come from a part of the explanation that the reader names. +- **The separation rule, verbatim:** "Condition A beats condition B when A's supported score is higher than B's in all three passes." +- **A pass** is a fresh reader run over every build, then one grading per subject under its own label shuffle (`random.Random(k)` for pass `k`). +- **Tests run offline.** They stub `agent.ask` and `agent.subprocess.run` and never call the `claude` CLI. +- **The baseline uses the `claude` CLI's default model,** so no `--model` and no `--reader-model`. + +## Review Focus + +1. **A build with no renderings** (a failed build). Expect the reader to be skipped and every step marked missing. The run must not crash, and the report shows it. The test is in Task 3, `test_read_chain_skips_a_build_with_no_renderings`. +2. **A rerun after the targets changed.** Expect the run to refuse, naming the changed subjects, so no folder mixes two target versions. The test is in Task 5, `test_rerun_refuses_changed_targets`. +3. **Absolute paths leaking into saved output** (public repo). Expect the source folder saved relative to `--out`, and no prompt or temp path saved. The test is in Task 5, `test_outputs_hold_no_absolute_paths`. +4. **Grader replies that don't fit:** too few or too many marks, an unknown word, "stated" on a building block, or a missing label. Expect normalisation, not a crash or a skewed score. The test is in Task 4, `test_marks_are_normalised`. +5. **A page whose idea lives in a control.** Expect the grader to see the page's script as well as its visible text, so a reader citing a slider can be checked. The test is in Task 4, `test_grader_text_includes_page_scripts`. + +--- + +### Task 0: Branch + +- [ ] **Step 1:** Create the branch. In the PRD, change `Status: draft, 2026-10-19` to `Status: approved, 2026-10-20`; committing it before any run is what pre-registers the separation rule. Commit the spec and the plan. + +```bash +git switch -c prd-002-rederivation +git add docs/prds/PRD-002-rederivation-eval.md docs/superpowers/plans/2026-10-20-rederivation-eval.md +git commit -m "PRD-002: a rederivation eval, and its plan" +``` + +### Task 1: Targets — format, validation, and the checker + +**Files:** +- Create: `explainer_kit/rederive.py` +- Modify: `explainer_kit/cli.py`. Add `targets` to the `eval` suite choices; `cmd_eval` dispatches it. +- Create: `tests/test_rederive.py` + +**Interfaces:** +- Produces: + - `checks_file() -> Path`, which returns `project_root()/"evals"/"artifact"/"rederive-checks.json"`. + - `target_problems(t: dict) -> list[str]`. + - `target_hash(t: dict) -> str`: the first 12 hex digits of the sha256 of `json.dumps(t, sort_keys=True, ensure_ascii=False)`. + - `check_prompt(subject: dict, reader: str) -> str`. + - `check_targets(only: list[str] | None = None, model: str | None = None) -> Path`. +- The target format in `questions.yaml`, under each subject: + + ```yaml + rederive: + q: "…" + steps: + - {kind: block, text: "…"} + - {kind: derived, text: "…"} + result: "…" + ``` + +- The checks file maps `{subject_id: {"verdict": "pass"|"fail", "tests": {...}, "reason": str, "hash": str, "attempt": int}}`. + +- [ ] **Step 1: Write the failing tests** in `tests/test_rederive.py`. + - `test_target_problems`: + - a valid target (3 steps, at least one `derived`, non-empty `q` and `result`) gives `[]`; + - each of these gives a non-empty list: 2 steps; 7 steps; no `derived` step; kind `"lemma"`; empty `result`; missing `q`. + - `test_target_hash_is_stable_and_sensitive`: the same dict in a different key order hashes the same, and changing one step's text changes the hash. + - `test_check_prompt_shows_the_building_blocks`. For the first dev subject, given a target, the prompt contains: + - every quiz `expect`, the reader line, every step text with its kind, and the result; + - the phrases `"any valid derivation needs"`, `"would not state"`, and `"no new fact about the subject"`. + - `test_check_targets_records_and_resumes`. Monkeypatch `rederive.checks_file` to a tmp file and `ae.questions_file` to a tmp YAML with two subjects that have targets, and a stub `agent.ask` that returns: + + ```json + {"tests": {"correct": true, "blocks": true, "derived_follow": true, "steps": true, "not_stated": true}, "reason": "ok"} + ``` + + for one subject, and `"derived_follow": false` for the other. Assert: + - verdicts `pass` and `fail` (the verdict is computed in code as all five tests true, never read from the reply); + - each `hash` equals `target_hash`, and `attempt == 1`; + - a second call makes **no** `agent.ask` call; + - after editing the failing target and calling again, that subject has `attempt == 2` and only it was asked. +- [ ] **Step 2: Run them to verify they fail.** + Run: `uv run pytest tests/test_rederive.py -q` + Expected: FAIL. The module is not found. +- [ ] **Step 3: Implement the four functions in `explainer_kit/rederive.py`.** + - Module docstring style matches `artifact_eval.py`. + - `CHECKER_SYSTEM = "You are a careful scientist and teacher. You check test questions. Reply with JSON only."` + - The checker prompt asks for five booleans: + - `correct`: the steps and the result are true; + - `blocks`: every `block` step is an idea from the subject's question, from the quiz's expected answers, or background this reader already has; + - `derived_follow`: every `derived` step follows from the steps before it plus background this reader already has, and introduces no new fact about the subject (otherwise only explanations that happen to contain that fact could support it, and length wins); + - `steps`: there are at least three steps, and each is an idea any valid derivation needs, not one route's bookkeeping; + - `not_stated`: a good answer to the subject's question would not state the result. + + It also asks for a one-sentence `reason` that names any failing step. + - `check_targets` walks `ae.load("all", only)[1]`. It skips subjects without `rederive` and subjects whose saved hash matches. It sets `attempt` to the saved attempt plus 1 when the hash changed. It writes the file sorted by id, with a trailing newline. + - `cmd_eval` with `suite == "targets"` calls `check_targets(args.only, args.model)` and prints a count of pass and fail. +- [ ] **Step 4: Run the tests to verify they pass.** + Run: `uv run pytest tests/test_rederive.py -q` + Expected: 4 passed. +- [ ] **Step 5: Commit.** + `git add explainer_kit/rederive.py explainer_kit/cli.py tests/test_rederive.py && git commit -m "rederive: target format, validation, and the checker"` + +### Task 2: Draft, check and freeze all 30 targets + +This task is content work plus live checker calls (`claude -p`), about 30–60 calls. Nothing from Tasks 3–5 may exist in the working tree yet. The targets are written before the reader or grader prompts exist, and stay unchanged afterwards. + +**Files:** +- Modify: `evals/artifact/questions.yaml`. Add a `rederive` block to every subject, and add a header paragraph explaining the block and that the builder and reader never see the steps or the result. +- Create: `evals/artifact/rederive-checks.json` (generated). +- Modify: `tests/test_rederive.py` (one test). + +**How to draft a target.** The worked example below is from outside the 30 subjects and is not added to the file: + +```yaml +rederive: + q: "A code was built for the letter frequencies q of one language and is used for text from another, with frequencies p. How many extra bits per letter does it cost on average, and can that cost ever be negative?" + steps: + - {kind: block, text: "In the best code, a symbol of probability p gets a codeword of about -log2 p bits."} + - {kind: block, text: "Entropy is the average codeword length of the best code: the sum of -p log2 p."} + - {kind: derived, text: "With q's code, each letter costs -log2 q instead of -log2 p, so the average extra cost is the sum of p log2(p/q)."} + - {kind: derived, text: "No code beats the one built for the true source, so the extra cost is never negative, and it is zero only when p = q."} + result: "The sum of p log2(p/q) bits per letter (the KL divergence); never negative, zero only when p = q." +``` + +- A `block` step restates an idea from the subject's `q`, from a quiz `expect`, or from background at the reader's level. If you cannot point to one, the step is not a block, and the target likely rewards length: redraft it. +- Derived steps follow from the steps before them plus reader-level background, and bring in no new fact about the subject. The result follows from those steps. +- The question names a concrete case, quantity, sign, or limit that is not one of the quiz questions. + +- [ ] **Step 1: Write the failing test** `test_every_subject_has_a_checked_target`. For each subject in `ae.load("all")`, with `checks = json.loads(rederive.checks_file().read_text())`: + - if the subject has a `rederive` block: `target_problems(...) == []`, `checks[id]["verdict"] == "pass"`, and `checks[id]["hash"] == target_hash(...)`; + - otherwise: `checks[id]["verdict"] == "fail"` and `checks[id]["attempt"] == 3`. This is the "listed as having none" case. +- [ ] **Step 2: Run it to verify it fails.** + Run: `uv run pytest tests/test_rederive.py::test_every_subject_has_a_checked_target -q` + Expected: FAIL, because the checks file is missing. +- [ ] **Step 3: Draft the 30 `rederive` blocks** in `questions.yaml`, using the guidance above, and run the checker. + Run: `uv run explainer eval targets` + Expected: one verdict per subject in `evals/artifact/rederive-checks.json`. +- [ ] **Step 4: Rewrite failures, at most twice.** + - For each `fail`, rewrite the target using the checker's `reason`, then rerun the checker. The attempt count reaches 2, then 3. + - If a subject still fails at attempt 3, delete its `rederive` block and keep its `fail` entry. + - Ledger each deletion as a ruling. +- [ ] **Step 5: Run the test to verify it passes.** + Run: `uv run pytest tests/test_rederive.py -q` + Expected: all pass. +- [ ] **Step 6: Commit. This is the freeze commit.** + - `git add evals/artifact/questions.yaml evals/artifact/rederive-checks.json tests/test_rederive.py && git commit -m "Freeze the rederivation targets: 30 subjects, model-checked"` + - Ledger the commit hash as `FREEZE`. + +### Task 3: The reader + +**Files:** +- Modify: `explainer_kit/artifact_eval.py`. Extract `copy_for_reader(files: list[Path], shown: Path) -> list[Path]` from `read()`. It empties `shown`, then copies each file into it and returns the copies. `read()` calls it, and its behaviour is unchanged. +- Modify: `explainer_kit/rederive.py` +- Modify: `tests/test_rederive.py` + +**Interfaces:** +- Consumes: `ae.renderings(ws, sid)` and `ae.copy_for_reader`. +- Produces: + - `OUTSIDE_RULE: str`, the exact text in Global Constraints; + - `reader_prompt(files: list[Path], reader: str, question: str) -> str`; + - `chain_reply(text: str) -> dict`, which returns `{"steps": [{"step": str, "part": str, "idea": str}], "result": str, "gap": str}`; + - `chain_text(chain: dict) -> str`; + - `read_chain(subject: dict, cond: str, reader: str, builds: Path, out: Path, k: int, model: str | None) -> dict`, which saves to `out/"rederive"/f"pass-{k}"/"read"//f"{cond}.json"`. + +- [ ] **Step 1: Write the failing tests.** + - `test_reader_prompt_is_blind_to_steps_and_result`: + - it contains the target `q`, `OUTSIDE_RULE`, every file path, `"gap"` and `"part"`; + - it contains no step text and no result text from a real dev target. + - `test_chain_reply_tolerates_shapes`: + - valid JSON round-trips; + - `{"steps": "x"}` gives `steps == []`; + - `"no json"` gives `{"steps": [], "result": "", "gap": ""}`; + - step items that are not dicts are dropped. + - `test_chain_text_numbers_steps`: two steps give lines starting `1.` and `2.`, each with its part and idea, then a `Result:` line. + - `test_read_chain_uses_copies_and_leaves_builds_untouched`. Use a tmp builds folder with `work//a/explanation.md` and a stub `agent.ask` that captures `read=` and returns a chain. Assert: + - every `read` folder lies outside `builds`; + - the saved file exists at the path above; + - a second call makes no `agent.ask` call; + - the sha256 of every file under `builds` is unchanged. + - `test_read_chain_skips_a_build_with_no_renderings`: there is no `agent.ask` call, and the result has `"missing": True` and `steps == []`. + - The existing `tests/test_artifact_eval.py` must still pass. +- [ ] **Step 2: Run them to verify they fail.** + Run: `uv run pytest tests/test_rederive.py -q` + Expected: the new tests FAIL with `AttributeError`. +- [ ] **Step 3: Implement.** + - The reader prompt states, in order: + - the reader line; + - "Read ONLY these files" and the file list; + - the `.html` rule from `ae.reader_prompt`; + - the question; + - "work it out step by step from the explanation"; + - per step: what it establishes (`step`), the part it rests on (`part`: a heading, paragraph, figure or diagram, a page control, an earlier step number, or `background`), and that part's idea in its own words (`idea`); + - `OUTSIDE_RULE`; + - "if the explanation lacks an idea you need, say which in `gap` and stop there"; + - the JSON shape. + - Reuse `ae.READER_SYSTEM`. + - Make copies in `tempfile.mkdtemp()` and remove them afterwards. + - Retry the call once if `steps` comes back empty and `gap` is empty, as `ae.read` does. +- [ ] **Step 4: Run the tests to verify they pass.** + Run: `uv run pytest tests/test_rederive.py tests/test_artifact_eval.py -q` + Expected: all pass. +- [ ] **Step 5: Commit.** + `git add -u && git commit -m "rederive: the reader works the target out and cites the parts it rests on"` + +### Task 4: The grader and the scores + +**Files:** modify `explainer_kit/rederive.py` and `tests/test_rederive.py`. + +**Interfaces:** +- Consumes: `chain_text`, `OUTSIDE_RULE`, `ae.shown_text`, `ae.renderings`, and `ae.GRADER_SYSTEM`. +- Produces: + - `MARKS = ("supported", "unsupported", "missing", "stated")`; + - `grader_text(path: Path) -> str`; + - `grade_prompt(subject: dict, reader: str, entries: dict[str, dict]) -> str`, where `entries` maps a label to `{"text": str, "chain": str}`; + - `marks(raw: object, kinds: list[str]) -> list[str]`; + - `tally(marks: list[str], kinds: list[str], right: bool, stated: bool) -> dict[str, int]`, with keys `supported, made, counted, stated, right, results, results_stated`; + - `pass_scores(tallies: dict[str, list[dict]]) -> dict[str, dict]`. It maps a condition to `{"supported": float, "raw": float, "result": float | None}`, pooled over subjects; + - `separations(passes: list[dict[str, dict]]) -> list[tuple[str, str]]`; + - `grade_subject(subject: dict, key: dict[str, str], reader: str, builds: Path, out: Path, k: int, model: str | None) -> dict`. It saves `out/"rederive"/f"pass-{k}"/"grades"/f"{id}.json"` as `{"key": key, "grades": {cond: {"marks": [...], "result_right": bool, "result_stated": bool}}}`. + +- [ ] **Step 1: Write the failing tests.** + - `test_grader_text_includes_page_scripts`. `

Drag the slider

` gives text containing both `"Drag the slider"` and `"const loss"`. A `.md` file gives the same as `ae.shown_text`. + - `test_grade_prompt_is_blind_and_carries_the_rule`. The prompt contains: + - `OUTSIDE_RULE`; + - each expected step with its kind; + - the result; + - `"=== Explanation A ==="` and the chain text; + - the words `"any order"` and `"stated"`. + + It does not contain `"principles"` or `"skill"`, in any case. + - `test_marks_are_normalised`: + - `marks(["supported", "stated", "bogus"], ["block", "block", "derived", "derived"]) == ["supported", "supported", "missing", "missing"]`; + - `marks("x", ["block"]) == ["missing"]`; + - extra marks are dropped. + - `test_tally`: + - `tally(["supported", "unsupported", "stated", "missing"], ["block", "derived", "derived", "derived"], True, False)` gives `{"supported": 1, "made": 2, "counted": 3, "stated": 1, "right": 1, "results": 1, "results_stated": 0}`; + - with `stated=True`, it gives `right == 0`, `results == 0` and `results_stated == 1`. + - `test_pass_scores_pool_subjects`. For condition `a`, tallies `{"supported": 1, "made": 2, "counted": 2, …, "right": 1, "results": 1}` and `{"supported": 2, "made": 2, "counted": 3, …, "right": 0, "results": 1}` give supported `0.6`, raw `0.8` and result `0.5`. A condition with `results == 0` everywhere gives `result is None`. + - `test_separations_need_every_pass`. + + ```python + separations([ + {"a": {"supported": .7}, "b": {"supported": .5}, "c": {"supported": .6}}, + {"a": {"supported": .7}, "b": {"supported": .5}, "c": {"supported": .4}}, + {"a": {"supported": .6}, "b": {"supported": .6}, "c": {"supported": .3}}, + ]) == [("a", "c")] + ``` + - `test_grade_subject_maps_labels_back_and_fills_missing`. Use a stub grader reply that grades label `A` only: + - the saved grades are keyed by condition; + - the condition whose label is missing gets all `missing` marks and `result_right: False`; + - a rerun makes no call. +- [ ] **Step 2: Run them to verify they fail.** + Run: `uv run pytest tests/test_rederive.py -q` + Expected: the new tests FAIL. +- [ ] **Step 3: Implement.** + - `grader_text`: for `.html`, `ae.shown_text(path)` plus the text of each `` under a `--- page script ---` line; otherwise `ae.shown_text(path)`. Each explanation is truncated at `ae.TEXT_LIMIT`. + - The grader prompt states: + - the subject `q`, the reader, and the target `q`; + - the expected steps, numbered, with kind, and the expected result; + - `OUTSIDE_RULE`; + - "the expected steps are a checklist of ideas: credit a step when the chain establishes the same idea in any order, merged, split, or by an equivalent route"; + - the four marks as the PRD defines them, adding that `stated` applies to derived steps only and that building blocks are meant to be in the explanation; + - `result_right`, and `result_stated` (the explanation states the result outright); + - the reply shape `{"grades": {"