diff --git a/.devflow/features/test-harness/KNOWLEDGE.md b/.devflow/features/test-harness/KNOWLEDGE.md index e0ce86ae..9e8618b2 100644 --- a/.devflow/features/test-harness/KNOWLEDGE.md +++ b/.devflow/features/test-harness/KNOWLEDGE.md @@ -5,7 +5,7 @@ description: "Use when adding a guard, seam or golden test, changing the agent-s category: conventions directories: [tests/helpers.ts, tests/git-agent.test.ts, tests/seams, tests/goldens, tests/guards, tests/fixtures, scripts/update-golden.ts, tests/integration, tests/tracker, tests/installer, tests/evidence, tests/provider-literals.test.ts, tests/tracker-agent.test.ts, tests/commands, tests/install-bound-input.test.ts, tests/curation, tests/decisions, tests/bug-analysis] created: 2026-09-06 -updated: 2026-11-09 +updated: 2026-11-10 --- # Test Harness @@ -24,6 +24,7 @@ updated: 2026-11-09 - **KB-INV-10** Every CLI-spawning test file names one `SUBPROCESS_TIMEOUT_MS` and uses it for the `spawnSync` timeout AND every `it()` timeout, directly or through a describe-level `{ timeout }` option. - **KB-INV-11** `/code-review` and `/bug-analysis` check their fresh-base and stored-head text through the shared `collectFreshBaseDefects` / `collectStoredHeadDefects` in `tests/helpers.ts` (shaped by a `DiffRangeShape`); a fact only one command states goes in that guard's wrapper collector or its `FreshBaseParts`, never a copy of the shared checks. - **KB-INV-12** A deny-list agent outside `MCP_DENY_AGENTS` keeps every `DENY_AGENT_KEEP` tool: `collectDenylistViolations` reports a deny of one, over `KEEP_AGENTS` derived from the deny rows. +- **KB-INV-13** An allow-list agent never grants a `SHARED_DENYLIST` name: `collectDenylistViolations` reports it, because every deny-list agent denies that name. - **KB-AP-1** `scanned > 0` over a corpus is necessary, not sufficient: assert completeness against a named set (`getAllAgentNames()`, a manifest) and, for a scan of two sources, by provenance that both contributed. - **KB-AP-2** A probe that reimplements the collector inline, or names one member of a set the gate ranges over, stays green after the guard breaks: call the real collector and loop the probe over the whole set. - **KB-AP-3** Never write into the real `dist/` or `src/` from a test, nor rebuild the repo's own `dist/`: parallel workers read those paths and an unscoped build repairs staleness mid-suite; use `buildCommittedTree()` or a temp root. @@ -40,7 +41,7 @@ updated: 2026-11-09 The test harness began as the "harness first" phase of the tracker initiative and is the shared infrastructure every tracker-initiative test builds on. It lives in `tests/helpers.ts` and the guard, golden, seam, tracker, installer, evidence, integration and fixture trees under `tests/`. One principle governs it: **a green test that exercises nothing is worse than no test**, so every major test carries a non-vacuity probe proving its detection logic is live. -The harness grows by reuse, not by new mechanisms. Every guard or seam file reuses the `helpers.ts` corpus builders, follows the named-collector plus known-bad-probe shape, and registers its pinned numbers in the same manifest; `tests/evidence/` (release traceability) adopted the identical discipline. When a monolithic prompt splits further, every guard literal that moved is reclassified one at a time rather than the suite being widened. The domain content these files pin (provider semantics, reference mechanics, install shape, release/evidence semantics) belongs to the `tracker-feature`, `tracker-references` and `installer-shadowing` knowledge bases; this family owns the harness shape they all share. +The harness grows by reuse, not by new mechanisms. Every guard or seam file reuses the `helpers.ts` corpus builders, follows the named-collector plus known-bad-probe shape, and registers its pinned numbers in the same manifest; `tests/evidence/` (release traceability) adopted the identical discipline. The domain content these files pin (provider semantics, reference mechanics, install shape, release/evidence semantics) belongs to the `tracker-feature`, `tracker-references` and `installer-shadowing` knowledge bases; this family owns the harness shape they all share. Four cohesive pieces: (1) `helpers.ts` exports the shared API (agent-source resolver, corpus extractors, golden loader, fence parsers, isolated-MDS-build helpers); (2) guard tests pin source-file invariants, each with a known-bad synthetic probe; (3) golden tests assert byte equality between agent source and a committed fixture; (4) integration tests spawn real `claude` CLI sessions or full tarball installs. @@ -54,9 +55,9 @@ Four cohesive pieces: (1) `helpers.ts` exports the shared API (agent-source reso **The unit suite runs under a temp HOME by construction (D-TEST-HOME-ISOLATION).** `vitest.config.ts` `setupFiles` runs `tests/setup/isolate-env.ts` before EVERY test file. It records the original HOME in `DEVFLOW_TEST_ORIGINAL_HOME`, gives the file a fresh `mkdtemp` HOME (prefix `devflow-test-home-`, removed in `afterAll`), sets `USERPROFILE` to match, deletes `DEVFLOW_DIR` / `CLAUDE_CODE_DIR` / `CLAUDE_CONFIG_DIR`, and throws if `os.homedir()` does not then report the temp dir. devflow itself ignores `DEVFLOW_DIR` and `CLAUDE_CODE_DIR` (still stripped, for any older binary a test reaches) and honours `CLAUDE_CONFIG_DIR`, so a test isolates devflow's machine root with HOME alone and relocates the Claude directory with `CLAUDE_CONFIG_DIR`; `tests/guards/one-home.test.ts` bans both retired tokens from `src/` and `dist/` (one exemption: the settings template's `${DEVFLOW_DIR}` install-time token at its template and substitution site). -The setup file's pure half, `tests/setup/home-isolation.ts` (`REDIRECT_ENV_VARS`, `realHomes`, `assertTempHome`, `isolatedEnv`), imports nothing from `src/` or `helpers.ts`: a setup file shares the test file's module graph, so a `src/` module loaded before the redirect could freeze a real-home path into a constant. For a spawned CLI or hook, build the env with `sandboxEnv(home, extra?)` from `helpers.ts`: an allowlist (`SANDBOX_ENV_ALLOWLIST`: PATH, TMPDIR, locale, TERM, USER and similar) plus `HOME`/`USERPROFILE`, `FORCE_COLOR=0`, `NO_COLOR=1`, `CI=1`; it throws unless `home` is under `os.tmpdir()` and not a real home, and a sandbox env is never built by spreading `...process.env`. The integration config deliberately has no such setup file. Its two live-`claude` files (`ambient-activation.test.ts`, `subagent-skill-preload.test.ts`) build their own temp HOME and temp git repository with `createSpawnContext`, and install devflow into that HOME through `installDevflow`. `installDevflow` runs `init` from a separate temp directory that is not a repository, and it fails when a path that init writes (`INIT_WRITTEN_PATHS`) changes in the checkout. They never use the real `~/.claude`. The other integration file that spawns `init` (`clause-ii-file-residue.test.ts`) goes through `sandboxEnv` and echoes the resolved HOME first. `tests/setup/isolate-env.test.ts` proves the isolation, including a red probe showing the redirect variables really do move `init`'s writes when left in place. +The setup file's pure half, `tests/setup/home-isolation.ts` (`REDIRECT_ENV_VARS`, `realHomes`, `assertTempHome`, `isolatedEnv`), imports nothing from `src/` or `helpers.ts`: a setup file shares the test file's module graph, so a `src/` module loaded before the redirect could freeze a real-home path into a constant. For a spawned CLI or hook, build the env with `sandboxEnv(home, extra?)` from `helpers.ts`: an allowlist (`SANDBOX_ENV_ALLOWLIST`: PATH, TMPDIR, locale, TERM, USER and similar) plus `HOME`/`USERPROFILE`, `FORCE_COLOR=0`, `NO_COLOR=1`, `CI=1`; it throws unless `home` is under `os.tmpdir()` and not a real home, and a sandbox env is never built by spreading `...process.env`. The integration config deliberately has no such setup file. Its two live-`claude` files (`ambient-activation.test.ts`, `subagent-skill-preload.test.ts`) build their own temp HOME and temp git repository with `createSpawnContext`, and install devflow into that HOME through `installDevflow`, which runs `init` from a separate temp directory that is not a repository and fails when a path that init writes (`INIT_WRITTEN_PATHS`) changes in the checkout. The other integration file that spawns `init` (`clause-ii-file-residue.test.ts`) goes through `sandboxEnv` and echoes the resolved HOME first. `tests/setup/isolate-env.test.ts` proves the isolation, including a red probe showing the redirect variables really do move `init`'s writes when left in place. -**No literal `src/assets/agents/` paths in new test files.** The `literal-agent-paths` guard (`tests/guards/literal-agent-paths.test.ts`) scans `tests/seams/`, `tests/goldens/`, `tests/guards/`, `tests/tracker/` and `tests/installer/` (`SCAN_DIRS`) for non-comment lines containing `src/assets/agents/`. Use `resolveAgentSource(name)` for all agent content access: most agents are `.mds` generator hosts, so a read or grep over `src/assets/agents/*.md` silently misses them. New guards under `tests/guards/` reach the agent directories through `agentSourceDirs()` / `agentsDir()` / `compiledAgentsDir()` / `compiledSkillRefsDir()` / `skillsDir()` / `commandsDir()` from `src/core/assets.ts`. Documented exceptions: `tests/helpers.ts` (doc comment only), the guard file itself, and `retired-wording.test.ts`'s `removedFrom` metadata. +**No literal `src/assets/agents/` paths in new test files.** The `literal-agent-paths` guard (`tests/guards/literal-agent-paths.test.ts`) scans `tests/seams/`, `tests/goldens/`, `tests/guards/`, `tests/tracker/` and `tests/installer/` (`SCAN_DIRS`) for non-comment lines containing `src/assets/agents/`. Use `resolveAgentSource(name)` for all agent content access: most agents are `.mds` generator hosts, so a read or grep over `src/assets/agents/*.md` silently misses them. New guards under `tests/guards/` reach the agent directories through `agentSourceDirs()` / `agentsDir()` / `compiledAgentsDir()` / `compiledSkillRefsDir()` / `skillsDir()` / `commandsDir()` from `src/core/assets.ts`. The exceptions are in Recorded Exceptions. ## Standard Patterns @@ -79,7 +80,7 @@ Extracts `## Operation: ` sections from a corpus. Every call **must** name The discipline behind the mode rule: classify every guard literal one at a time, widen a corpus only where its literal provably moved, never blanket-widen a whole suite to make it green. The worked examples are in `test-harness-tracker-guards`. -Both ends of a section, start AND end, are located through the SAME memoised unfenced-heading index (`unfencedH2Index`, private to `tests/helpers.ts`: a `collectUnfencedH2` call cached by exact document text, FIFO-evicted at `UNFENCED_H2_MEMO_LIMIT` so a long-running vitest worker does not retain every corpus file it ever extracted from). Sections end at the next UNFENCED column-0 `## ` line. The boundary rule lives in the named collector `collectUnfencedH2(text)`, shared by the extractor and by `tests/tracker/reference-structure.test.ts`'s structural guard, so neither can drift from the other. Two sites remain intentionally scoped outside the extractor, for reasons unrelated to truncation: Guard 10 (AC-0.10 containment) reads each op's own section via `opSection = extractOpSection(soleCorpus, op, 'sole')`, and the seam test's `collectMissingProducers` stays body-scoped because both of its known-bad probes mutate the joined operation-reference text under test (`contractText()`), and a corpus-shaped signature would push the mutation into a fixture instead of the input under test. +Both ends of a section, start AND end, are located through the SAME memoised unfenced-heading index (`unfencedH2Index`, private to `tests/helpers.ts`: a `collectUnfencedH2` call cached by exact document text, FIFO-evicted at `UNFENCED_H2_MEMO_LIMIT` so a long-running vitest worker does not retain every corpus file it ever extracted from). Sections end at the next UNFENCED column-0 `## ` line. The boundary rule lives in the named collector `collectUnfencedH2(text)`, shared by the extractor and by `tests/tracker/reference-structure.test.ts`'s structural guard, so neither can drift from the other. Two sites remain intentionally scoped outside the extractor, for reasons unrelated to truncation: Guard 10 (AC-0.10 containment, below), and the seam test's `collectMissingProducers` stays body-scoped because both of its known-bad probes mutate the joined operation-reference text under test (`contractText()`), and a corpus-shaped signature would push the mutation into a fixture instead of the input under test. ### collectUnfencedH2 (fence-aware `## ` boundary) @@ -99,7 +100,7 @@ All three throw with a build hint when the artifact is absent: `requireDistFile` **Doctrine: build artifact -> throw (fail-loud); external binary the repo cannot produce -> `skipIf` capability gate.** A build artifact is anything `npm run build` produces; the `claude` CLI is the external-binary case. Every subprocess-CLI test file calls `requireBuiltCli()` at module scope, so an unbuilt tree is a collection error (exit 1), not a green SKIP. -**Every CLI-spawning test file names one `SUBPROCESS_TIMEOUT_MS` constant.** A real `devflow init` (or a feature toggle) spawn under load runs past vitest's 5 s default. The constant feeds the `spawnSync` `timeout` and every `it(...)` timeout, so the two ceilings cannot drift apart: the `init-*` e2e files and `init-review-publication.test.ts` use `60_000`, and `scoped-install-e2e.test.ts` uses `120_000` for its heavier three-`$HOME` install. `feature-toggle-config.test.ts` carries the same `60_000` as literals on its spawn and its `it`s. A file that launches `node` or `git` instead of the CLI follows the same rule: the two resolver suites (`resolve-base.test.ts`, `resolve-base-process.test.ts` in `tests/evidence/`) share the `SUBPROCESS_TIMEOUT_MS` that `seam.ts` exports and pass `SPAWN_BUDGET = { timeout: SUBPROCESS_TIMEOUT_MS }` as the options of every spawning `describe`, so a test added to a block inherits it. +**Every CLI-spawning test file names one `SUBPROCESS_TIMEOUT_MS` constant.** A real `devflow init` (or a feature toggle) spawn under load runs past vitest's 5 s default. The constant feeds the `spawnSync` `timeout` and every `it(...)` timeout, so the two ceilings cannot drift apart. Each file sets its own value (`scoped-install-e2e.test.ts` a larger one for its three-`$HOME` install); `feature-toggle-config.test.ts` repeats one value as literals on its spawn and its `it`s. A file that launches `node` or `git` instead of the CLI follows the same rule: the two resolver suites (`resolve-base.test.ts`, `resolve-base-process.test.ts` in `tests/evidence/`) share the `SUBPROCESS_TIMEOUT_MS` that `seam.ts` exports and pass `SPAWN_BUDGET = { timeout: SUBPROCESS_TIMEOUT_MS }` as the options of every spawning `describe`, so a test added to a block inherits it. ### walkFiles @@ -139,23 +140,22 @@ Every guard in `tests/guards/` (and the additions in `tests/tracker/`, `tests/in **1. Named collector.** The violation-detection logic is a named function called by both the main guard assertion AND the non-vacuity probe. A probe that reimplements the loop inline stays green after the real collector changes (M12b). -**2. Corpus non-vacuity.** Before asserting zero violations, assert that the corpus is non-empty AND, where a guard claims to scan two sources, assert by provenance that BOTH contributed, not just that the total crossed a floor. +**2. Corpus non-vacuity.** Before asserting zero violations, assert the corpus is non-empty AND, for a two-source scan, that BOTH sources contributed (provenance, not a total floor). -**3. Known-bad probe (mechanic 2 / H10).** Build a synthetic corpus entry or temp root that contains a real violation and confirm the collector flags it. This proves the detection logic is live without touching any committed source file. +**3. Known-bad probe (mechanic 2 / H10).** Build a synthetic corpus entry or temp root holding a real violation and confirm the collector flags it, without touching committed source. ### De-vacuumed guard anti-pattern (AC-0.10 lesson) -The AC-0.10 containment guard had a combined predicate (` || `) with one floor. On unmodified `main`, three pre-existing `` ops satisfied the floor, so the guard passed without ever touching an `` op. The fix splits it into two independent assertions with **named matching op sets**: issue-body (predicate `` only, named set `{setup-task, fetch-issue, fetch-issues-batch}`) and external-thread (predicate `` only, named set `{fetch-review-threads, post-resolution-summary}`), registered as `containment-issue-body-floor` and `containment-external-thread-floor`. Rule: when a guard predicate is a logical OR, you cannot tell which branch is carrying the floor. - +The AC-0.10 containment guard had a combined predicate (` || `) with one floor. On unmodified `main`, three pre-existing `` ops satisfied the floor, so the guard passed without ever touching an `` op. The fix splits it into two independent assertions with **named matching op sets**: issue-body (predicate `` only, named set `{setup-task, fetch-issue, fetch-issues-batch}`) and external-thread (predicate `` only, named set `{fetch-review-threads, post-resolution-summary}`), registered as `containment-issue-body-floor` and `containment-external-thread-floor`. ### Guard 10: scope a de-vacuumed predicate to the op's own section -Guard 10 reads each operation in `GIT_OPERATIONS` through `opSection = extractOpSection(soleCorpus, op, 'sole')`: the op's own section in its one reference, cut at the next UNFENCED `## `, never a region that runs "to the next `## Operation:` anchor, or EOF", which sweeps in whatever trails the section (a shared trailer). Both external-thread ops are read the same way. **Lesson**: a named-set assertion proves an op is REACHABLE from the marker; it does not prove the marker lives in the op's OWN section unless the extraction is scoped to exactly that section. +Guard 10 reads each operation in `GIT_OPERATIONS` through `opSection = extractOpSection(soleCorpus, op, 'sole')`: the op's own section in its one reference, cut at the next UNFENCED `## `, never a region that runs "to the next `## Operation:` anchor, or EOF", which sweeps in whatever trails the section (a shared trailer). **Lesson**: a named-set assertion proves an op is REACHABLE from the marker; it does not prove the marker lives in the op's OWN section unless the extraction is scoped to exactly that section. ## Shared Denylist and Absence Guards **`retired-wording.test.ts`** is one shared grep guard with a denylist that grows over time, never a new grep and never emptied. Each `RETIRED_LITERALS` entry carries an optional `scope: readonly string[]` (corpus path-prefixes the literal is retired FROM; absent means the whole corpus). Rows include `gh issue` (scope `dist/commands/`), `sleep 60`, `