diff --git a/CLAUDE.md b/CLAUDE.md index cb269871..da8686a1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,7 +18,7 @@ Experimental package — breaking all the time and loving the learning curve. St **Facts not worth re-deriving:** - **One run_uid, more than one link SHA.** In `log`, 239 groups stamp `ee21eb13` and KITR and UPET `a17e9177`, the fix for the two bugs that failed them, re-run on m1 under the same run_uid. In `log_recompute`, all 241 stamp `a17e9177`, so the shipped `streams_access` / `streams_mapping_code` were written at it. UPET was recomputed again after `91a968ab` (the mapping-code fix, below), from a clean tree at `eb93f343`; one stray row between those is `link_dirty` TRUE (an uncommitted review file), and the clean re-run follows it. - **The driver read completeness from log text, and it missed 18 finished groups that way** (#347): an R warning put the `done` marker mid-line, the driver consolidated only what it counted, and its burn would have destroyed the rest; the operator held it with `kill -STOP` and consolidated by hand. Since #347 the count is each host's database, a cypher is held from dispatch until no `streams` rows are left on it after consolidate, and a kept host is listed in `_kept_hosts.tsv` (RUNBOOK §8d, "Which host may be burned"). -- **A failed group with no earlier rows skips the recompute for every group** (#348): the coverage gate exits before it. A failed group that still has an earlier run's rows passes the gate, and its stale rows go unnoticed there. +- **A failed group no longer skips the recompute for every group** (#348). Until then the coverage gate exited before the recompute, and a failed group holding an earlier run's rows passed it unnoticed. Coverage now reads which groups have rows from a build this run finished, the recompute and compare run over those, and the rest are named with a non-zero exit. `--recompute-only --run-uid=` finishes a run once they are re-modelled under that uid (RUNBOOK §8d, "A missing group costs only itself"). - **The recompute takes 154 min at 241 groups and `-j4`** (median 2.0 min per group, longest 16 min), not the ~1 h the plan assumed. - **A present species with no barrier in a group is accessible** (`access_` 1 or 2) **and has `has_barriers__dnstr` FALSE**, so its mapping code is built. Zero barrier rows mean NA ("no data") only when nothing says the species is present. Before `91a968ab` the flag stayed NA and UPET RB shipped with `mapping_code_rb` "" on every segment; UPET RB is the only such case in the 241. - **`fresh_default` is 31 GB** (14 GB tables, 17 GB indexes; 11.39 M segments), up from 7.3 GB. Summing `pg_total_relation_size()` over every `pg_class` row counts each index twice (48 GB). The fwapg volume is 600 GiB with 253 G free after the run. diff --git a/RUNBOOK.md b/RUNBOOK.md index 5f8cf83d..55e16afd 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -1209,11 +1209,85 @@ that DELETE, so a group's log history survives it; link#354.) So: - before consolidate, every host must account for its whole bucket, read from its **database**: `.log` rows for the run_uid with `date_end` set, plus `SKIP` markers from the log, which write no row (link#347); -- after consolidate, every run WSG must have rows in `.streams`. +- after consolidate, every run WSG must have rows in `.streams` from a + build **this run finished**. The second is detection rather than prevention, and it is the one that would have caught link#246 on day one regardless of cause. +### A missing group costs only itself (link#348) + +The coverage check used to `exit 1` on the first run group with no rows, +before the recompute. One failed group then left every other group's access +computed against an incomplete barrier set, with no recompute rows at all. +It also asked only whether rows existed, so a failed group holding an earlier +run's rows passed and was recomputed as this run's. + +Now `run_coverage` (`data-raw/study_area_finish.sh`) splits the run's groups +(every bucket, less SKIPs) into four classes, read on the dispatcher after +consolidate: + +| class | rows in `streams` | finished this run | what happens | +|---|---|---|---| +| recompute | yes | yes | recomputed and compared | +| no rows | no | no | named at the end | +| earlier or partial | yes | no | named; its rows are left as they are | +| lost at consolidate | no | yes | named | + +"Finished this run" means the group's **newest** `.log` row (max +`date_start`, the rule `schema_consolidate.R` uses for which build a group's +persist rows hold) is the run_uid's and has `date_end` set. Consolidate +copies a group's log rows home (link#354), and the driver adds the +dispatcher's `done` markers and each cypher's consolidated bucket. A failed +build leaves its row open (`.lnk_log_run_fail` sets only `notes`), and its +pre-persist can leave `streams` rows, so rows alone do not make a group +"this run's". "Newest" matters for a `--recompute-only` days later: a group a +later run rebuilt, or a hand re-run under the same uid that failed after its +pre-persist, has a closed row for the uid that no longer names its rows. The +classes are written to `_coverage.tsv`, beside the compare CSV, which +covers the first class only. The recompute and the compare run over the first class only. If +that class is empty, neither runs and the report says why. Any other class +makes the run exit non-zero, and the report names the groups and the recovery: + +```bash +LNK_RUN_UID= [LNK_RUN_LABEL=