Repository navigation
One failed group skips the post-consolidate recompute for every group #348
Description
Problem
In study_area_run.sh, one failed group skips the post-consolidate recompute for every group in the run.
The coverage check after consolidate asks whether every run group has rows in <schema>.streams, and exit 1s on the first miss. The recompute (lnk_access(merge = TRUE), the correctness guarantee for cross-group access) comes after it. So when one group fails to model, which is a per-group soft failure everywhere else in the driver, the groups that did model lose their recompute and the compare.
The gate tests for rows, not for rows from this run. So a failed group that still holds an earlier run's rows passes it, the recompute runs, and that group's stale rows go unremarked at this point (the final RUN_INCOMPLETE notice is the only signal). The skip happens when a failed group has no rows at all, as KITR and UPET had none.
Run 20261017_004619-3f4c713b (#246): KITR and UPET failed on bugs since fixed. Had the driver been resumed with them still missing, 239 groups would have kept access computed against an incomplete barrier set, and the run would have ended with no recompute rows at all.
Proposal
- The coverage check reports the missing groups and sets
RUN_INCOMPLETE, rather than exiting. - The recompute runs over the groups that have rows from this run_uid. That is the set its result is valid for, and an earlier run's rows for a missing group are left as they are.
- The run still exits non-zero at the end, naming the missing groups, so nothing reads it as complete.
- A recompute-only entry point (
--recompute-only --run-uid=<uid>) for finishing a run after its gaps are filled by hand. That is what Provenanced province-wide default run: all 217 modelable WSGs into fresh_default #246 needed after its two failed groups were re-modelled.
Acceptance
- One missing group: the recompute runs over the rest, and the driver exits non-zero naming the group (fixture schema).
- Zero groups with rows: the recompute does not run, and the driver says why.
As built (PR #357)
- "Rows from this run_uid" means rows in
<schema>.streamsfor a group whose newest<schema>.logrow is this run_uid's and closed. That is the ruleschema_consolidate.Ruses. In a full run, the driver'sdonemarkers and consolidated buckets also count. A group rebuilt by a later run, or re-run under the same uid and failed after its pre-persist, is therefore not counted as this run's. - Coverage puts each group in one of four classes: recompute, no rows, not this run's (earlier or partial), and lost at consolidate. The classes are written to
<TS>_coverage.tsv. The compare runs over the recompute class only. --recompute-only --run-uid=<uid>also takes--wsgs=, because the log on the dispatcher cannot see a group that failed on a cloud host. It refuses a--configthat the run's log rows do not name.- Acceptance is tested on
run_finish, indata-raw/study_area_finish.sh, against a scratch schema (tests/testthat/test-study_area_finish.R). The driver writes its logs under the repo, so the library function stands in for the driver. - Follow-up: study_area_run.sh: a run that stops before its tail leaves no record of which groups it was running #358.
Activity
Code-check round 5 fixes (#246)
Merge pull request #357 from NewGraphEnvironment/348-one-failed-group…