Skip to content

One failed group skips the post-consolidate recompute for every group #348

Closed

Description

@NewGraphEnvironment

Problem

In study_area_run.sh, one failed group skips the post-consolidate recompute for every group in the run.

The coverage check after consolidate asks whether every run group has rows in <schema>.streams, and exit 1s on the first miss. The recompute (lnk_access(merge = TRUE), the correctness guarantee for cross-group access) comes after it. So when one group fails to model, which is a per-group soft failure everywhere else in the driver, the groups that did model lose their recompute and the compare.

The gate tests for rows, not for rows from this run. So a failed group that still holds an earlier run's rows passes it, the recompute runs, and that group's stale rows go unremarked at this point (the final RUN_INCOMPLETE notice is the only signal). The skip happens when a failed group has no rows at all, as KITR and UPET had none.

Run 20261017_004619-3f4c713b (#246): KITR and UPET failed on bugs since fixed. Had the driver been resumed with them still missing, 239 groups would have kept access computed against an incomplete barrier set, and the run would have ended with no recompute rows at all.

Proposal

  • The coverage check reports the missing groups and sets RUN_INCOMPLETE, rather than exiting.
  • The recompute runs over the groups that have rows from this run_uid. That is the set its result is valid for, and an earlier run's rows for a missing group are left as they are.
  • The run still exits non-zero at the end, naming the missing groups, so nothing reads it as complete.
  • A recompute-only entry point (--recompute-only --run-uid=<uid>) for finishing a run after its gaps are filled by hand. That is what Provenanced province-wide default run: all 217 modelable WSGs into fresh_default #246 needed after its two failed groups were re-modelled.

Acceptance

  • One missing group: the recompute runs over the rest, and the driver exits non-zero naming the group (fixture schema).
  • Zero groups with rows: the recompute does not run, and the driver says why.

As built (PR #357)

  • "Rows from this run_uid" means rows in <schema>.streams for a group whose newest <schema>.log row is this run_uid's and closed. That is the rule schema_consolidate.R uses. In a full run, the driver's done markers and consolidated buckets also count. A group rebuilt by a later run, or re-run under the same uid and failed after its pre-persist, is therefore not counted as this run's.
  • Coverage puts each group in one of four classes: recompute, no rows, not this run's (earlier or partial), and lost at consolidate. The classes are written to <TS>_coverage.tsv. The compare runs over the recompute class only.
  • --recompute-only --run-uid=<uid> also takes --wsgs=, because the log on the dispatcher cannot see a group that failed on a cloud host. It refuses a --config that the run's log rows do not name.
  • Acceptance is tested on run_finish, in data-raw/study_area_finish.sh, against a scratch schema (tests/testthat/test-study_area_finish.R). The driver writes its logs under the repo, so the library function stands in for the driver.
  • Follow-up: study_area_run.sh: a run that stops before its tail leaves no record of which groups it was running #358.

Activity

  1. added a commit that references this issue on Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions