Skip to content

schema_consolidate deletes the destination's log history for every group it copies #354

Closed

Description

@NewGraphEnvironment

If done: a group's earlier builds stay in <schema>.log on the dispatcher after a multi-host run. If never: every cloud-built group keeps only its newest log row, and what it was built from before is lost (it happened to DUNC, KOTL, LARL and SLOC in #246).

Problem

schema_consolidate() deletes the destination's rows for a source's bucket from every table with a watershed_group_code column before it copies (data-raw/schema_consolidate.R:279-286). <schema>.log and <schema>.log_recompute have that column, so a group consolidated from a cypher loses every earlier run's log row on the dispatcher.

Measured 2026-10-17 in fresh_default: DUNC, KOTL, LARL and SLOC each hold one log row, from run 20261017_004619-3f4c713b. Before that run they held the August (KOTL, LARL, SLOC) and 2026-10-12 (DUNC) rows. PINE and PARS also hold one row each; whether their earlier rows went the same way depends on which host built them in that run, which was not checked.

The persist tables are rightly replaced per group. The log tables are history, and losing them means the record of what a group was built from before is gone.

Proposal

  • Leave the log tables (log, log_recompute, log_input and the other log_* tables keyed by run) out of the destination DELETE, and copy their rows additively; run_id is already a unique key.
  • Check the same for lnk_pipeline_persist() on the dispatcher path.

Outcome (PR #355)

  • PINE lost a row too. It was built on a cloud host in Provenanced province-wide default run: all 217 modelable WSGs into fresh_default #246, and its 2026-08-06 row went the same way. PARS had no earlier row. The 5 log rows and 60 log_input rows were restored from the driver's pre-consolidate dump (data-raw/logs/consolidate_history_354/).
  • lnk_pipeline_persist() was never affected. It deletes from named persist tables only; a test now pins that.
  • Rows are replaced by full primary key, not by run_id alone. That deletes only destination rows an inbound row rewrites. Only the three log tables with watershed_group_code travel. The config-snapshot tables still do not.
  • Added beyond the proposal. With history kept, a group's newest log row is what names the build its persist rows hold. Consolidate now refuses a source (nothing deleted) when a group has a log row on the dispatcher and the source's is older or missing. Groups with persist rows but no log row (59 of fresh's 98) are not protected; RUNBOOK §8d.

Activity

  1. added a commit that references this issue on Oct 18, 2026
  2. added a commit that references this issue on Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions