You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
schema_consolidate deletes the destination's log history for every group it copies #354
If done: a group's earlier builds stay in <schema>.log on the dispatcher after a multi-host run. If never: every cloud-built group keeps only its newest log row, and what it was built from before is lost (it happened to DUNC, KOTL, LARL and SLOC in #246).
Problem
schema_consolidate() deletes the destination's rows for a source's bucket from every table with a watershed_group_code column before it copies (data-raw/schema_consolidate.R:279-286). <schema>.log and <schema>.log_recompute have that column, so a group consolidated from a cypher loses every earlier run's log row on the dispatcher.
Measured 2026-10-17 in fresh_default: DUNC, KOTL, LARL and SLOC each hold one log row, from run 20261017_004619-3f4c713b. Before that run they held the August (KOTL, LARL, SLOC) and 2026-10-12 (DUNC) rows. PINE and PARS also hold one row each; whether their earlier rows went the same way depends on which host built them in that run, which was not checked.
The persist tables are rightly replaced per group. The log tables are history, and losing them means the record of what a group was built from before is gone.
Proposal
Leave the log tables (log, log_recompute, log_input and the other log_* tables keyed by run) out of the destination DELETE, and copy their rows additively; run_id is already a unique key.
Check the same for lnk_pipeline_persist() on the dispatcher path.
lnk_pipeline_persist() was never affected. It deletes from named persist tables only; a test now pins that.
Rows are replaced by full primary key, not by run_id alone. That deletes only destination rows an inbound row rewrites. Only the three log tables with watershed_group_code travel. The config-snapshot tables still do not.
Added beyond the proposal. With history kept, a group's newest log row is what names the build its persist rows hold. Consolidate now refuses a source (nothing deleted) when a group has a log row on the dispatcher and the source's is older or missing. Groups with persist rows but no log row (59 of fresh's 98) are not protected; RUNBOOK §8d.
Activity
Merge pull request #355 from NewGraphEnvironment/354-schema-consolida…
Archive #354: correct the wrong turn as recorded