Keep the log history through schema_consolidate (#354)

[NewGraphEnvironment]

Summary

schema_consolidate() deleted the destination's bucket from every table with a watershed_group_code column before copying. The three log tables that carry that column (log, log_input, log_recompute) were deleted with it, so each cloud-built group kept only its newest row, and every earlier build's provenance on the dispatcher was lost.

• The log tables are named once. .lnk_log_specs() (in R/lnk_log.R) gives each table's name, columns and primary key. .lnk_log_create_tables() and the consolidate script both read it.
• Consolidate deletes the bucket from persist tables only. A log table is loaded by a one-transaction psql -1 -X -f script. The script stages the rows, deletes the destination rows whose primary key matches an inbound row, and inserts. Earlier builds stay, and running a consolidate again over the same rows changes nothing.
• A source can be refused before anything is deleted. The persist tables carry no run id, so a group's newest log row is what says which build they hold. Consolidate refuses the source when a group has a log row on the dispatcher and the source's newest row is older or missing (dest_log_newer). It also refuses when the source's log cannot be read (source_log_query), or when a log table's key is not a shared column (log_pk_unshared).
• lnk_pipeline_persist() never deleted from a log table. That was the issue's second point; a test now pins it.
• The rows Provenanced province-wide default run: all 217 modelable WSGs into fresh_default #246 lost are back in fresh_default. They came from the driver's pre-consolidate dump: 5 log rows and 60 log_input rows. The groups are DUNC, KOTL, LARL, SLOC and PINE; PINE was not named in the issue. Evidence: data-raw/logs/consolidate_history_354/.
• Docs. RUNBOOK §6b, §6c and §8d are updated, along with data-raw/README.md, the host banners and CLAUDE.md. NULLS LAST is added to two diagnostic log readers, which would otherwise pick an unfinished run first once history is kept.

Related Issues

• Fixes schema_consolidate deletes the destination's log history for every group it copies #354
• Relates to NewGraphEnvironment/sred#24

Test plan

• devtools::test(), run locally: 3,147 pass, 5 fail. None of the failures comes from this branch: 3 need the :63333 tunnel, which was down, and 2 are the golden ADMS dams pin (8 → 7), already recorded in CLAUDE.md.
• New tests:
  • test-schema_consolidate.R (26 expectations). They cover the split, the DELETE list, the stage-2 command and the refusal. A live block on scratch schemas checks that history is kept, that a re-copy replaces rows, that a malformed file and a script run without -1 both leave the table unchanged, and that a repeat run changes nothing.
  • test-lnk_log.R: the specs cover every log table, and no log table's key includes the group.
  • test-lnk_pipeline_persist.R: persist never deletes from a log table.
• Each guard was broken on purpose, in copies, to check its test fails. All five mutations fail the tests: group-keyed DELETE, unsplit tables, unsplit DELETE list, -1 dropped, refusal turned off.
• After the restore, study_area_verify.sql and verify_extra.sql pass at run_uid=20261017_004619-3f4c713b, expected_n=241. lnk_log_read() still returns this run as the newest row for all 241 groups.
• Not run: a consolidate over ssh to a live cloud host. That needs a paid spin, and ssh to this machine itself is not set up. A reviewer drove schema_consolidate() end to end with a fake ssh on scratch schemas, covering five cases including both refusals.

Notes

• The plan named a per-build run_key. The code replaces rows by the full primary key instead, which deletes only rows that are being rewritten.
• The refusal does not protect a group that has persist rows but no log row on the dispatcher, i.e. one built before the log tables existed. On 2026-10-17 that is 59 of fresh's 98 groups and 0 of fresh_default's 241. RUNBOOK §8d says so.
• log_parameters_fresh, log_parameters_habitat_thresholds and log_dimensions are keyed on config_hash and still never travel between hosts. This was already noted in the RUNBOOK and is unchanged here.
• NEWS is left for /gh-pr-merge, as in this repo's earlier releases.
• Review: a plan review, then three /code-check rounds. Round 3 found a wording defect inside round 1's fix. The loop ended by enumerating every added sentence that restates the history rule or the refusal: 47 lines, 7 corrected. Records are in planning/archive/2026-10-issue-354-consolidate-log-history/.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx