From b9b66125ec47facd136d5b8776c69e6745e2455c Mon Sep 17 00:00:00 2001 From: almac2022 Date: Sat, 17 Oct 2026 20:36:03 -0700 Subject: [PATCH 01/11] Initialize PWF baseline for #354 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx --- planning/active/findings.md | 47 +++++++++++++++++++++++++++++++++++ planning/active/progress.md | 8 ++++++ planning/active/task_plan.md | 48 ++++++++++++++++++++++++++++++++++++ 3 files changed, 103 insertions(+) create mode 100644 planning/active/findings.md create mode 100644 planning/active/progress.md create mode 100644 planning/active/task_plan.md diff --git a/planning/active/findings.md b/planning/active/findings.md new file mode 100644 index 00000000..331c8a1d --- /dev/null +++ b/planning/active/findings.md @@ -0,0 +1,47 @@ +# Findings — schema_consolidate deletes the destination's log history for every group it copies (#354) + +## Issue context + +**If done:** a group's earlier builds stay in `.log` on the dispatcher after a multi-host run. **If never:** every cloud-built group keeps only its newest log row, and what it was built from before is lost (it happened to DUNC, KOTL, LARL and SLOC in #246). + +## Problem + +`schema_consolidate()` deletes the destination's rows for a source's bucket from every table with a `watershed_group_code` column before it copies (`data-raw/schema_consolidate.R:279-286`). `.log` and `.log_recompute` have that column, so a group consolidated from a cypher loses every earlier run's log row on the dispatcher. + +Measured 2026-10-17 in `fresh_default`: DUNC, KOTL, LARL and SLOC each hold one `log` row, from run `20261017_004619-3f4c713b`. Before that run they held the August (KOTL, LARL, SLOC) and 2026-10-12 (DUNC) rows. PINE and PARS also hold one row each; whether their earlier rows went the same way depends on which host built them in that run, which was not checked. + +The persist tables are rightly replaced per group. The log tables are history, and losing them means the record of what a group was built from before is gone. + +## Proposal + +- Leave the log tables (`log`, `log_recompute`, `log_input` and the other `log_*` tables keyed by run) out of the destination DELETE, and copy their rows additively; `run_id` is already a unique key. +- Check the same for `lnk_pipeline_persist()` on the dispatcher path. + + +## Exploration (plan mode, 2026-10-17) + +`schema_consolidate()` deletes the destination's bucket rows from every BASE TABLE with a +`watershed_group_code` column before it COPYs (`data-raw/schema_consolidate.R:279-286`). +`log`, `log_recompute` and `log_input` carry that column (so consolidate discovers them), so +each cloud-built group keeps only the source's rows and loses the dispatcher's history. + +Measured during exploration (2026-10-17, local fwapg): +- `fresh_default.log`: all 241 groups hold exactly 1 row. `fresh.log` holds 2–4 rows for 20 + groups, so the readers (`lnk_log_read()` DISTINCT ON newest, `study_area_verify.sql` keyed + on `run_uid`) already cope with history. +- PKs: persist tables all include `watershed_group_code`; the log tables do not + (`log` run_id, `log_input` (run_id, table_name), `log_recompute` recompute_id). +- `/tmp/fresh_default_pre_consolidate_202610162242.dump` still holds the lost rows: `log` for + DUNC (2026-10-12), KOTL/LARL/SLOC (2026-08-27), **PINE (2026-08-06)** — so PINE lost one too + (built on cypher-job2); PARS had none. 60 `log_input` rows, 0 `log_recompute`. +- `lnk_pipeline_persist()` (dispatcher path) DELETEs only from named persist tables + (streams, habitat, barriers, barrier_overrides, streams_access, streams_mapping_code), never + a log table. Issue's second bullet: no change, pin it with a test. +- RUNBOOK §6c "Known exposure" (a second consolidate wipes `log_recompute`) is this bug. + +Operator decision (plan gate): restore the lost rows from the dump. + +## Errors Encountered + +| Error | Resolution | +|-------|------------| diff --git a/planning/active/progress.md b/planning/active/progress.md new file mode 100644 index 00000000..e1efc69e --- /dev/null +++ b/planning/active/progress.md @@ -0,0 +1,8 @@ +# Progress — schema_consolidate deletes the destination's log history for every group it copies (#354) + +## Session 2026-10-17 + +- Plan-mode exploration — phases approved by user; operator chose to restore the rows #246 lost +- Created branch `354-schema-consolidate-deletes-the-destinati` off main +- Scaffolded PWF baseline from issue #354 with approved phases +- Next: start Phase 1 diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md new file mode 100644 index 00000000..978bba22 --- /dev/null +++ b/planning/active/task_plan.md @@ -0,0 +1,48 @@ +# Task: schema_consolidate deletes the destination's log history for every group it copies (#354) + +`schema_consolidate()` deletes the destination's rows for a source's bucket from every table with a `watershed_group_code` column before it copies (`data-raw/schema_consolidate.R:279-286`). `.log` and `.log_recompute` have that column, so a group consolidated from a cypher loses every earlier run's log row on the dispatcher. + +**Approach (approved):** name the history tables once in `.lnk_log_specs()` (with a `run_key`); consolidate DELETEs the bucket from persist tables only and replaces a history table's rows by run key, atomically, through a staged `psql -1 -f` script. Restore the rows #246 lost from the pre-consolidate dump (operator, plan gate). + +## Phase 1: Tests first +- [ ] `test-lnk_log.R`: `.lnk_log_specs()` names every table `.lnk_log_create_tables()` + builds, and each `run_key` is a column of its table and the PK or its leading column +- [ ] New `tests/testthat/test-schema_consolidate.R` (sources the script, + `skip_if_not(file.exists())` as `test-lnk_fanout_judge.R` does): + `consolidate_history_sql()` emits the staged replace keyed on `run_key`, no + `watershed_group_code` predicate +- [ ] Same file, live (`skip_if_no_db()`): on a scratch schema, an earlier run's row survives + a re-copy of the same group, and a re-copied same `run_id` replaces rather than duplicates + or collides; restore the old DELETE and watch it go red +- [ ] `test-lnk_pipeline_persist.R`: captured SQL never DELETEs from a `log*` table + +## Phase 2: Name the history tables once +- [ ] `.lnk_log_specs()` with `run_key`; `.lnk_log_create_tables()` reads it (DDL unchanged — + existing DDL tests pass as is) + +## Phase 3: Consolidate keeps history +- [ ] Split persist / history tables; DELETE persist only +- [ ] `consolidate_history_sql()` + stage 2 via `psql -1 -f` for history tables +- [ ] Update the comments that describe the bucket DELETE (script header, step 3.5, + `study_area_run.sh:1389-1392` line refs) + +## Phase 4: Docs +- [ ] RUNBOOK §6c: replace "Known exposure" with the history rule; §6c note at ~507 and §8d + recovery passage (~1268) say the log history survives a re-consolidate +- [ ] NEWS entry (version bump left to `/gh-pr-merge`) + +## Phase 5: Restore the lost rows (operator: yes) +- [ ] Extract `log` + `log_input` rows for DUNC, KOTL, LARL, SLOC, PINE from the 22:42 dump + into a scratch schema (`pg_restore --data-only -t` into a copy), INSERT … ON CONFLICT DO + NOTHING into `fresh_default`; counts before/after recorded in + `data-raw/logs/consolidate_history_354/` +- [ ] Re-run `study_area_verify.sql -v run_uid=20261017_004619-3f4c713b -v expected_n=241` + and `verify_extra.sql`; both pass; `lnk_log_read()` still returns this run's row per group +- [ ] CLAUDE.md #246 status: the four (five, with PINE) groups' earlier log rows restored + +## Validation + +- [ ] Tests pass +- [ ] `/code-check` clean (each commit, or once over the branch with `/code-check branch`) +- [ ] PWF checkboxes match landed work +- [ ] `/planning-archive` on completion From 0a1f85340e9e839335e8b5fd44c1a5202a076d08 Mon Sep 17 00:00:00 2001 From: almac2022 Date: Sat, 17 Oct 2026 20:40:34 -0700 Subject: [PATCH 02/11] Name the log tables once in .lnk_log_specs() (#354) .lnk_log_create_tables() reads it; schema_consolidate.R will too. Tests: every built table is named, no log table's primary key includes watershed_group_code, and lnk_pipeline_persist() never deletes from one. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx --- R/lnk_log.R | 40 ++++++++++++++++------ tests/testthat/test-lnk_log.R | 22 ++++++++++++ tests/testthat/test-lnk_pipeline_persist.R | 29 ++++++++++++++++ 3 files changed, 81 insertions(+), 10 deletions(-) diff --git a/R/lnk_log.R b/R/lnk_log.R index 5a0544ef..6534cfee 100644 --- a/R/lnk_log.R +++ b/R/lnk_log.R @@ -492,19 +492,20 @@ cols_log_input <- c( } -#' Create the provenance tables. Idempotent. +#' The provenance tables: name, columns and primary key. #' -#' Called from [lnk_persist_init()] so a standalone init produces a complete -#' schema, and again from the run-start path so a run never fails for want of -#' a table. +#' These tables are history, not per-group state, and no primary key here +#' includes `watershed_group_code`. `data-raw/schema_consolidate.R` replaces +#' their rows by primary key and never deletes them by watershed group, so a +#' group copied home from another host keeps the rows of its earlier builds +#' (link#354). #' +#' Read by [.lnk_log_create_tables()] and by `schema_consolidate.R`, so the set +#' of history tables is named here and nowhere else. +#' +#' @return A list of specs named by table. #' @noRd -.lnk_log_create_tables <- function(conn, schema) { - # The log is opened BEFORE lnk_persist_init runs (the open row must predate - # any write, so wsg_upstream reflects the state the run started from), so on - # a brand-new persist schema nothing has created it yet. - .lnk_db_execute(conn, sprintf("CREATE SCHEMA IF NOT EXISTS %s", schema)) - +.lnk_log_specs <- function() { specs <- list( list(table = "log", cols = cols_log, pk = "run_id"), list(table = "log_recompute", cols = cols_log_recompute, @@ -519,6 +520,25 @@ cols_log_input <- c( list(table = "log_dimensions", cols = .lnk_cols_log_dimensions(), pk = c("config_hash", "species")) ) + names(specs) <- vapply(specs, `[[`, character(1), "table") + specs +} + + +#' Create the provenance tables. Idempotent. +#' +#' Called from [lnk_persist_init()] so a standalone init produces a complete +#' schema, and again from the run-start path so a run never fails for want of +#' a table. +#' +#' @noRd +.lnk_log_create_tables <- function(conn, schema) { + # The log is opened BEFORE lnk_persist_init runs (the open row must predate + # any write, so wsg_upstream reflects the state the run started from), so on + # a brand-new persist schema nothing has created it yet. + .lnk_db_execute(conn, sprintf("CREATE SCHEMA IF NOT EXISTS %s", schema)) + + specs <- .lnk_log_specs() for (s in specs) { .lnk_db_execute(conn, sprintf( diff --git a/tests/testthat/test-lnk_log.R b/tests/testthat/test-lnk_log.R index cf2968fc..54f78b5e 100644 --- a/tests/testthat/test-lnk_log.R +++ b/tests/testthat/test-lnk_log.R @@ -1239,3 +1239,25 @@ test_that(".lnk_config_hash does not depend on the host's collation locale", { skip_if(is.na(h_en), "en_US.UTF-8 collation not available on this host") expect_identical(h_c, h_en) }) + +test_that(".lnk_log_specs names every table .lnk_log_create_tables builds", { + # One list, read by the DDL and by schema_consolidate.R (link#354): a log + # table missing from it would be consolidated as a persist table, and its + # history deleted by group. + sql <- capture_ddl(.lnk_log_create_tables(NULL, "s")) + built <- sub("^CREATE TABLE IF NOT EXISTS s\\.(\\w+) .*", "\\1", + grep("^CREATE TABLE IF NOT EXISTS", sql, value = TRUE)) + expect_setequal(names(.lnk_log_specs()), built) + expect_identical(names(.lnk_log_specs()), + unname(vapply(.lnk_log_specs(), `[[`, character(1), "table"))) +}) + +test_that("no log table's primary key includes watershed_group_code", { + # What lets schema_consolidate.R replace log rows by key and keep a group's + # earlier builds: one key value is one build, not one group (link#354). + for (s in .lnk_log_specs()) { + expect_true(length(s$pk) > 0L && all(s$pk %in% names(s$cols)), + info = s$table) + expect_false("watershed_group_code" %in% s$pk, info = s$table) + } +}) diff --git a/tests/testthat/test-lnk_pipeline_persist.R b/tests/testthat/test-lnk_pipeline_persist.R index 11003d4b..dbd312f8 100644 --- a/tests/testthat/test-lnk_pipeline_persist.R +++ b/tests/testthat/test-lnk_pipeline_persist.R @@ -208,3 +208,32 @@ test_that("lnk_pipeline_persist skips barriers branch when staging table absent" expect_no_match(joined, "DELETE FROM fresh\\.barriers") expect_no_match(joined, "INSERT INTO fresh\\.barriers") }) + +test_that("lnk_pipeline_persist never deletes from a log table (link#354)", { + # The log tables are history: a group's earlier builds must survive its next + # build. Every staging table reported present, so every DELETE branch runs. + captured <- character(0) + local_mocked_bindings( + .lnk_db_execute = function(conn, sql) { + captured <<- c(captured, sql); invisible(NULL) + } + ) + with_mocked_bindings( + dbGetQuery = function(...) data.frame(x = 1L), + dbQuoteString = function(...) DBI::SQL("'x'"), + .package = "DBI", + { + cfg <- lnk_config("bcfishpass") + lnk_pipeline_persist( + structure(list(), class = "DBIConnection"), + aoi = "PARS", cfg = cfg, species = c("BT", "SK")) + } + ) + deletes <- grep("^\\s*DELETE FROM", captured, value = TRUE) + # streams, one per species (2), barriers, barrier_overrides, + # streams_access, streams_mapping_code. + expect_length(deletes, 7L) + targets <- sub("^\\s*DELETE FROM (\\S+).*", "\\1", deletes) + expect_false(any(sub("^[^.]+\\.", "", targets) %in% + names(.lnk_log_specs()))) +}) From 79a247e0adf7fe347ddeee37b89a8d2703bdf391 Mon Sep 17 00:00:00 2001 From: almac2022 Date: Sat, 17 Oct 2026 20:40:34 -0700 Subject: [PATCH 03/11] Keep the log history through schema_consolidate (#354) The bucket DELETE now runs over persist tables only. A log table's rows go through a one-transaction staged replace by primary key, so a group copied home from another host keeps the rows of its earlier builds, and a re-consolidate of the same rows changes nothing. A source whose log primary key is not shared is refused before anything is deleted. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx --- data-raw/schema_consolidate.R | 108 +++++++++++++++-- data-raw/study_area_run.sh | 7 +- planning/active/findings.md | 17 +++ planning/active/task_plan.md | 23 ++-- tests/testthat/test-schema_consolidate.R | 147 +++++++++++++++++++++++ 5 files changed, 278 insertions(+), 24 deletions(-) create mode 100644 tests/testthat/test-schema_consolidate.R diff --git a/data-raw/schema_consolidate.R b/data-raw/schema_consolidate.R index e6374bdb..5dbf6f37 100644 --- a/data-raw/schema_consolidate.R +++ b/data-raw/schema_consolidate.R @@ -31,9 +31,68 @@ # COPY-streaming with `WHERE wsg IN (bucket)` at source eliminates the # over-fetch class entirely. # +# The log tables are the exception to "replace the bucket" (link#354). They +# carry `watershed_group_code`, so they are discovered and copied like the +# rest, but they are history: a group's earlier builds must survive a later +# build copied home from another host. Their rows are replaced by primary key +# (`.lnk_log_specs()`), never deleted by group. +# # Cross-refs: rtj#94 (general orchestrator); link#112 (first usage); # link#180 (additive Step 0 unblocked by this source-filter fix). +#' Split discovered tables into persist tables and log (history) tables. +#' +#' Persist tables hold one group's current build and are replaced per group. +#' Log tables are those `.lnk_log_specs()` names (link#354). +#' +#' @param wgc_tables Character. Tables discovered on both sides. +#' @param specs The log table specs, named by table. +#' @return `list(persist = , history = )`, each in `wgc_tables` order. +consolidate_tables_split <- function(wgc_tables, + specs = link:::.lnk_log_specs()) { + is_history <- wgc_tables %in% names(specs) + list(persist = wgc_tables[!is_history], history = wgc_tables[is_history]) +} + +#' psql script that adds a log table's inbound rows without touching history. +#' +#' Run with `psql -1 -v ON_ERROR_STOP=1 -f`, so it is one transaction: the +#' rows are staged in a temp table, the destination rows that share a primary +#' key with an inbound row are deleted (the same build copied again, e.g. a +#' consolidate re-run, or a log row whose `date_end` was set later), and the +#' inbound rows are inserted. Every other row stays, whatever its group +#' (link#354). +#' +#' @param schema,table Destination table. +#' @param cols Character. The shared columns, in the order of `path`'s fields. +#' @param pk Character. The table's primary key columns; all must be in `cols`. +#' @param path Path of the COPY text-format file to load. +#' @return Character vector, one line per statement. +consolidate_history_sql <- function(schema, table, cols, pk, path) { + missing_pk <- setdiff(pk, cols) + if (length(missing_pk) > 0L) { + stop("consolidate_history_sql: primary key column(s) ", + paste(missing_pk, collapse = ", "), " of ", schema, ".", table, + " are not shared by source and destination", call. = FALSE) + } + if (grepl("'", path, fixed = TRUE)) { + stop("consolidate_history_sql: path contains a single quote: ", path, + call. = FALSE) + } + col_list <- paste(cols, collapse = ", ") + key_d <- paste0("d.", pk, collapse = ", ") + key_s <- paste0("s.", pk, collapse = ", ") + c( + sprintf("CREATE TEMP TABLE consolidate_in ON COMMIT DROP AS SELECT %s FROM %s.%s WITH NO DATA;", + col_list, schema, table), + sprintf("\\copy consolidate_in (%s) FROM '%s'", col_list, path), + sprintf("DELETE FROM %s.%s d USING consolidate_in s WHERE (%s) = (%s);", + schema, table, key_d, key_s), + sprintf("INSERT INTO %s.%s (%s) SELECT %s FROM consolidate_in;", + schema, table, col_list, col_list) + ) +} + #' Consolidate a Postgres schema from N remote hosts onto local fwapg. #' #' @param schema Character. Schema to consolidate (must already exist on @@ -276,11 +335,27 @@ schema_consolidate <- function(schema, } # 3.5. Bucket-aware destination cleanup. DELETE the bucket's WSGs - # from every destination table BEFORE the COPY-INSERTs so PK - # constraints don't fire on the inbound rows. + # from every destination PERSIST table BEFORE the COPY-INSERTs so PK + # constraints don't fire on the inbound rows. Log tables are left out: + # their rows are history, replaced by primary key in step 4 (link#354). + split_tables <- consolidate_tables_split(wgc_tables) + log_specs <- link:::.lnk_log_specs() + # Checked before anything is deleted: a log table whose key is not shared + # cannot be replaced by key, and failing at its COPY would leave this + # bucket's persist rows already deleted here. + pk_unshared <- Filter(function(t) !all(log_specs[[t]]$pk %in% shared_cols[[t]]), + split_tables$history) + if (length(pk_unshared) > 0L) { + log(src$host, " -> ERROR: primary key not shared on log table(s) ", + paste(pk_unshared, collapse = ", "), " — nothing deleted or copied") + results[[src$host]] <- list(ok = FALSE, stage = "log_pk_unshared", + rc = -1L, table = pk_unshared[[1]]) + next + } log(src$host, " -> DELETE bucket (", length(src$bucket), - " WSGs) from ", length(wgc_tables), " tables") - for (t in wgc_tables) { + " WSGs) from ", length(split_tables$persist), " tables; ", + length(split_tables$history), " log tables keep their history") + for (t in split_tables$persist) { DBI::dbExecute(dest_conn, sprintf( "DELETE FROM %s.%s WHERE watershed_group_code IN (%s)", schema, t, wsg_list_sql)) @@ -358,13 +433,26 @@ schema_consolidate <- function(schema, } # Stage 2: local temp file -> destination COPY FROM STDIN. Same # explicit shared-column list as the source SELECT so the transfer - # is by-name (shape-tolerant), not positional. - dest_sql <- sprintf("COPY %s.%s (%s) FROM STDIN", schema, t, col_list) - stage2 <- sprintf( - "PGHOST=localhost PGPORT=5432 PGDATABASE=fwapg PGUSER=postgres PGPASSWORD=postgres psql -v ON_ERROR_STOP=1 -c \"%s\" < %s", - dest_sql, shQuote(tmpf)) + # is by-name (shape-tolerant), not positional. A log table goes + # through a one-transaction staged replace by primary key instead, so + # its earlier rows for the bucket survive (link#354). + if (t %in% split_tables$history) { + sqlf <- tempfile(pattern = sprintf("consolidate_%s_", t), + fileext = ".sql") + writeLines(consolidate_history_sql(schema, t, shared_cols[[t]], + log_specs[[t]]$pk, tmpf), sqlf) + stage2 <- sprintf( + "PGHOST=localhost PGPORT=5432 PGDATABASE=fwapg PGUSER=postgres PGPASSWORD=postgres psql -1 -v ON_ERROR_STOP=1 -f %s", + shQuote(sqlf)) + } else { + sqlf <- NULL + dest_sql <- sprintf("COPY %s.%s (%s) FROM STDIN", schema, t, col_list) + stage2 <- sprintf( + "PGHOST=localhost PGPORT=5432 PGDATABASE=fwapg PGUSER=postgres PGPASSWORD=postgres psql -v ON_ERROR_STOP=1 -c \"%s\" < %s", + dest_sql, shQuote(tmpf)) + } rc_dest <- system(stage2) - unlink(tmpf) + unlink(c(tmpf, sqlf)) if (rc_dest != 0L) { log(src$host, " -> WARN: dest_copy failed on table '", t, "' (continuing)") diff --git a/data-raw/study_area_run.sh b/data-raw/study_area_run.sh index 46cccfa6..5a751e5f 100755 --- a/data-raw/study_area_run.sh +++ b/data-raw/study_area_run.sh @@ -1386,10 +1386,11 @@ COMPARE_CSV="$LOG_DIR/${TS}_compare.csv" # leak spend, and BEFORE the recompute so a partial result is never painted # as complete. # -# Necessary because schema_consolidate DELETEs the destination bucket -# (schema_consolidate.R:272-276) and then COPYs (:313-316). A source that +# Necessary because schema_consolidate DELETEs the destination bucket from +# every persist table (its step 3.5) and then COPYs (step 4). A source that # produced nothing therefore removes the destination's prior rows for those -# WSGs and still returns ok = TRUE. +# WSGs and still returns ok = TRUE. The log tables keep their rows (#354), so +# this check reads streams, never the log. echo "=== verify WSG coverage in $SCHEMA ===" if MISSING=$(PGPASSWORD=postgres psql -h localhost -p 5432 -U postgres -d fwapg -t -A -c " WITH expect(w) AS (SELECT unnest(string_to_array('$ALL_WSGS', ','))) diff --git a/planning/active/findings.md b/planning/active/findings.md index 331c8a1d..fd70f6fd 100644 --- a/planning/active/findings.md +++ b/planning/active/findings.md @@ -41,7 +41,24 @@ Measured during exploration (2026-10-17, local fwapg): Operator decision (plan gate): restore the lost rows from the dump. +## Primary key, not a run key (Phase 2, 2026-10-17) + +The plan named a `run_key` (run_id / recompute_id) to replace a build's rows wholly. Replacing +by the full primary key instead deletes only destination rows that an inbound row rewrites, so +a destination `log_input` row the source no longer sends (e.g. one with a NULL group code, +which the bucket SELECT cannot see) is never lost. Measured: 0 NULL-group `log_input` rows in +`fresh_default` (of 2,892) and `fresh` (of 715), so the two rules agree on today's data; the +PK rule needs no new field, since `.lnk_log_specs()` already carries `pk`. + +## Restore-the-bug (Phase 3) + +Run on copies of the test and script under the scratchpad: +- DELETE by group (`d.watershed_group_code = s.watershed_group_code`): 2 unit + 3 live + expectations fail. +- Every table treated as persist (`is_history` all FALSE): 3 split expectations fail. + ## Errors Encountered | Error | Resolution | |-------|------------| +| `Already disconnected` warning in the live test | `skip_if_no_db()` defers its own disconnect; dropped the test's `on.exit` | diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md index 978bba22..d29279ce 100644 --- a/planning/active/task_plan.md +++ b/planning/active/task_plan.md @@ -2,28 +2,29 @@ `schema_consolidate()` deletes the destination's rows for a source's bucket from every table with a `watershed_group_code` column before it copies (`data-raw/schema_consolidate.R:279-286`). `.log` and `.log_recompute` have that column, so a group consolidated from a cypher loses every earlier run's log row on the dispatcher. -**Approach (approved):** name the history tables once in `.lnk_log_specs()` (with a `run_key`); consolidate DELETEs the bucket from persist tables only and replaces a history table's rows by run key, atomically, through a staged `psql -1 -f` script. Restore the rows #246 lost from the pre-consolidate dump (operator, plan gate). +**Approach (approved):** name the history tables once in `.lnk_log_specs()`; consolidate DELETEs the bucket from persist tables only and replaces a history table's rows by primary key, atomically, through a staged `psql -1 -f` script. Restore the rows #246 lost from the pre-consolidate dump (operator, plan gate). ## Phase 1: Tests first -- [ ] `test-lnk_log.R`: `.lnk_log_specs()` names every table `.lnk_log_create_tables()` - builds, and each `run_key` is a column of its table and the PK or its leading column -- [ ] New `tests/testthat/test-schema_consolidate.R` (sources the script, +- [x] `test-lnk_log.R`: `.lnk_log_specs()` names every table `.lnk_log_create_tables()` + builds, and no log table's PK includes `watershed_group_code` (corrected from `run_key`: + replace by full primary key, which deletes only rows being rewritten; see findings) +- [x] New `tests/testthat/test-schema_consolidate.R` (sources the script, `skip_if_not(file.exists())` as `test-lnk_fanout_judge.R` does): - `consolidate_history_sql()` emits the staged replace keyed on `run_key`, no + `consolidate_history_sql()` emits the staged replace keyed on the primary key, no `watershed_group_code` predicate -- [ ] Same file, live (`skip_if_no_db()`): on a scratch schema, an earlier run's row survives +- [x] Same file, live (`skip_if_no_db()`): on a scratch schema, an earlier run's row survives a re-copy of the same group, and a re-copied same `run_id` replaces rather than duplicates or collides; restore the old DELETE and watch it go red -- [ ] `test-lnk_pipeline_persist.R`: captured SQL never DELETEs from a `log*` table +- [x] `test-lnk_pipeline_persist.R`: captured SQL never DELETEs from a `log*` table ## Phase 2: Name the history tables once -- [ ] `.lnk_log_specs()` with `run_key`; `.lnk_log_create_tables()` reads it (DDL unchanged — +- [x] `.lnk_log_specs()` (table, cols, pk); `.lnk_log_create_tables()` reads it (DDL unchanged — existing DDL tests pass as is) ## Phase 3: Consolidate keeps history -- [ ] Split persist / history tables; DELETE persist only -- [ ] `consolidate_history_sql()` + stage 2 via `psql -1 -f` for history tables -- [ ] Update the comments that describe the bucket DELETE (script header, step 3.5, +- [x] Split persist / history tables; DELETE persist only; refuse a source before deleting if a log PK is not shared +- [x] `consolidate_history_sql()` + stage 2 via `psql -1 -f` for history tables +- [x] Update the comments that describe the bucket DELETE (script header, step 3.5, `study_area_run.sh:1389-1392` line refs) ## Phase 4: Docs diff --git a/tests/testthat/test-schema_consolidate.R b/tests/testthat/test-schema_consolidate.R new file mode 100644 index 00000000..b7776aa1 --- /dev/null +++ b/tests/testthat/test-schema_consolidate.R @@ -0,0 +1,147 @@ +# Tests for data-raw/schema_consolidate.R's handling of the log tables +# (link#354). The script runs over ssh, so these drive its two pure helpers +# and, live, the destination-side psql script it writes. +# +# data-raw/ is .Rbuildignore'd, so an installed package legitimately does not +# have the script; that, not skip_on_cran(), is the honest guard. +consolidate_script <- function() { + testthat::test_path("..", "..", "data-raw", "schema_consolidate.R") +} + +source_consolidate <- function(env = parent.frame()) { + script <- consolidate_script() + testthat::skip_if_not(file.exists(script), + "data-raw/schema_consolidate.R not in this tree") + e <- new.env(parent = globalenv()) + sys.source(script, envir = e) + e +} + +test_that("log tables are split from persist tables, in discovery order", { + e <- source_consolidate() + wgc <- c("barriers", "log", "log_input", "log_recompute", "streams", + "streams_habitat_bt") + out <- e$consolidate_tables_split(wgc) + expect_identical(out$history, c("log", "log_input", "log_recompute")) + expect_identical(out$persist, c("barriers", "streams", "streams_habitat_bt")) + # Every log table is history, including the config-hash ones that carry no + # watershed_group_code today and so are never discovered. + expect_identical(e$consolidate_tables_split(names(.lnk_log_specs()))$persist, + character(0)) +}) + +test_that("consolidate_history_sql replaces by primary key, never by group", { + e <- source_consolidate() + sql <- e$consolidate_history_sql("s", "log_input", + c("run_id", "watershed_group_code", "table_name", "row_count"), + c("run_id", "table_name"), "/tmp/x.tsv") + joined <- paste(sql, collapse = "\n") + expect_match(joined, "CREATE TEMP TABLE consolidate_in ON COMMIT DROP AS SELECT run_id, watershed_group_code, table_name, row_count FROM s.log_input WITH NO DATA;", fixed = TRUE) + expect_match(joined, "\\copy consolidate_in (run_id, watershed_group_code, table_name, row_count) FROM '/tmp/x.tsv'", fixed = TRUE) + expect_match(joined, "DELETE FROM s.log_input d USING consolidate_in s WHERE (d.run_id, d.table_name) = (s.run_id, s.table_name);", fixed = TRUE) + expect_match(joined, "INSERT INTO s.log_input (run_id, watershed_group_code, table_name, row_count) SELECT run_id, watershed_group_code, table_name, row_count FROM consolidate_in;", fixed = TRUE) + delete <- grep("^DELETE", sql, value = TRUE) + expect_no_match(delete, "watershed_group_code") +}) + +test_that("consolidate_history_sql refuses an unshared key or a quoted path", { + e <- source_consolidate() + expect_error( + e$consolidate_history_sql("s", "log", c("watershed_group_code", "host"), + "run_id", "/tmp/x.tsv"), + "primary key column\\(s\\) run_id of s.log are not shared") + expect_error( + e$consolidate_history_sql("s", "log", "run_id", "run_id", "/tmp/it's.tsv"), + "single quote") +}) + +test_that("the staged script keeps a group's earlier builds (live)", { + e <- source_consolidate() + conn <- skip_if_no_db() + psql <- Sys.which("psql") + skip_if(!nzchar(psql), "psql not on PATH") + info <- DBI::dbGetInfo(conn) + skip_if(!identical(as.character(info$port), "5432") || + !info$host %in% c("localhost", "127.0.0.1"), + "not the local docker fwapg the script writes to") + + dst <- paste0("lnk_test_consol_dst_", Sys.getpid()) + src <- paste0("lnk_test_consol_src_", Sys.getpid()) + DBI::dbExecute(conn, "SET client_min_messages TO warning") + for (sc in c(dst, src)) { + DBI::dbExecute(conn, sprintf("DROP SCHEMA IF EXISTS %s CASCADE", sc)) + link:::.lnk_log_create_tables(conn, sc) + } + withr::defer(for (sc in c(dst, src)) { + DBI::dbExecute(conn, sprintf("DROP SCHEMA IF EXISTS %s CASCADE", sc)) + }) + + # Destination: ADMS built twice before (A, then B still open), BULK once. + DBI::dbExecute(conn, sprintf( + "INSERT INTO %s.log (run_id, watershed_group_code, date_start, date_end) + VALUES ('A', 'ADMS', '2026-08-01', '2026-08-01'), + ('B', 'ADMS', '2026-10-01', NULL), + ('D', 'BULK', '2026-10-01', '2026-10-01')", dst)) + DBI::dbExecute(conn, sprintf( + "INSERT INTO %s.log_input (run_id, watershed_group_code, table_name, row_count) + VALUES ('A', 'ADMS', 'streams', 1), ('B', 'ADMS', 'streams', 2)", dst)) + # Source: B again, now closed, and a new build C of ADMS. + DBI::dbExecute(conn, sprintf( + "INSERT INTO %s.log (run_id, watershed_group_code, date_start, date_end) + VALUES ('B', 'ADMS', '2026-10-01', '2026-10-02'), + ('C', 'ADMS', '2026-10-17', '2026-10-17')", src)) + DBI::dbExecute(conn, sprintf( + "INSERT INTO %s.log_input (run_id, watershed_group_code, table_name, row_count) + VALUES ('B', 'ADMS', 'streams', 20), ('C', 'ADMS', 'streams', 3), + ('C', 'ADMS', 'falls', 4)", src)) + + env <- c(PGHOST = info$host, PGPORT = "5432", PGDATABASE = info$dbname, + PGUSER = "postgres", PGPASSWORD = "postgres") + # system2() warns on a non-zero exit; the status is read off the result. + run_psql <- function(args, stderr = TRUE) { + withr::with_envvar(env, suppressWarnings( + system2(psql, args, stdout = TRUE, stderr = stderr))) + } + stage <- function(table, cols, pk, bad = FALSE) { + tsv <- withr::local_tempfile(fileext = ".tsv") + # stderr dropped: a NOTICE on it would land in the data file. + out <- run_psql(c("-X", "-c", shQuote(sprintf( + "COPY (SELECT %s FROM %s.%s WHERE watershed_group_code IN ('ADMS')) TO STDOUT", + paste(cols, collapse = ", "), src, table))), stderr = FALSE) + writeLines(c(out, if (bad) "not\ta\trow"), tsv) + sqlf <- withr::local_tempfile(fileext = ".sql") + writeLines(e$consolidate_history_sql(dst, table, cols, pk, tsv), sqlf) + res <- run_psql(c("-X", "-1", "-v", "ON_ERROR_STOP=1", "-f", shQuote(sqlf))) + attr(res, "status") %||% 0L + } + cols_log <- c("run_id", "watershed_group_code", "date_start", "date_end") + cols_in <- c("run_id", "watershed_group_code", "table_name", "row_count") + + # A malformed file fails the whole transaction: nothing is deleted. + expect_false(identical(stage("log", cols_log, "run_id", bad = TRUE), 0L)) + expect_identical( + DBI::dbGetQuery(conn, sprintf("SELECT count(*)::int n FROM %s.log", dst))$n, + 3L) + + expect_identical(stage("log", cols_log, "run_id"), 0L) + expect_identical(stage("log_input", cols_in, c("run_id", "table_name")), 0L) + + log <- DBI::dbGetQuery(conn, sprintf( + "SELECT run_id, watershed_group_code, date_end::date::text AS date_end + FROM %s.log ORDER BY run_id", dst)) + # A survives (history), B is replaced (its date_end set), C is added, and + # BULK, outside the bucket, is untouched. + expect_identical(log$run_id, c("A", "B", "C", "D")) + expect_identical(log$date_end[log$run_id == "B"], "2026-10-02") + inp <- DBI::dbGetQuery(conn, sprintf( + "SELECT run_id, table_name, row_count::int AS n FROM %s.log_input + ORDER BY run_id, table_name", dst)) + expect_identical(paste(inp$run_id, inp$table_name, inp$n), + c("A streams 1", "B streams 20", "C falls 4", "C streams 3")) + + # Copying the same rows again changes nothing (a consolidate re-run). + expect_identical(stage("log", cols_log, "run_id"), 0L) + expect_identical( + DBI::dbGetQuery(conn, sprintf("SELECT count(*)::int n FROM %s.log", dst))$n, + 4L) +}) From 7d85b244ef0305ddbde935baacdc1c58a8a46d78 Mon Sep 17 00:00:00 2001 From: almac2022 Date: Sat, 17 Oct 2026 20:40:57 -0700 Subject: [PATCH 04/11] RUNBOOK: the log tables keep their history through consolidate (#354) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx --- RUNBOOK.md | 28 +++++++++++++++++----------- planning/active/task_plan.md | 4 ++-- 2 files changed, 19 insertions(+), 13 deletions(-) diff --git a/RUNBOOK.md b/RUNBOOK.md index 445f9829..44885e77 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -504,8 +504,13 @@ EXCEPT SELECT watershed_group_code FROM .log; persist schema). This table is what is loaded here, read by runs into every schema, and `state_clean.sh` drops `fresh` but not `provenance`, so loader rows that cannot be re-created (link#270's FWA rows) survive it. -- `log` and `log_input` carry `watershed_group_code` so `schema_consolidate.R` - auto-discovers them; `log_parameters_fresh` / `log_parameters_habitat_thresholds` / +- `log`, `log_input` and `log_recompute` carry `watershed_group_code` so + `schema_consolidate.R` auto-discovers them. They are history, not per-group + state: consolidate replaces their rows by primary key (one key is one build) + and never deletes them by group, so a group copied home from a cloud host + keeps its earlier builds' rows (link#354; before it, every cloud-built group + kept only its newest row). The set is `.lnk_log_specs()`, read by the DDL + and by consolidate. `log_parameters_fresh` / `log_parameters_habitat_thresholds` / `log_dimensions` deliberately do not (they key on `config_hash`), so **they do not yet travel between hosts** — follow-up PR. @@ -586,10 +591,10 @@ and their rows would be indistinguishable from a real post-consolidate recompute — so the verify script's model-vs-recompute check would pass on benchmark noise. -**Known exposure:** `schema_consolidate.R` DELETEs its bucket from every -discovered table before COPYing. `log_recompute` is populated *after* -consolidate, so a second or resumed consolidate over the same bucket will wipe -those rows. Re-run the recompute if that happens. +**A second or resumed consolidate keeps these rows.** `log_recompute` is +populated *after* consolidate, and consolidate leaves the log tables out of its +bucket DELETE: their rows are replaced by primary key, in one transaction per +table (link#354). Before #354 a re-consolidate over the same bucket wiped them. ### The bcfp reference pin, tunnel-free @@ -1194,10 +1199,11 @@ what stops `=== READY (install stage only; ...)` satisfying a full-prep check. ### Two post-conditions, and why they are not optional -`schema_consolidate` DELETEs the destination bucket -(`schema_consolidate.R:272-276`) and *then* COPYs (`:313-316`). A host that -produced nothing therefore does not merely fail to add rows — it **removes** the -rows already there for those WSGs and returns `ok = TRUE`. So: +`schema_consolidate` DELETEs the destination bucket from every persist table +(its step 3.5) and *then* COPYs (step 4). A host that produced nothing +therefore does not merely fail to add rows — it **removes** the rows already +there for those WSGs and returns `ok = TRUE`. (The log tables are left out of +that DELETE, so a group's log history survives it; link#354.) So: - before consolidate, every host must account for its whole bucket, read from its **database**: `.log` rows for the run_uid with `date_end` set, @@ -1268,7 +1274,7 @@ then the finished work still on the host. `unsettled` means wait until nothing runs there and re-measure. `schema_consolidate` deletes its bucket on the dispatcher before it copies, so consolidating a group the host no longer holds (already consolidated, then deleted from the source), or holds only in part, loses the -dispatcher's copy. Where it says `no`, find out what is left first. If the +dispatcher's copy of its persist rows. Its log rows survive (link#354). Where it says `no`, find out what is left first. If the run's recompute ran, it ran without the kept groups: recompute each one and each run group upstream of it (`wsg_recompute_one.R`). If it did not run (consolidate failed, coverage FATAL), recompute every group of the run; the diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md index d29279ce..c9e20a6f 100644 --- a/planning/active/task_plan.md +++ b/planning/active/task_plan.md @@ -28,9 +28,9 @@ `study_area_run.sh:1389-1392` line refs) ## Phase 4: Docs -- [ ] RUNBOOK §6c: replace "Known exposure" with the history rule; §6c note at ~507 and §8d +- [x] RUNBOOK §6c: replace "Known exposure" with the history rule; §6c note at ~507 and §8d recovery passage (~1268) say the log history survives a re-consolidate -- [ ] NEWS entry (version bump left to `/gh-pr-merge`) +- [ ] NEWS entry — deferred to `/gh-pr-merge`: in this repo NEWS is written by the release commit (`bf1521d3`), not on the branch ## Phase 5: Restore the lost rows (operator: yes) - [ ] Extract `log` + `log_input` rows for DUNC, KOTL, LARL, SLOC, PINE from the 22:42 dump From 80d5d4aa508f1779bac8dfc31e1032bd19adfc39 Mon Sep 17 00:00:00 2001 From: almac2022 Date: Sat, 17 Oct 2026 20:42:10 -0700 Subject: [PATCH 05/11] Restore the log rows #246's consolidate deleted (#354) 5 log rows (DUNC, KOTL, LARL, SLOC and PINE) and their 60 log_input rows, from the driver's pre-consolidate backup, added with ON CONFLICT DO NOTHING. study_area_verify.sql and verify_extra.sql pass at 241. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx --- CLAUDE.md | 2 +- .../logs/consolidate_history_354/README.md | 31 + .../restore_log_rows.out | 23 + .../restore_log_rows.sql | 39 ++ .../study_area_verify_after_restore.out | 570 ++++++++++++++++++ .../verify_extra_after_restore.out | 296 +++++++++ planning/active/progress.md | 3 + planning/active/task_plan.md | 6 +- 8 files changed, 966 insertions(+), 4 deletions(-) create mode 100644 data-raw/logs/consolidate_history_354/README.md create mode 100644 data-raw/logs/consolidate_history_354/restore_log_rows.out create mode 100644 data-raw/logs/consolidate_history_354/restore_log_rows.sql create mode 100644 data-raw/logs/consolidate_history_354/study_area_verify_after_restore.out create mode 100644 data-raw/logs/consolidate_history_354/verify_extra_after_restore.out diff --git a/CLAUDE.md b/CLAUDE.md index 41d74305..9d3e9413 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -22,7 +22,7 @@ Experimental package — breaking all the time and loving the learning curve. St - **The recompute takes 154 min at 241 groups and `-j4`** (median 2.0 min per group, longest 16 min), not the ~1 h the plan assumed. - **A present species with no barrier in a group is accessible** (`access_` 1 or 2) **and has `has_barriers__dnstr` FALSE**, so its mapping code is built. Zero barrier rows mean NA ("no data") only when nothing says the species is present. Before `91a968ab` the flag stayed NA and UPET RB shipped with `mapping_code_rb` "" on every segment; UPET RB is the only such case in the 241. - **`fresh_default` is 31 GB** (14 GB tables, 17 GB indexes; 11.39 M segments), up from 7.3 GB. Summing `pg_total_relation_size()` over every `pg_class` row counts each index twice (48 GB). The fwapg volume is 600 GiB with 253 G free after the run. -- **The run replaced the August KOTL, LARL and SLOC builds and the 2026-10-12 DUNC build** (status below), which published floodplains items had come from. Those four old builds survive in a machine-local `pg_dump`, the driver's pre-consolidate backup: it was taken after m1 had rebuilt its own 76 groups but before the cloud hosts' groups (all four were on job3) were copied in, so it is **not** the whole pre-run schema. The pre-run state of the other pre-run groups m1 rebuilt is gone. Operator, 2026-10-17: no floodplains item needs the old builds restored. +- **The run replaced the August KOTL, LARL and SLOC builds and the 2026-10-12 DUNC build** (status below), which published floodplains items had come from. Those four old builds survive in a machine-local `pg_dump`, the driver's pre-consolidate backup: it was taken after m1 had rebuilt its own 76 groups but before the cloud hosts' groups (all four were on job3) were copied in, so it is **not** the whole pre-run schema. The pre-run state of the other pre-run groups m1 rebuilt is gone. Operator, 2026-10-17: no floodplains item needs the old builds restored. **Their log rows were restored from that dump (#354):** DUNC, KOTL, LARL, SLOC and PINE (also cypher-built, not named before) each hold their earlier `log` row again, and their `log_input` rows (5 + 60; `data-raw/logs/consolidate_history_354/`). Since #354 consolidate keeps the log tables' history, replacing their rows by primary key. ## Status (2026-10-16) — `default_tuned` reads wet's discharge; BT's MAD minima 0.043 / 0.043 (#331) diff --git a/data-raw/logs/consolidate_history_354/README.md b/data-raw/logs/consolidate_history_354/README.md new file mode 100644 index 00000000..4040deb3 --- /dev/null +++ b/data-raw/logs/consolidate_history_354/README.md @@ -0,0 +1,31 @@ +# Restoring the log rows #246's consolidate deleted (#354) + +2026-10-17, local fwapg, `fresh_default`. Operator call at the #354 plan gate: restore. + +Run `20261017_004619-3f4c713b`'s consolidate deleted each cloud-built group's earlier `log` / +`log_input` rows on the dispatcher. The driver's pre-consolidate backup +(`/tmp/fresh_default_pre_consolidate_202610162242.dump`, machine-local, taken after m1's own +groups and before the cloud hosts' were copied in) still held them. Against the current schema +it had exactly **5 `log` rows and 60 `log_input` rows** whose primary key was gone, and an +empty `log_recompute`: + +| group | run_id | built | link | +|---|---|---|---| +| DUNC | `m1-20261012T153135.985-b33d3a` | 2026-10-12 | 0.63.1 | +| KOTL | `m1-20260827T145219.059-18b6fe` | 2026-08-27 | 0.45.1 | +| LARL | `m1-20260827T144852.009-9d37a9` | 2026-08-27 | 0.45.1 | +| PINE | `m1-20260806T184954.325-afdf9b` | 2026-08-06 | 0.44.3 | +| SLOC | `m1-20260827T145652.093-c69f2d` | 2026-08-27 | 0.45.1 | + +PINE was not named in the issue; it was built on `cypher-job2` in the run and lost its August +row the same way. PARS had no earlier row. + +- `restore_log_rows.sql` → `restore_log_rows.out`: additive `INSERT … ON CONFLICT DO NOTHING`, + `log` 241 → 246, `log_input` 2,892 → 2,952, 12 `log_input` rows per restored build. +- `study_area_verify_after_restore.out`, `verify_extra_after_restore.out`: both pass at + `run_uid=20261017_004619-3f4c713b`, `expected_n=241`. +- `lnk_log_read()`: 241 groups, all 241 newest rows from this run; 246 rows with + `latest = FALSE`. + +The persist rows of those old builds are not restored (operator, 2026-10-17: no floodplains +item needs them). The scratch schema `zz354_restore` was dropped after the restore. diff --git a/data-raw/logs/consolidate_history_354/restore_log_rows.out b/data-raw/logs/consolidate_history_354/restore_log_rows.out new file mode 100644 index 00000000..22983a94 --- /dev/null +++ b/data-raw/logs/consolidate_history_354/restore_log_rows.out @@ -0,0 +1,23 @@ +BEGIN + at | log | log_input +--------+-----+----------- + before | 241 | 2892 +(1 row) + +INSERT 0 5 +INSERT 0 60 + at | log | log_input +-------+-----+----------- + after | 246 | 2952 +(1 row) + + watershed_group_code | run_id | date_start | link_version | log_input_rows +----------------------+-------------------------------+------------+--------------+---------------- + DUNC | m1-20261012T153135.985-b33d3a | 2026-10-12 | 0.63.1 | 12 + KOTL | m1-20260827T145219.059-18b6fe | 2026-08-27 | 0.45.1 | 12 + LARL | m1-20260827T144852.009-9d37a9 | 2026-08-27 | 0.45.1 | 12 + PINE | m1-20260806T184954.325-afdf9b | 2026-08-06 | 0.44.3 | 12 + SLOC | m1-20260827T145652.093-c69f2d | 2026-08-27 | 0.45.1 | 12 +(5 rows) + +COMMIT diff --git a/data-raw/logs/consolidate_history_354/restore_log_rows.sql b/data-raw/logs/consolidate_history_354/restore_log_rows.sql new file mode 100644 index 00000000..cbe8df48 --- /dev/null +++ b/data-raw/logs/consolidate_history_354/restore_log_rows.sql @@ -0,0 +1,39 @@ +-- Restore the log rows run_uid 20261017_004619-3f4c713b's consolidate deleted +-- from fresh_default (link#354). Source: the driver's pre-consolidate backup +-- /tmp/fresh_default_pre_consolidate_202610162242.dump (machine-local), its +-- log / log_input / log_recompute data loaded into zz354_restore first: +-- +-- pg_restore --data-only -t -f .sql +-- sed 's/^COPY fresh_default\./COPY zz354_restore./' .sql | psql +-- +-- Additive: a row whose primary key is present is left alone, so re-running +-- this changes nothing. Columns added since the dump take NULL. +\set ON_ERROR_STOP on +BEGIN; + +SELECT 'before' AS at, + (SELECT count(*) FROM fresh_default.log) AS log, + (SELECT count(*) FROM fresh_default.log_input) AS log_input; + +INSERT INTO fresh_default.log +SELECT * FROM zz354_restore.log +ON CONFLICT (run_id) DO NOTHING; + +INSERT INTO fresh_default.log_input +SELECT * FROM zz354_restore.log_input +ON CONFLICT (run_id, table_name) DO NOTHING; + +SELECT 'after' AS at, + (SELECT count(*) FROM fresh_default.log) AS log, + (SELECT count(*) FROM fresh_default.log_input) AS log_input; + +-- Every restored log_input row belongs to a restored log row. +SELECT l.watershed_group_code, l.run_id, l.date_start::date, l.link_version, + (SELECT count(*) FROM fresh_default.log_input i + WHERE i.run_id = l.run_id) AS log_input_rows + FROM fresh_default.log l + WHERE l.run_id IN (SELECT run_id FROM zz354_restore.log) + AND l.run_uid IS DISTINCT FROM '20261017_004619-3f4c713b' + ORDER BY 1; + +COMMIT; diff --git a/data-raw/logs/consolidate_history_354/study_area_verify_after_restore.out b/data-raw/logs/consolidate_history_354/study_area_verify_after_restore.out new file mode 100644 index 00000000..8a3e2c15 --- /dev/null +++ b/data-raw/logs/consolidate_history_354/study_area_verify_after_restore.out @@ -0,0 +1,570 @@ + +=== run under verification === + run_uid | run_label | n_attempted | n_completed | n_hosts | bcfp_reference | bcfp_pin_source | started | finished +--------------------------+----------------------------+-------------+-------------+---------+---------------------+-----------------+-------------------------------+------------------------------- + 20261017_004619-3f4c713b | provincial-241-default-246 | 241 | 241 | 4 | v0.7.15-50-g620dbe4 | env | 2026-10-17 01:05:40.282962+00 | 2026-10-17 06:48:58.764204+00 +(1 row) + + scope_assertion +------------------ + expected_n = 241 +(1 row) + + +=== 1. Provenance per host === + host | n_wsg | has_link_sha | has_fresh_sha | has_fwapg_sha | has_bcfishobs_sha | has_bcfp_version | n_dirty | n_dirty_unknown | first_done | last_done +-------------+-------+--------------+---------------+---------------+-------------------+------------------+---------+-----------------+-------------------------------+------------------------------- + cypher-job1 | 49 | 49 | 49 | 49 | 49 | 49 | 0 | 0 | 2026-10-17 01:10:13.850851+00 | 2026-10-17 03:56:28.382763+00 + cypher-job2 | 55 | 55 | 55 | 55 | 55 | 55 | 0 | 0 | 2026-10-17 01:09:42.478699+00 | 2026-10-17 04:33:01.112688+00 + cypher-job3 | 59 | 59 | 59 | 59 | 59 | 59 | 0 | 0 | 2026-10-17 01:09:16.570281+00 | 2026-10-17 05:42:44.955319+00 + m1 | 78 | 78 | 78 | 78 | 78 | 78 | 0 | 0 | 2026-10-17 01:08:11.053768+00 | 2026-10-17 06:48:58.764204+00 +(4 rows) + + +=== 1b. Provenance verdict (per host) === + host | verdict +-------------+--------- + cypher-job1 | OK + cypher-job2 | OK + cypher-job3 | OK + m1 | OK +(4 rows) + + +=== 1c. Started but never finished (must be empty) === + host | watershed_group_code | date_start +------+----------------------+------------ +(0 rows) + + +=== 2. The distinct SHAs -- all hosts must agree === + link_sha | fresh_sha | fwapg_sha | bcfishobs_sha | bcfp_model_version +------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+--------------------- + ee21eb1371cbc4d7a2d598f8bbeeaeb0534d7994 | e247ca19a7803982dc7e5f55dbbf377dfc85d007 | f9718cb739d376155641f484456eb2a93ce2f481 | 08630cf4ac14b2f17cc4d7058b127f3996f3fd8a | v0.7.15-50-g620dbe4 + a17e91773b283b021811f8b5b76b887d5f4c16e4 | e247ca19a7803982dc7e5f55dbbf377dfc85d007 | f9718cb739d376155641f484456eb2a93ce2f481 | 08630cf4ac14b2f17cc4d7058b127f3996f3fd8a | v0.7.15-50-g620dbe4 +(2 rows) + + +=== 2b. Inputs every host must agree on (must be one row) === + n_distinct_input_sets | verdict +-----------------------+--------- + 1 | OK +(1 row) + + +=== 3. Coverage: every WSG of this run has streams rows === + wsg | n_segments +------+------------ + ADMS | 39424 + ALBN | 53918 + ATLL | 70907 + ATNA | 35786 + BABL | 45789 + BABR | 56983 + BARR | 38057 + BBAR | 62623 + BEAV | 44924 + BELA | 31874 + BIGC | 11291 + BLAR | 19126 + BLUR | 30739 + BONP | 47809 + BOWR | 36727 + BRID | 9225 + BRKS | 42690 + BULK | 87641 + BULL | 44835 + CAMB | 22181 + CANO | 37217 + CARP | 19296 + CARR | 40786 + CHES | 11886 + CHIL | 10171 + CHIR | 36822 + CHWK | 14346 + CLAY | 51047 + CLRH | 83688 + CLWR | 57451 + COAL | 72602 + COLR | 70505 + COMX | 22463 + COTR | 36816 + COWN | 32966 + CRKD | 25330 + CRYL | 26110 + DEAD | 16908 + DEAL | 48608 + DEAR | 24331 + DOGC | 27242 + DRIR | 21375 + DUNC | 63067 + DUNE | 40486 + ELKR | 83358 + EUCH | 8137 + EUCL | 15146 + FINA | 70329 + FINL | 96218 + FIRE | 68747 + FONT | 18240 + FOXR | 71953 + FRAN | 53354 + FRCN | 77154 + FROG | 83129 + GATA | 60814 + GLAR | 27351 + GOLD | 42759 + GRAI | 78089 + GRNL | 8887 + GUIC | 12411 + HARR | 44112 + HERR | 39334 + HOLB | 19788 + HOMA | 84798 + HORS | 28456 + INGR | 73469 + INKR | 100064 + ISKR | 52765 + JENR | 24136 + JERV | 58691 + KAHN | 12956 + KAKC | 24364 + KEEC | 17815 + KETL | 102085 + KHOR | 77163 + KHTZ | 53937 + KINR | 31576 + KISK | 18527 + KISP | 76518 + KITL | 61968 + KITR | 69069 + KLAR | 72129 + KLIN | 73384 + KLUM | 73905 + KNIG | 83528 + KOTL | 109950 + KOTR | 82546 + KSHR | 61433 + KTSU | 16473 + KUMR | 56518 + KUSR | 50494 + LARL | 91090 + LBIR | 50308 + LBTN | 29869 + LCHL | 21173 + LCHR | 28046 + LDEN | 43348 + LEUT | 9786 + LFRA | 58201 + LFRT | 16684 + LHAF | 58899 + LIAR | 114814 + LILL | 82729 + LISR | 42964 + LKEC | 46783 + LKEL | 7448 + LMUS | 35452 + LNAR | 88116 + LNIC | 50948 + LNRS | 13788 + LNTH | 64335 + LOMI | 50232 + LPCE | 13698 + LPET | 9549 + LPRO | 9422 + LRAN | 40674 + LRDO | 40334 + LSAL | 11609 + LSIK | 20598 + LSKE | 77495 + LSTR | 42383 + LTRE | 11460 + MAHD | 42363 + MBNK | 10219 + MCGR | 49005 + MDEA | 50174 + MESC | 33244 + MESI | 42928 + MFRA | 15088 + MFRT | 12759 + MIDR | 15545 + MILL | 8952 + MMUS | 90002 + MORI | 59847 + MORK | 110511 + MORR | 58613 + MPRO | 20119 + MSKE | 84222 + MSTR | 78567 + MURR | 79173 + MURT | 37569 + MUSK | 8228 + NAHR | 31448 + NAKR | 69078 + NARC | 21766 + NASC | 65252 + NASR | 39290 + NATR | 47300 + NAZR | 24517 + NBNK | 4214 + NECL | 47143 + NECR | 29952 + NEVI | 14478 + NICL | 18923 + NIEL | 56249 + NIMP | 34905 + OKAN | 69057 + OSPK | 60735 + OWIK | 105081 + PARA | 36927 + PARK | 14115 + PARS | 97533 + PCEA | 106080 + PINE | 113304 + PITR | 35580 + PORI | 10364 + QUES | 59909 + REVL | 70301 + SAHD | 7055 + SAJR | 10396 + SALM | 25844 + SALR | 11392 + SANJ | 60522 + SETN | 188554 + SEYM | 52336 + SHER | 66049 + SHUL | 50703 + SIML | 94852 + SKGT | 17607 + SLOC | 55733 + SMAR | 78683 + SMOK | 71577 + SPAT | 58082 + SQAM | 79864 + STHM | 40673 + STIR | 41587 + STUL | 21375 + STUR | 13820 + SUST | 60652 + SWIR | 30618 + TABR | 14652 + TAHR | 20239 + TAHS | 26187 + TAKL | 28461 + TASR | 30537 + TATR | 79209 + TAYR | 30135 + TESR | 30305 + THOM | 62147 + TOAD | 140093 + TOBA | 83496 + TOOD | 97296 + TSAY | 48761 + TSEA | 10600 + TSIT | 38586 + TURN | 95863 + TUTR | 84560 + TUYR | 29857 + TWAC | 15561 + UARL | 75565 + UBIR | 36191 + UBTN | 26991 + UCHR | 19968 + UDEN | 24601 + UEUT | 35610 + UFRA | 96301 + UFRT | 20036 + UHAF | 71366 + UISR | 75060 + UJER | 21867 + UKEC | 40219 + ULRD | 42059 + UMUS | 69838 + UNAR | 85478 + UNRS | 52747 + UNTH | 72906 + UNUR | 36385 + UOMI | 44543 + UPCE | 46623 + UPET | 5103 + UPRO | 55503 + USHU | 50838 + USIK | 71518 + USKE | 80375 + USTK | 45584 + UTRE | 11305 + VICT | 4839 + WILL | 26163 + WORC | 53926 + ZYMO | 59171 +(241 rows) + + +=== 4. Modelled vs recomputed -- the difference, per WSG === + wsg | modelled | recomputed | model_done | recompute_done | state +------+----------+------------+-------------------------------+-------------------------------+------- + ADMS | t | t | 2026-10-17 02:45:24.289298+00 | 2026-10-17 07:37:16.180939+00 | ok + ALBN | t | t | 2026-10-17 01:10:13.850851+00 | 2026-10-17 07:37:01.357207+00 | ok + ATLL | t | t | 2026-10-17 02:53:16.487051+00 | 2026-10-17 07:35:26.455085+00 | ok + ATNA | t | t | 2026-10-17 02:15:33.987745+00 | 2026-10-17 07:39:04.977501+00 | ok + BABL | t | t | 2026-10-17 05:32:24.378915+00 | 2026-10-17 07:38:52.718527+00 | ok + BABR | t | t | 2026-10-17 04:50:59.736065+00 | 2026-10-17 07:40:08.966419+00 | ok + BARR | t | t | 2026-10-17 03:01:51.993367+00 | 2026-10-17 07:42:08.630552+00 | ok + BBAR | t | t | 2026-10-17 02:23:26.230398+00 | 2026-10-17 07:19:32.556628+00 | ok + BEAV | t | t | 2026-10-17 02:15:29.482985+00 | 2026-10-17 07:40:08.86516+00 | ok + BELA | t | t | 2026-10-17 01:09:16.570281+00 | 2026-10-17 07:41:57.645199+00 | ok + BIGC | t | t | 2026-10-17 03:22:23.237402+00 | 2026-10-17 07:40:35.107324+00 | ok + BLAR | t | t | 2026-10-17 03:43:34.801041+00 | 2026-10-17 07:41:05.651382+00 | ok + BLUR | t | t | 2026-10-17 03:29:54.092712+00 | 2026-10-17 07:41:29.966297+00 | ok + BONP | t | t | 2026-10-17 02:26:56.835087+00 | 2026-10-17 07:45:20.376631+00 | ok + BOWR | t | t | 2026-10-17 04:11:28.808048+00 | 2026-10-17 07:26:51.623988+00 | ok + BRID | t | t | 2026-10-17 03:17:40.060746+00 | 2026-10-17 07:41:53.415727+00 | ok + BRKS | t | t | 2026-10-17 01:13:35.902828+00 | 2026-10-17 07:45:34.244516+00 | ok + BULK | t | t | 2026-10-17 03:58:27.174029+00 | 2026-10-17 07:20:03.487615+00 | ok + BULL | t | t | 2026-10-17 03:06:52.076596+00 | 2026-10-17 07:44:28.464566+00 | ok + CAMB | t | t | 2026-10-17 01:08:11.053768+00 | 2026-10-17 07:45:24.229868+00 | ok + CANO | t | t | 2026-10-17 04:53:13.155128+00 | 2026-10-17 07:45:24.629906+00 | ok + CARP | t | t | 2026-10-17 03:34:49.974829+00 | 2026-10-17 09:30:20.418455+00 | ok + CARR | t | t | 2026-10-17 03:46:49.399618+00 | 2026-10-17 07:48:41.882082+00 | ok + CHES | t | t | 2026-10-17 04:27:56.387419+00 | 2026-10-17 09:25:44.359466+00 | ok + CHIL | t | t | 2026-10-17 04:12:26.900536+00 | 2026-10-17 07:46:04.286049+00 | ok + CHIR | t | t | 2026-10-17 03:24:34.616814+00 | 2026-10-17 07:47:31.637162+00 | ok + CHWK | t | t | 2026-10-17 01:41:10.763777+00 | 2026-10-17 07:22:16.648172+00 | ok + CLAY | t | t | 2026-10-17 01:09:42.478699+00 | 2026-10-17 07:48:46.293527+00 | ok + CLRH | t | t | 2026-10-17 04:04:00.901925+00 | 2026-10-17 07:48:19.310028+00 | ok + CLWR | t | t | 2026-10-17 02:48:26.90073+00 | 2026-10-17 07:50:51.612846+00 | ok + COAL | t | t | 2026-10-17 02:50:46.894262+00 | 2026-10-17 07:49:55.563786+00 | ok + COLR | t | t | 2026-10-17 05:42:44.955319+00 | 2026-10-17 07:50:40.090703+00 | ok + COMX | t | t | 2026-10-17 01:11:40.10847+00 | 2026-10-17 07:52:34.119206+00 | ok + COTR | t | t | 2026-10-17 03:34:48.256702+00 | 2026-10-17 07:24:02.959477+00 | ok + COWN | t | t | 2026-10-17 01:14:55.578278+00 | 2026-10-17 09:29:19.131116+00 | ok + CRKD | t | t | 2026-10-17 03:56:08.146558+00 | 2026-10-17 09:29:40.207185+00 | ok + CRYL | t | t | 2026-10-17 03:40:50.739832+00 | 2026-10-17 07:50:36.178581+00 | ok + DEAD | t | t | 2026-10-17 02:29:02.900736+00 | 2026-10-17 07:53:17.088126+00 | ok + DEAL | t | t | 2026-10-17 03:43:41.923288+00 | 2026-10-17 07:51:56.00447+00 | ok + DEAR | t | t | 2026-10-17 03:11:46.328243+00 | 2026-10-17 07:51:38.745242+00 | ok + DOGC | t | t | 2026-10-17 02:51:06.692801+00 | 2026-10-17 07:29:43.756801+00 | ok + DRIR | t | t | 2026-10-17 04:50:26.105285+00 | 2026-10-17 07:53:52.954502+00 | ok + DUNC | t | t | 2026-10-17 03:10:56.864909+00 | 2026-10-17 07:54:14.569327+00 | ok + DUNE | t | t | 2026-10-17 01:16:07.646938+00 | 2026-10-17 07:54:14.028089+00 | ok + ELKR | t | t | 2026-10-17 04:13:11.186166+00 | 2026-10-17 07:57:15.984428+00 | ok + EUCH | t | t | 2026-10-17 03:52:16.198752+00 | 2026-10-17 07:54:36.104184+00 | ok + EUCL | t | t | 2026-10-17 03:53:33.626417+00 | 2026-10-17 07:55:21.814944+00 | ok + FINA | t | t | 2026-10-17 03:40:11.005501+00 | 2026-10-17 07:56:52.404346+00 | ok + FINL | t | t | 2026-10-17 04:01:16.768317+00 | 2026-10-17 09:22:22.628264+00 | ok + FIRE | t | t | 2026-10-17 04:28:43.6147+00 | 2026-10-17 09:22:14.210127+00 | ok + FONT | t | t | 2026-10-17 02:58:23.70067+00 | 2026-10-17 07:55:30.372625+00 | ok + FOXR | t | t | 2026-10-17 04:33:01.112688+00 | 2026-10-17 09:24:22.008877+00 | ok + FRAN | t | t | 2026-10-17 04:32:28.019561+00 | 2026-10-17 07:27:11.192189+00 | ok + FRCN | t | t | 2026-10-17 01:49:52.223388+00 | 2026-10-17 07:09:04.314665+00 | ok + FROG | t | t | 2026-10-17 03:47:28.864007+00 | 2026-10-17 07:57:19.737826+00 | ok + GATA | t | t | 2026-10-17 03:56:28.382763+00 | 2026-10-17 07:57:22.537381+00 | ok + GLAR | t | t | 2026-10-17 02:15:34.498448+00 | 2026-10-17 07:57:58.425749+00 | ok + GOLD | t | t | 2026-10-17 01:19:06.932409+00 | 2026-10-17 08:00:08.413161+00 | ok + GRAI | t | t | 2026-10-17 01:27:38.877555+00 | 2026-10-17 08:01:28.068496+00 | ok + GRNL | t | t | 2026-10-17 02:51:47.283149+00 | 2026-10-17 07:57:49.365253+00 | ok + GUIC | t | t | 2026-10-17 02:53:25.389429+00 | 2026-10-17 07:59:23.391623+00 | ok + HARR | t | t | 2026-10-17 01:36:24.502824+00 | 2026-10-17 07:15:35.055544+00 | ok + HERR | t | t | 2026-10-17 04:34:33.143599+00 | 2026-10-17 07:59:32.614219+00 | ok + HOLB | t | t | 2026-10-17 01:11:24.159287+00 | 2026-10-17 08:00:47.889244+00 | ok + HOMA | t | t | 2026-10-17 01:17:34.634169+00 | 2026-10-17 08:04:16.952219+00 | ok + HORS | t | t | 2026-10-17 03:49:16.72481+00 | 2026-10-17 08:02:36.768607+00 | ok + INGR | t | t | 2026-10-17 04:05:50.999678+00 | 2026-10-17 09:22:11.993377+00 | ok + INKR | t | t | 2026-10-17 01:35:04.560526+00 | 2026-10-17 08:08:26.16836+00 | ok + ISKR | t | t | 2026-10-17 03:13:52.923537+00 | 2026-10-17 08:02:24.095415+00 | ok + JENR | t | t | 2026-10-17 02:17:07.851644+00 | 2026-10-17 08:03:24.09063+00 | ok + JERV | t | t | 2026-10-17 01:19:47.233644+00 | 2026-10-17 08:06:39.062356+00 | ok + KAHN | t | t | 2026-10-17 03:13:04.005079+00 | 2026-10-17 08:04:06.542108+00 | ok + KAKC | t | t | 2026-10-17 04:14:58.65896+00 | 2026-10-17 09:30:01.767555+00 | ok + KEEC | t | t | 2026-10-17 01:21:15.739966+00 | 2026-10-17 08:04:35.498307+00 | ok + KETL | t | t | 2026-10-17 01:21:48.682401+00 | 2026-10-17 08:06:03.744817+00 | ok + KHOR | t | t | 2026-10-17 05:38:07.529345+00 | 2026-10-17 08:06:38.917703+00 | ok + KHTZ | t | t | 2026-10-17 01:38:20.173616+00 | 2026-10-17 08:10:06.347095+00 | ok + KINR | t | t | 2026-10-17 02:56:03.62604+00 | 2026-10-17 08:08:39.583754+00 | ok + KISK | t | t | 2026-10-17 02:20:42.419175+00 | 2026-10-17 08:07:34.308196+00 | ok + KISP | t | t | 2026-10-17 03:22:31.5574+00 | 2026-10-17 07:02:52.382194+00 | ok + KITL | t | t | 2026-10-17 02:20:29.604182+00 | 2026-10-17 08:12:34.274051+00 | ok + KITR | t | t | 2026-10-17 06:47:41.061539+00 | 2026-10-17 08:13:56.265711+00 | ok + KLAR | t | t | 2026-10-17 04:19:50.785383+00 | 2026-10-17 08:10:53.363975+00 | ok + KLIN | t | t | 2026-10-17 01:27:41.987059+00 | 2026-10-17 08:14:39.236487+00 | ok + KLUM | t | t | 2026-10-17 02:22:58.629571+00 | 2026-10-17 07:16:46.071486+00 | ok + KNIG | t | t | 2026-10-17 01:27:23.843675+00 | 2026-10-17 08:15:36.470458+00 | ok + KOTL | t | t | 2026-10-17 02:31:24.06899+00 | 2026-10-17 08:16:02.98882+00 | ok + KOTR | t | t | 2026-10-17 04:58:43.813714+00 | 2026-10-17 08:16:00.435592+00 | ok + KSHR | t | t | 2026-10-17 01:31:25.245887+00 | 2026-10-17 08:18:51.481524+00 | ok + KTSU | t | t | 2026-10-17 01:09:30.495919+00 | 2026-10-17 08:17:08.860025+00 | ok + KUMR | t | t | 2026-10-17 01:30:49.499771+00 | 2026-10-17 08:18:44.41689+00 | ok + KUSR | t | t | 2026-10-17 01:33:08.790813+00 | 2026-10-17 08:17:09.631355+00 | ok + LARL | t | t | 2026-10-17 01:35:50.8151+00 | 2026-10-17 08:19:55.974805+00 | ok + LBIR | t | t | 2026-10-17 02:59:46.070869+00 | 2026-10-17 08:19:52.658335+00 | ok + LBTN | t | t | 2026-10-17 02:24:57.095596+00 | 2026-10-17 09:27:49.728323+00 | ok + LCHL | t | t | 2026-10-17 03:55:37.619426+00 | 2026-10-17 09:17:55.398717+00 | ok + LCHR | t | t | 2026-10-17 03:19:50.062379+00 | 2026-10-17 08:20:12.444296+00 | ok + LDEN | t | t | 2026-10-17 01:39:43.453482+00 | 2026-10-17 08:21:47.452302+00 | ok + LEUT | t | t | 2026-10-17 04:39:15.151727+00 | 2026-10-17 08:20:12.984958+00 | ok + LFRA | t | t | 2026-10-17 01:18:01.991288+00 | 2026-10-17 07:07:10.795518+00 | ok + LFRT | t | t | 2026-10-17 02:22:27.984543+00 | 2026-10-17 08:20:37.898801+00 | ok + LHAF | t | t | 2026-10-17 03:04:30.370464+00 | 2026-10-17 08:22:05.066643+00 | ok + LIAR | t | t | 2026-10-17 02:39:58.29392+00 | 2026-10-17 08:23:05.116126+00 | ok + LILL | t | t | 2026-10-17 01:57:40.181693+00 | 2026-10-17 08:29:20.854252+00 | ok + LISR | t | t | 2026-10-17 02:36:40.178111+00 | 2026-10-17 08:29:17.840727+00 | ok + LKEC | t | t | 2026-10-17 03:14:49.307198+00 | 2026-10-17 08:22:59.789616+00 | ok + LKEL | t | t | 2026-10-17 02:38:32.071617+00 | 2026-10-17 07:30:43.310829+00 | ok + LMUS | t | t | 2026-10-17 02:54:21.653329+00 | 2026-10-17 08:24:37.294805+00 | ok + LNAR | t | t | 2026-10-17 01:40:56.531459+00 | 2026-10-17 08:28:46.872239+00 | ok + LNIC | t | t | 2026-10-17 02:33:10.796193+00 | 2026-10-17 08:29:03.355226+00 | ok + LNRS | t | t | 2026-10-17 04:36:52.034529+00 | 2026-10-17 08:29:07.9293+00 | ok + LNTH | t | t | 2026-10-17 02:38:24.013855+00 | 2026-10-17 07:14:21.667327+00 | ok + LOMI | t | t | 2026-10-17 04:09:23.636058+00 | 2026-10-17 09:26:43.203965+00 | ok + LPCE | t | t | 2026-10-17 01:43:08.221784+00 | 2026-10-17 09:30:25.253335+00 | ok + LPET | t | t | 2026-10-17 01:41:11.772429+00 | 2026-10-17 09:30:13.159434+00 | ok + LPRO | t | t | 2026-10-17 03:00:14.177043+00 | 2026-10-17 08:29:56.720096+00 | ok + LRAN | t | t | 2026-10-17 03:17:12.763375+00 | 2026-10-17 08:30:21.292786+00 | ok + LRDO | t | t | 2026-10-17 01:44:06.008562+00 | 2026-10-17 08:31:53.309473+00 | ok + LSAL | t | t | 2026-10-17 03:56:40.8878+00 | 2026-10-17 09:29:25.879485+00 | ok + LSIK | t | t | 2026-10-17 03:02:00.178157+00 | 2026-10-17 08:29:53.744277+00 | ok + LSKE | t | t | 2026-10-17 01:55:03.807291+00 | 2026-10-17 07:10:48.632546+00 | ok + LSTR | t | t | 2026-10-17 01:58:21.033201+00 | 2026-10-17 09:20:21.018113+00 | ok + LTRE | t | t | 2026-10-17 04:38:34.825538+00 | 2026-10-17 08:31:17.959758+00 | ok + MAHD | t | t | 2026-10-17 03:08:03.972737+00 | 2026-10-17 08:32:07.109252+00 | ok + MBNK | t | t | 2026-10-17 01:59:45.307151+00 | 2026-10-17 08:30:50.530375+00 | ok + MCGR | t | t | 2026-10-17 04:15:17.940245+00 | 2026-10-17 08:32:30.718806+00 | ok + MDEA | t | t | 2026-10-17 03:32:47.853126+00 | 2026-10-17 08:32:52.14239+00 | ok + MESC | t | t | 2026-10-17 03:24:34.8061+00 | 2026-10-17 08:32:50.97875+00 | ok + MESI | t | t | 2026-10-17 04:12:30.250464+00 | 2026-10-17 09:26:59.128514+00 | ok + MFRA | t | t | 2026-10-17 03:09:33.668855+00 | 2026-10-17 09:25:15.736179+00 | ok + MFRT | t | t | 2026-10-17 02:41:24.462108+00 | 2026-10-17 08:32:42.96525+00 | ok + MIDR | t | t | 2026-10-17 04:41:51.931773+00 | 2026-10-17 08:33:56.272916+00 | ok + MILL | t | t | 2026-10-17 03:20:50.069016+00 | 2026-10-17 08:33:11.09889+00 | ok + MMUS | t | t | 2026-10-17 03:06:30.491845+00 | 2026-10-17 08:34:19.430663+00 | ok + MORI | t | t | 2026-10-17 01:47:03.205956+00 | 2026-10-17 08:36:01.103704+00 | ok + MORK | t | t | 2026-10-17 04:03:45.154128+00 | 2026-10-17 07:23:42.702175+00 | ok + MORR | t | t | 2026-10-17 05:06:21.922011+00 | 2026-10-17 07:24:11.554998+00 | ok + MPRO | t | t | 2026-10-17 03:19:47.437893+00 | 2026-10-17 08:34:24.648536+00 | ok + MSKE | t | t | 2026-10-17 04:28:16.787719+00 | 2026-10-17 07:29:07.093897+00 | ok + MSTR | t | t | 2026-10-17 03:31:13.80664+00 | 2026-10-17 07:33:15.378165+00 | ok + MURR | t | t | 2026-10-17 02:30:24.709213+00 | 2026-10-17 08:36:30.52856+00 | ok + MURT | t | t | 2026-10-17 03:10:43.795813+00 | 2026-10-17 08:34:57.241485+00 | ok + MUSK | t | t | 2026-10-17 04:16:00.300224+00 | 2026-10-17 08:34:41.337969+00 | ok + NAHR | t | t | 2026-10-17 02:25:09.317346+00 | 2026-10-17 08:36:03.990026+00 | ok + NAKR | t | t | 2026-10-17 02:29:02.165752+00 | 2026-10-17 08:38:48.851051+00 | ok + NARC | t | t | 2026-10-17 03:27:21.851621+00 | 2026-10-17 07:32:00.420068+00 | ok + NASC | t | t | 2026-10-17 02:03:48.667002+00 | 2026-10-17 08:39:39.636382+00 | ok + NASR | t | t | 2026-10-17 02:33:47.69429+00 | 2026-10-17 08:38:40.855305+00 | ok + NATR | t | t | 2026-10-17 03:45:30.17598+00 | 2026-10-17 09:28:24.673382+00 | ok + NAZR | t | t | 2026-10-17 04:05:11.83553+00 | 2026-10-17 08:37:23.258932+00 | ok + NBNK | t | t | 2026-10-17 02:05:08.890097+00 | 2026-10-17 08:39:38.475486+00 | ok + NECL | t | t | 2026-10-17 01:50:25.66688+00 | 2026-10-17 08:41:53.974642+00 | ok + NECR | t | t | 2026-10-17 04:19:11.161647+00 | 2026-10-17 07:31:32.249378+00 | ok + NEVI | t | t | 2026-10-17 01:43:32.241869+00 | 2026-10-17 08:41:14.205299+00 | ok + NICL | t | t | 2026-10-17 02:54:56.452547+00 | 2026-10-17 08:40:50.098368+00 | ok + NIEL | t | t | 2026-10-17 01:47:32.458306+00 | 2026-10-17 08:43:54.003062+00 | ok + NIMP | t | t | 2026-10-17 01:53:50.967455+00 | 2026-10-17 08:45:00.629902+00 | ok + OKAN | t | t | 2026-10-17 01:20:54.175163+00 | 2026-10-17 08:43:30.31921+00 | ok + OSPK | t | t | 2026-10-17 04:15:43.442977+00 | 2026-10-17 09:25:36.817995+00 | ok + OWIK | t | t | 2026-10-17 01:53:48.877417+00 | 2026-10-17 08:48:34.94294+00 | ok + PARA | t | t | 2026-10-17 03:23:38.502855+00 | 2026-10-17 09:29:05.336494+00 | ok + PARK | t | t | 2026-10-17 01:55:54.021494+00 | 2026-10-17 08:46:04.677851+00 | ok + PARS | t | t | 2026-10-17 03:53:58.536675+00 | 2026-10-17 09:24:02.224329+00 | ok + PCEA | t | t | 2026-10-17 03:09:58.937302+00 | 2026-10-17 09:24:38.571877+00 | ok + PINE | t | t | 2026-10-17 02:40:55.990606+00 | 2026-10-17 09:26:47.298282+00 | ok + PITR | t | t | 2026-10-17 04:30:11.320929+00 | 2026-10-17 08:44:17.93271+00 | ok + PORI | t | t | 2026-10-17 01:55:24.973959+00 | 2026-10-17 08:45:28.443277+00 | ok + QUES | t | t | 2026-10-17 03:39:46.913446+00 | 2026-10-17 08:49:38.688+00 | ok + REVL | t | t | 2026-10-17 03:35:44.92438+00 | 2026-10-17 08:47:20.157698+00 | ok + SAHD | t | t | 2026-10-17 03:36:54.154074+00 | 2026-10-17 08:46:27.809458+00 | ok + SAJR | t | t | 2026-10-17 03:28:40.810817+00 | 2026-10-17 08:47:43.336778+00 | ok + SALM | t | t | 2026-10-17 01:58:01.931169+00 | 2026-10-17 08:50:23.897316+00 | ok + SALR | t | t | 2026-10-17 04:20:12.740668+00 | 2026-10-17 09:29:52.006556+00 | ok + SANJ | t | t | 2026-10-17 01:23:44.391533+00 | 2026-10-17 08:51:37.870255+00 | ok + SETN | t | t | 2026-10-17 02:14:12.275417+00 | 2026-10-17 07:12:58.434556+00 | ok + SEYM | t | t | 2026-10-17 01:26:12.037331+00 | 2026-10-17 08:51:33.11226+00 | ok + SHER | t | t | 2026-10-17 02:33:13.6056+00 | 2026-10-17 08:52:30.660398+00 | ok + SHUL | t | t | 2026-10-17 03:00:47.209112+00 | 2026-10-17 08:57:03.582968+00 | ok + SIML | t | t | 2026-10-17 02:02:51.733016+00 | 2026-10-17 08:52:58.126709+00 | ok + SKGT | t | t | 2026-10-17 01:57:11.721195+00 | 2026-10-17 08:53:16.689567+00 | ok + SLOC | t | t | 2026-10-17 02:42:16.430722+00 | 2026-10-17 08:54:10.944474+00 | ok + SMAR | t | t | 2026-10-17 04:37:06.020262+00 | 2026-10-17 08:54:59.833046+00 | ok + SMOK | t | t | 2026-10-17 02:01:30.96635+00 | 2026-10-17 08:55:30.808182+00 | ok + SPAT | t | t | 2026-10-17 05:10:45.615023+00 | 2026-10-17 08:56:15.443682+00 | ok + SQAM | t | t | 2026-10-17 02:11:56.730471+00 | 2026-10-17 09:01:49.665857+00 | ok + STHM | t | t | 2026-10-17 02:41:55.963138+00 | 2026-10-17 08:59:41.026013+00 | ok + STIR | t | t | 2026-10-17 02:46:14.249573+00 | 2026-10-17 07:33:52.053281+00 | ok + STUL | t | t | 2026-10-17 04:36:08.094912+00 | 2026-10-17 08:57:43.592323+00 | ok + STUR | t | t | 2026-10-17 04:21:55.017534+00 | 2026-10-17 08:58:24.423094+00 | ok + SUST | t | t | 2026-10-17 05:18:31.559014+00 | 2026-10-17 09:02:20.029262+00 | ok + SWIR | t | t | 2026-10-17 02:43:38.424655+00 | 2026-10-17 08:59:53.928924+00 | ok + TABR | t | t | 2026-10-17 03:51:23.187206+00 | 2026-10-17 07:33:37.120403+00 | ok + TAHR | t | t | 2026-10-17 03:39:08.031857+00 | 2026-10-17 09:01:17.63666+00 | ok + TAHS | t | t | 2026-10-17 01:27:55.983646+00 | 2026-10-17 09:02:26.310677+00 | ok + TAKL | t | t | 2026-10-17 04:48:18.350218+00 | 2026-10-17 09:05:28.124455+00 | ok + TASR | t | t | 2026-10-17 03:41:52.892662+00 | 2026-10-17 09:03:56.13506+00 | ok + TATR | t | t | 2026-10-17 02:04:59.50841+00 | 2026-10-17 09:05:28.026657+00 | ok + TAYR | t | t | 2026-10-17 03:25:35.862846+00 | 2026-10-17 09:03:26.114213+00 | ok + TESR | t | t | 2026-10-17 02:05:06.685639+00 | 2026-10-17 09:04:46.489335+00 | ok + THOM | t | t | 2026-10-17 02:19:16.216539+00 | 2026-10-17 07:11:59.571567+00 | ok + TOAD | t | t | 2026-10-17 02:47:05.21288+00 | 2026-10-17 09:05:25.997944+00 | ok + TOBA | t | t | 2026-10-17 02:11:36.185963+00 | 2026-10-17 09:10:16.298484+00 | ok + TOOD | t | t | 2026-10-17 04:21:00.459917+00 | 2026-10-17 07:36:02.871198+00 | ok + TSAY | t | t | 2026-10-17 02:08:29.073556+00 | 2026-10-17 09:08:33.354959+00 | ok + TSEA | t | t | 2026-10-17 02:47:35.745665+00 | 2026-10-17 09:30:21.877721+00 | ok + TSIT | t | t | 2026-10-17 01:30:00.891786+00 | 2026-10-17 09:25:45.383734+00 | ok + TURN | t | t | 2026-10-17 03:53:08.071432+00 | 2026-10-17 09:06:48.174624+00 | ok + TUTR | t | t | 2026-10-17 02:46:20.199621+00 | 2026-10-17 09:06:58.706178+00 | ok + TUYR | t | t | 2026-10-17 03:41:40.121382+00 | 2026-10-17 09:07:45.047088+00 | ok + TWAC | t | t | 2026-10-17 03:21:35.713864+00 | 2026-10-17 09:20:06.008021+00 | ok + UARL | t | t | 2026-10-17 02:51:47.140795+00 | 2026-10-17 09:09:07.828536+00 | ok + UBIR | t | t | 2026-10-17 03:28:20.843507+00 | 2026-10-17 09:09:23.815202+00 | ok + UBTN | t | t | 2026-10-17 03:13:17.366897+00 | 2026-10-17 09:09:35.122935+00 | ok + UCHR | t | t | 2026-10-17 03:30:45.304028+00 | 2026-10-17 09:10:25.525071+00 | ok + UDEN | t | t | 2026-10-17 02:57:58.625152+00 | 2026-10-17 09:12:04.810364+00 | ok + UEUT | t | t | 2026-10-17 04:43:22.358757+00 | 2026-10-17 09:10:24.46173+00 | ok + UFRA | t | t | 2026-10-17 04:26:34.064004+00 | 2026-10-17 07:25:34.622154+00 | ok + UFRT | t | t | 2026-10-17 02:56:00.928626+00 | 2026-10-17 09:10:49.063346+00 | ok + UHAF | t | t | 2026-10-17 03:32:47.79029+00 | 2026-10-17 09:12:17.798779+00 | ok + UISR | t | t | 2026-10-17 04:39:30.416875+00 | 2026-10-17 09:11:05.791456+00 | ok + UJER | t | t | 2026-10-17 03:15:01.704913+00 | 2026-10-17 09:11:42.778244+00 | ok + UKEC | t | t | 2026-10-17 03:35:32.088086+00 | 2026-10-17 09:12:28.219907+00 | ok + ULRD | t | t | 2026-10-17 03:09:53.232314+00 | 2026-10-17 09:13:14.476467+00 | ok + UMUS | t | t | 2026-10-17 03:23:56.335677+00 | 2026-10-17 09:14:19.035455+00 | ok + UNAR | t | t | 2026-10-17 03:19:36.199039+00 | 2026-10-17 09:14:38.651911+00 | ok + UNRS | t | t | 2026-10-17 04:45:16.579601+00 | 2026-10-17 09:28:54.083797+00 | ok + UNTH | t | t | 2026-10-17 03:05:37.76838+00 | 2026-10-17 07:20:55.084525+00 | ok + UNUR | t | t | 2026-10-17 02:13:39.010253+00 | 2026-10-17 09:13:33.6837+00 | ok + UOMI | t | t | 2026-10-17 04:24:07.990639+00 | 2026-10-17 09:28:04.464154+00 | ok + UPCE | t | t | 2026-10-17 02:50:01.388704+00 | 2026-10-17 09:29:43.419025+00 | ok + UPET | t | t | 2026-10-17 06:48:58.764204+00 | 2026-10-17 10:59:45.189666+00 | ok + UPRO | t | t | 2026-10-17 03:39:30.32655+00 | 2026-10-17 09:15:46.394846+00 | ok + USHU | t | t | 2026-10-17 03:16:52.432544+00 | 2026-10-17 09:19:57.400379+00 | ok + USIK | t | t | 2026-10-17 03:27:50.663859+00 | 2026-10-17 09:16:08.971206+00 | ok + USKE | t | t | 2026-10-17 05:25:48.083238+00 | 2026-10-17 07:29:59.927223+00 | ok + USTK | t | t | 2026-10-17 04:44:41.097333+00 | 2026-10-17 09:16:08.822094+00 | ok + UTRE | t | t | 2026-10-17 04:40:23.160685+00 | 2026-10-17 09:16:34.291694+00 | ok + VICT | t | t | 2026-10-17 01:31:53.656277+00 | 2026-10-17 09:20:19.516231+00 | ok + WILL | t | t | 2026-10-17 04:07:45.129835+00 | 2026-10-17 07:33:03.242048+00 | ok + WORC | t | t | 2026-10-17 02:12:31.382417+00 | 2026-10-17 09:18:57.052424+00 | ok + ZYMO | t | t | 2026-10-17 03:46:06.261132+00 | 2026-10-17 07:29:19.20508+00 | ok +(241 rows) + + +=== 5. Whole-schema picture: the mixture this run leaves behind === + wsg_in_streams +---------------- + 241 +(1 row) + + wsg_with_any_log +------------------ + 241 +(1 row) + + runs_ever_labelled +-------------------- + 1 +(1 row) + + +=== 6. ASSERTIONS (raise, so this script can actually fail) === +psql:data-raw/study_area_verify.sql:522: NOTICE: PASS: run 20261017_004619-3f4c713b -- 241 WSG(s), all modelled, recomputed, segmented and provenanced +DO + +=== verify: OK === diff --git a/data-raw/logs/consolidate_history_354/verify_extra_after_restore.out b/data-raw/logs/consolidate_history_354/verify_extra_after_restore.out new file mode 100644 index 00000000..5d0ab7d6 --- /dev/null +++ b/data-raw/logs/consolidate_history_354/verify_extra_after_restore.out @@ -0,0 +1,296 @@ + set_config | set_config | set_config +---------------+--------------------------+------------ + fresh_default | 20261017_004619-3f4c713b | 241 +(1 row) + +psql:data-raw/logs/provincial_246/verify_extra.sql:114: NOTICE: verify_extra PASS: 241 groups, config_hash sha256:7e9dffa4a5b84d99c30928ecaf5eda30e7ccf9342f1fc53a2d9a3ce2840eb66b, run_uid 20261017_004619-3f4c713b +DO + host | table_name | artifact_recorded_by | artifact_version_id | n_rows | row_count_min | row_count_max +-------------+----------------------------------------------------+----------------------+---------------------+--------+---------------+--------------- + cypher-job1 | bcfishobs.observations | loader | 1790057230537 | 49 | 375962 | 375962 + cypher-job2 | bcfishobs.observations | loader | 1790057230537 | 55 | 375962 | 375962 + cypher-job3 | bcfishobs.observations | loader | 1790057230537 | 59 | 375962 | 375962 + m1 | bcfishobs.observations | loader | 1790057230537 | 78 | 375962 | 375962 + cypher-job1 | cabd.dams | loader | | 49 | 2592 | 2592 + cypher-job2 | cabd.dams | loader | | 55 | 2592 | 2592 + cypher-job3 | cabd.dams | loader | | 59 | 2592 | 2592 + m1 | cabd.dams | loader | | 78 | 2592 | 2592 + cypher-job1 | fresh.modelled_stream_crossings | loader | 1791925276485 | 49 | 524578 | 524578 + cypher-job2 | fresh.modelled_stream_crossings | loader | 1791925276485 | 55 | 524578 | 524578 + cypher-job3 | fresh.modelled_stream_crossings | loader | 1791925276485 | 59 | 524578 | 524578 + m1 | fresh.modelled_stream_crossings | loader | 1791925276485 | 78 | 524578 | 524578 + cypher-job1 | whse_basemapping.fwa_lakes_poly | probe | 1776306159523 | 49 | 386025 | 386025 + cypher-job2 | whse_basemapping.fwa_lakes_poly | probe | 1776306159523 | 55 | 386025 | 386025 + cypher-job3 | whse_basemapping.fwa_lakes_poly | probe | 1776306159523 | 59 | 386025 | 386025 + m1 | whse_basemapping.fwa_lakes_poly | probe | 1776306159523 | 78 | 386429 | 386429 + cypher-job1 | whse_basemapping.fwa_obstructions_sp | probe | 1776306166790 | 49 | 32541 | 32541 + cypher-job2 | whse_basemapping.fwa_obstructions_sp | probe | 1776306166790 | 55 | 32541 | 32541 + cypher-job3 | whse_basemapping.fwa_obstructions_sp | probe | 1776306166790 | 59 | 32541 | 32541 + m1 | whse_basemapping.fwa_obstructions_sp | probe | 1776306166790 | 78 | 32541 | 32541 + cypher-job1 | whse_basemapping.fwa_rivers_poly | probe | 1776306167722 | 49 | 37083 | 37083 + cypher-job2 | whse_basemapping.fwa_rivers_poly | probe | 1776306167722 | 55 | 37083 | 37083 + cypher-job3 | whse_basemapping.fwa_rivers_poly | probe | 1776306167722 | 59 | 37083 | 37083 + m1 | whse_basemapping.fwa_rivers_poly | probe | 1776306167722 | 78 | 37083 | 37083 + cypher-job1 | whse_basemapping.fwa_stream_networks_channel_width | probe | 1768525131162 | 49 | 4907429 | 4907429 + cypher-job2 | whse_basemapping.fwa_stream_networks_channel_width | probe | 1768525131162 | 55 | 4907429 | 4907429 + cypher-job3 | whse_basemapping.fwa_stream_networks_channel_width | probe | 1768525131162 | 59 | 4907429 | 4907429 + m1 | whse_basemapping.fwa_stream_networks_channel_width | probe | 1768525131162 | 78 | 4907429 | 4907429 + cypher-job1 | whse_basemapping.fwa_stream_networks_discharge | probe | 1768521748856 | 49 | 2716652 | 2716652 + cypher-job2 | whse_basemapping.fwa_stream_networks_discharge | probe | 1768521748856 | 55 | 2716652 | 2716652 + cypher-job3 | whse_basemapping.fwa_stream_networks_discharge | probe | 1768521748856 | 59 | 2716652 | 2716652 + m1 | whse_basemapping.fwa_stream_networks_discharge | probe | 1768521748856 | 78 | 2716652 | 2716652 + cypher-job1 | whse_basemapping.fwa_stream_networks_order_parent | probe | | 49 | 1509493 | 1509493 + cypher-job2 | whse_basemapping.fwa_stream_networks_order_parent | probe | | 55 | 1509493 | 1509493 + cypher-job3 | whse_basemapping.fwa_stream_networks_order_parent | probe | | 59 | 1509493 | 1509493 + m1 | whse_basemapping.fwa_stream_networks_order_parent | probe | | 78 | 1509493 | 1509493 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306184475 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177285 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177596 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179947 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306170103 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306187600 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306185980 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306186396 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177402 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306172569 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306172032 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306168809 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183555 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306167832 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306176850 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306170640 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306171733 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179596 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306178186 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306176809 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306170200 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306180994 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306171929 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306168585 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306173935 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306174075 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306176688 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306169892 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306176245 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306172679 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306182743 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306182282 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179319 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306178821 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306170463 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306187062 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306173090 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306184019 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306187086 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306182180 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306186111 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306184312 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179866 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306185580 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306171206 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306180497 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306178255 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306168453 | 1 | 4898813 | 4898813 + cypher-job1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306180466 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183356 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306181382 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306169047 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306173412 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306171893 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306172438 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179365 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306169452 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306168002 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306181031 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306185971 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306174741 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306186642 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306181066 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306180732 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306180541 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306184012 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306184330 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306175276 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306184771 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306186620 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306171125 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177577 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306178157 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306175881 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183162 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306172578 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179880 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306184466 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177536 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306180899 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306184810 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306178380 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179701 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306175665 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306170930 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306180243 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306173125 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306186377 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306172686 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306180416 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306172096 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306185788 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183439 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306170325 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306171134 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306171323 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306176548 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306180206 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306178552 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177020 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306175260 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306173637 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306182481 | 1 | 4898813 | 4898813 + cypher-job2 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306186440 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306185454 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306175287 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306169901 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306175288 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177028 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306178914 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183149 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183140 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306174430 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179311 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306168424 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306185899 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306182693 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306174550 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306184189 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306173106 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306182601 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306169024 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306175236 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177842 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306169364 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306170167 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306169824 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306187215 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306181028 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306167884 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179798 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306175853 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183021 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306176453 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306174197 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177964 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306168413 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177361 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183031 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306173043 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306178216 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179739 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306174807 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306170925 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306181215 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306185410 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183069 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179016 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306170624 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306173905 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306187758 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306185464 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306176482 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306169366 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306173689 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306173334 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306169986 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306172927 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306168472 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177780 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177207 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306181749 | 1 | 4898813 | 4898813 + cypher-job3 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306187195 | 1 | 4898813 | 4898813 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306185085 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179992 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306182145 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306176955 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306168456 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306170705 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306170155 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306186684 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306169062 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306187027 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306171914 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306168534 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306169197 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306168819 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306169234 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306175768 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183056 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306169361 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177440 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177837 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177139 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179146 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306181948 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306182022 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306181439 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183427 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306170364 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179021 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306182077 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306180315 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306184644 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306186356 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183078 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183379 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306174275 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306182915 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306172010 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306178063 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306168700 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306168555 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306171382 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306170120 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179387 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306168874 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183317 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306170611 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306186567 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306169879 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306187330 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306187012 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306181302 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179699 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306168310 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306170242 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306175662 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306176503 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306172448 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183677 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306171695 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306181463 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306169046 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306178006 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306174977 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177353 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306183222 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306175977 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306184450 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306175983 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306180103 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306179983 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306177729 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306167950 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306185615 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306185301 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306171916 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306178029 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306187080 | 1 | 4907804 | 4907804 + m1 | whse_basemapping.fwa_stream_networks_sp | probe | 1776306172205 | 1 | 4907804 | 4907804 + cypher-job1 | whse_basemapping.fwa_wetlands_poly | probe | 1776306203731 | 49 | 375178 | 375178 + cypher-job2 | whse_basemapping.fwa_wetlands_poly | probe | 1776306203731 | 55 | 375178 | 375178 + cypher-job3 | whse_basemapping.fwa_wetlands_poly | probe | 1776306203731 | 59 | 375178 | 375178 + m1 | whse_basemapping.fwa_wetlands_poly | probe | 1776306203731 | 78 | 374560 | 374560 + cypher-job1 | whse_fish.pscis_assessment_svw | loader | | 49 | 19905 | 19905 + cypher-job2 | whse_fish.pscis_assessment_svw | loader | | 55 | 19905 | 19905 + cypher-job3 | whse_fish.pscis_assessment_svw | loader | | 59 | 19905 | 19905 + m1 | whse_fish.pscis_assessment_svw | loader | | 78 | 19905 | 19905 +(285 rows) + diff --git a/planning/active/progress.md b/planning/active/progress.md index e1efc69e..aa435b98 100644 --- a/planning/active/progress.md +++ b/planning/active/progress.md @@ -6,3 +6,6 @@ - Created branch `354-schema-consolidate-deletes-the-destinati` off main - Scaffolded PWF baseline from issue #354 with approved phases - Next: start Phase 1 +- Phases 1–3 (`0a1f8534`, `79a247e0`): `.lnk_log_specs()`; consolidate DELETEs persist tables only and replaces log rows by primary key in one transaction per table. Correction to the plan: full PK, not a `run_key` (findings). Restore-the-bug: both mutations go red. +- Phase 4 (`7d85b244`): RUNBOOK. NEWS left to `/gh-pr-merge` (this repo writes NEWS in the release commit). +- Phase 5: restored 5 `log` + 60 `log_input` rows into `fresh_default` from the 22:42 pre-consolidate dump (PINE was a fifth loss); both verify scripts pass at 241; `lnk_log_read()` newest is this run for all 241. diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md index c9e20a6f..ebcd4e66 100644 --- a/planning/active/task_plan.md +++ b/planning/active/task_plan.md @@ -33,13 +33,13 @@ - [ ] NEWS entry — deferred to `/gh-pr-merge`: in this repo NEWS is written by the release commit (`bf1521d3`), not on the branch ## Phase 5: Restore the lost rows (operator: yes) -- [ ] Extract `log` + `log_input` rows for DUNC, KOTL, LARL, SLOC, PINE from the 22:42 dump +- [x] Extract `log` + `log_input` rows for DUNC, KOTL, LARL, SLOC, PINE from the 22:42 dump into a scratch schema (`pg_restore --data-only -t` into a copy), INSERT … ON CONFLICT DO NOTHING into `fresh_default`; counts before/after recorded in `data-raw/logs/consolidate_history_354/` -- [ ] Re-run `study_area_verify.sql -v run_uid=20261017_004619-3f4c713b -v expected_n=241` +- [x] Re-run `study_area_verify.sql -v run_uid=20261017_004619-3f4c713b -v expected_n=241` and `verify_extra.sql`; both pass; `lnk_log_read()` still returns this run's row per group -- [ ] CLAUDE.md #246 status: the four (five, with PINE) groups' earlier log rows restored +- [x] CLAUDE.md #246 status: the four (five, with PINE) groups' earlier log rows restored ## Validation From 72cd2ea5e95cba9dee6ae4dc145315395f604780 Mon Sep 17 00:00:00 2001 From: almac2022 Date: Sat, 17 Oct 2026 20:52:50 -0700 Subject: [PATCH 06/11] Refuse a source whose log is older than the destination's (#354) Plan-review fixes. With history kept, a group's newest log row is what names the build its persist rows hold, so consolidate now refuses a source (nothing deleted) when that row on the destination is newer than the source's, or the source has none. The bucket DELETE list and the stage-2 command move into tested helpers; psql gets -X; dropping -1 is measured to fail closed. NULLS LAST in two diagnostic log readers, and the stale consolidate docs and banners are corrected. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx --- R/lnk_log.R | 5 +- RUNBOOK.md | 8 +- data-raw/README.md | 2 +- data-raw/habitat_diff_segments.R | 2 +- .../logs/consolidate_history_354/README.md | 6 +- .../restore_log_rows.sql | 9 +- data-raw/logs/dunc_default_328/README.md | 4 +- data-raw/query_habitat_thresholds_fiss.R | 2 +- data-raw/schema_consolidate.R | 119 +++++++++++++++--- data-raw/study_area_hosts.sh | 5 +- planning/active/progress.md | 2 + planning/active/review-1.md | 17 +++ planning/active/task_plan.md | 7 +- tests/testthat/test-schema_consolidate.R | 70 +++++++++-- 14 files changed, 219 insertions(+), 39 deletions(-) create mode 100644 planning/active/review-1.md diff --git a/R/lnk_log.R b/R/lnk_log.R index 6534cfee..f44f83de 100644 --- a/R/lnk_log.R +++ b/R/lnk_log.R @@ -500,8 +500,9 @@ cols_log_input <- c( #' group copied home from another host keeps the rows of its earlier builds #' (link#354). #' -#' Read by [.lnk_log_create_tables()] and by `schema_consolidate.R`, so the set -#' of history tables is named here and nowhere else. +#' Read by [.lnk_log_create_tables()] and by `schema_consolidate.R`, so the two +#' agree on which tables are history. (`lnk_persist_init()`'s drift check keeps +#' its own list of log tables.) #' #' @return A list of specs named by table. #' @noRd diff --git a/RUNBOOK.md b/RUNBOOK.md index 44885e77..48f8995f 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -1274,7 +1274,13 @@ then the finished work still on the host. `unsettled` means wait until nothing runs there and re-measure. `schema_consolidate` deletes its bucket on the dispatcher before it copies, so consolidating a group the host no longer holds (already consolidated, then deleted from the source), or holds only in part, loses the -dispatcher's copy of its persist rows. Its log rows survive (link#354). Where it says `no`, find out what is left first. If the +dispatcher's copy of its persist rows. Its log rows survive (link#354), and +consolidate refuses a source outright (`stage = "dest_log_newer"`, nothing +deleted) when a group's newest log row on the dispatcher is newer than the +source's, or the source has none: the persist tables carry no run id, so that +newest row is what says which build they hold. A group re-run on the +dispatcher during recovery is therefore never overwritten by the kept host's +older build. Where it says `no`, find out what is left first. If the run's recompute ran, it ran without the kept groups: recompute each one and each run group upstream of it (`wsg_recompute_one.R`). If it did not run (consolidate failed, coverage FATAL), recompute every group of the run; the diff --git a/data-raw/README.md b/data-raw/README.md index acf1427c..be4b8d79 100644 --- a/data-raw/README.md +++ b/data-raw/README.md @@ -90,7 +90,7 @@ Run-adjacent helpers (planning, consolidation across hosts). | Script | Purpose | |--------|---------| | `buckets_balance.R` | Standalone LPT planner for the 3-host case. Reads per-host wall times from prior runs and prints buckets ready to paste into `wsgs_dispatch.sh --m4-bucket=…`. **Superseded for the N-host orchestrator** — `wsgs_dispatch.sh` now computes the LPT plan inline at dispatch time using the same algorithm. Kept here for one-off planning + cross-checks. Dedups `(wsg, host)` and across hosts before LPT so multi-run CSV accumulation doesn't double-assign WSGs. | -| `schema_consolidate.R` | pg_dump from M1 + cypher → scp to M4 → pg_restore --data-only. Bucket-aware destination cleanup (DELETEs each source host's WSG bucket from destination tables before restore — avoids duplicate-key violations on re-consolidation). `ok = TRUE` requires pg_restore rc=0 AND post-restore row count > 0; rc=0 with empty schema flags as failure. | +| `schema_consolidate.R` | Per-table, bucket-filtered `COPY` from each source host over ssh into the dispatcher. Deletes each source's bucket from the destination's persist tables before copying; the log tables are history, so their rows are replaced by primary key and never deleted by group, and a source is refused if a group's destination log is newer than its own (#354). `ok = TRUE` requires every table's COPY to succeed and the row count to rise. | | `runs_archive.sh` | Moves the current top-level `_per_wsg_times.csv` + `*.rds` + `*_annotated.csv` artifacts in `provincial_/` to `archive//`. Operator cadence: run between provincial runs when you want the LPT planner to use the most recent run only. Skip to median-over multiple recent runs. | | `trifecta_smoke.sh` | Thin shim over `wsgs_dispatch.sh` — one small WSG per host (m4→DEAD, m1→ELKR, cyN→ADMS/BABL/BULL). ~3 min wall. Exercises every orchestrator code path (preflight, dispatch, tunnel, RDS pull-back, annotation) before committing to a 200-WSG run. All flags pass through (e.g. `--cy-workspaces=`, `--mapping-code`). | diff --git a/data-raw/habitat_diff_segments.R b/data-raw/habitat_diff_segments.R index d0e64855..16e4a119 100644 --- a/data-raw/habitat_diff_segments.R +++ b/data-raw/habitat_diff_segments.R @@ -161,7 +161,7 @@ git_sha <- tryCatch(system2("git", c("rev-parse", "--short", "HEAD"), stdout = T error = function(e) NA_character_) logrow <- q(sprintf("SELECT run_id, date_end, link_version, link_sha, fresh_version, config_name FROM %s.log WHERE watershed_group_code = %s - ORDER BY date_end DESC LIMIT 1", schema, wsg_lit)) + ORDER BY date_end DESC NULLS LAST LIMIT 1", schema, wsg_lit)) view_n <- q("SELECT count(*) AS n FROM fresh.streams_vw_bcfp")$n writeLines(c( sprintf("run: %s (UTC)", format(Sys.time(), "%Y-%m-%dT%H:%MZ", tz = "UTC")), diff --git a/data-raw/logs/consolidate_history_354/README.md b/data-raw/logs/consolidate_history_354/README.md index 4040deb3..459fef5d 100644 --- a/data-raw/logs/consolidate_history_354/README.md +++ b/data-raw/logs/consolidate_history_354/README.md @@ -6,7 +6,7 @@ Run `20261017_004619-3f4c713b`'s consolidate deleted each cloud-built group's ea `log_input` rows on the dispatcher. The driver's pre-consolidate backup (`/tmp/fresh_default_pre_consolidate_202610162242.dump`, machine-local, taken after m1's own groups and before the cloud hosts' were copied in) still held them. Against the current schema -it had exactly **5 `log` rows and 60 `log_input` rows** whose primary key was gone, and an +(an anti-join on each table's primary key) it had exactly **5 `log` rows and 60 `log_input` rows** whose primary key was gone, and an empty `log_recompute`: | group | run_id | built | link | @@ -18,7 +18,9 @@ empty `log_recompute`: | SLOC | `m1-20260827T145652.093-c69f2d` | 2026-08-27 | 0.45.1 | PINE was not named in the issue; it was built on `cypher-job2` in the run and lost its August -row the same way. PARS had no earlier row. +row the same way. PARS had no earlier row. The later backup (`…_202610162328.dump`, taken +before the hand consolidate of the 18 groups) has nothing missing in any of the three tables, +so the 22:42 dump alone was complete. - `restore_log_rows.sql` → `restore_log_rows.out`: additive `INSERT … ON CONFLICT DO NOTHING`, `log` 241 → 246, `log_input` 2,892 → 2,952, 12 `log_input` rows per restored build. diff --git a/data-raw/logs/consolidate_history_354/restore_log_rows.sql b/data-raw/logs/consolidate_history_354/restore_log_rows.sql index cbe8df48..f0fa6af6 100644 --- a/data-raw/logs/consolidate_history_354/restore_log_rows.sql +++ b/data-raw/logs/consolidate_history_354/restore_log_rows.sql @@ -1,11 +1,18 @@ -- Restore the log rows run_uid 20261017_004619-3f4c713b's consolidate deleted -- from fresh_default (link#354). Source: the driver's pre-consolidate backup -- /tmp/fresh_default_pre_consolidate_202610162242.dump (machine-local), its --- log / log_input / log_recompute data loaded into zz354_restore first: +-- log / log_input / log_recompute data loaded into zz354_restore first. The +-- scratch tables are `LIKE fresh_default.`, so the `SELECT *` below is +-- positional over identical column lists: -- +-- CREATE TABLE zz354_restore. (LIKE fresh_default.); -- pg_restore --data-only -t -f .sql -- sed 's/^COPY fresh_default\./COPY zz354_restore./' .sql | psql -- +-- Before it ran, every one of the 5 log and 60 log_input rows was absent from +-- fresh_default by primary key (an anti-join, README), so ON CONFLICT skipped +-- nothing: the INSERT counts below are those numbers. +-- -- Additive: a row whose primary key is present is left alone, so re-running -- this changes nothing. Columns added since the dump take NULL. \set ON_ERROR_STOP on diff --git a/data-raw/logs/dunc_default_328/README.md b/data-raw/logs/dunc_default_328/README.md index 469f52c4..9c8306f6 100644 --- a/data-raw/logs/dunc_default_328/README.md +++ b/data-raw/logs/dunc_default_328/README.md @@ -22,7 +22,9 @@ LNK_LOAD=loadall LNK_SCHEMA=fresh_default LNK_RUN_LABEL=dunc_default_328 \ ## Results -All 19 checks pass. +All 19 checks pass (2026-10-12). `verify.R` is a dated record, not a regression test: after +#246 rebuilt DUNC its `link_sha` check fails, and since #354 restored DUNC's 2026-10-12 log row +beside #246's, its `exactly one fresh_default.log row` check fails too (2 rows). - **Coverage.** `fresh_default.streams` holds 63,067 DUNC segments. Each of `streams_habitat_bt` / `_wct` / `_rb`, `streams_access` and `streams_mapping_code` has one row per segment, on the full PK. - **Network.** Order ≥ 2 is 3,493.0 km, the same as raw FWA and `fresh.streams`. `fresh` has 26,827 segments, from its pre-#223 build in the 2026-05-13 trifecta provincial run (link 0.36.0, `data-raw/logs/202605132248_trifecta_provincial_m1.txt`). - **Log row.** `fresh_default.log` records config `default`, the config hash, link / fresh versions and SHAs, `fwapg_sha`, `schema_persist` and `date_end`. diff --git a/data-raw/query_habitat_thresholds_fiss.R b/data-raw/query_habitat_thresholds_fiss.R index e73de5e0..b0abf7e6 100644 --- a/data-raw/query_habitat_thresholds_fiss.R +++ b/data-raw/query_habitat_thresholds_fiss.R @@ -223,7 +223,7 @@ vintage <- dbGetQuery(conn, sprintf( FROM (SELECT DISTINCT ON (watershed_group_code) watershed_group_code, date_end, fresh_version, fresh_sha FROM %s.log WHERE watershed_group_code = ANY($1) - ORDER BY watershed_group_code, date_end DESC) x", schema), + ORDER BY watershed_group_code, date_end DESC NULLS LAST) x", schema), params = list(paste0("{", paste(toupper(wsgs), collapse = ","), "}")))[[1]] dbDisconnect(conn) writeLines(c( diff --git a/data-raw/schema_consolidate.R b/data-raw/schema_consolidate.R index 5dbf6f37..82702b08 100644 --- a/data-raw/schema_consolidate.R +++ b/data-raw/schema_consolidate.R @@ -93,6 +93,67 @@ consolidate_history_sql <- function(schema, table, cols, pk, path) { ) } +#' DELETE statements that clear a bucket from the destination's persist tables. +#' +#' Log tables are never in the list (link#354): their rows are history. +#' +#' @param schema Character. Destination schema. +#' @param wgc_tables Character. Tables discovered on both sides. +#' @param bucket Character. The source's watershed groups. +#' @return Character vector, one DELETE per persist table. +consolidate_dest_delete_sql <- function(schema, wgc_tables, bucket) { + persist <- consolidate_tables_split(wgc_tables)$persist + wsg_list_sql <- paste0("'", bucket, "'", collapse = ", ") + sprintf("DELETE FROM %s.%s WHERE watershed_group_code IN (%s)", + schema, persist, wsg_list_sql) +} + +#' Shell command that loads one table's COPY file into the destination. +#' +#' A persist table is a plain `COPY FROM STDIN`. A log table writes +#' [consolidate_history_sql()] to `sql_path` and runs it with `psql -1`, one +#' transaction (link#354). Dropping `-1` fails closed rather than open: under +#' autocommit `ON COMMIT DROP` removes the temp table before `\copy` runs. +#' `-X` keeps a `~/.psqlrc` out of both. +#' +#' @param schema,table Destination table. +#' @param cols Character. The shared columns, in the order of `tsv`'s fields. +#' @param tsv Path of the COPY text-format file. +#' @param sql_path Path the log-table script is written to. +#' @param specs The log table specs, named by table. +#' @return The command, a single string. +consolidate_stage2_cmd <- function(schema, table, cols, tsv, sql_path, + specs = link:::.lnk_log_specs()) { + psql <- "PGHOST=localhost PGPORT=5432 PGDATABASE=fwapg PGUSER=postgres PGPASSWORD=postgres psql -X -v ON_ERROR_STOP=1" + if (table %in% names(specs)) { + writeLines(consolidate_history_sql(schema, table, cols, + specs[[table]]$pk, tsv), sql_path) + return(sprintf("%s -1 -f %s", psql, shQuote(sql_path))) + } + sprintf("%s -c \"COPY %s.%s (%s) FROM STDIN\" < %s", psql, schema, table, + paste(cols, collapse = ", "), shQuote(tsv)) +} + +#' Groups whose newest log row on the destination is not the source's. +#' +#' The persist tables carry no run id: a group's newest `log` row is what says +#' which build they hold. Consolidating a group whose destination log is newer +#' than the source's (or whose source has no log row) would put an older +#' build in the persist tables under a log that names a newer one, so the +#' source is refused (link#354). Before #354 the bucket DELETE removed the +#' destination's log rows, which hid the mismatch. +#' +#' @param dest,src Data frames of `watershed_group_code` and `newest` (epoch +#' seconds of the group's latest `date_start`), one row per group with a +#' log row on that side. +#' @return Character vector of groups, sorted. +consolidate_newer_on_dest <- function(dest, src) { + m <- merge(dest, src, by = "watershed_group_code", all.x = TRUE, + suffixes = c("_dest", "_src")) + sort(m$watershed_group_code[is.na(m$newest_src) | + m$newest_dest > m$newest_src]) +} + #' Consolidate a Postgres schema from N remote hosts onto local fwapg. #' #' @param schema Character. Schema to consolidate (must already exist on @@ -340,6 +401,39 @@ schema_consolidate <- function(schema, # their rows are history, replaced by primary key in step 4 (link#354). split_tables <- consolidate_tables_split(wgc_tables) log_specs <- link:::.lnk_log_specs() + # Checked before anything is deleted: never overwrite a group's persist + # rows with a build older than the one its log already names. + if ("log" %in% split_tables$history) { + newest_sql <- sprintf(paste( + "SELECT watershed_group_code, extract(epoch FROM max(date_start))", + "AS newest FROM %s.log WHERE watershed_group_code IN (%s) GROUP BY 1"), + schema, wsg_list_sql) + dest_newest <- DBI::dbGetQuery(dest_conn, newest_sql) + src_newest_inner <- if (via == "docker") { + sprintf("docker exec %s psql -X -U %s -d %s -t -A -c \"%s\"", + container, pg_user, pg_db, newest_sql) + } else { + sprintf("PGHOST=localhost PGPORT=5432 PGDATABASE=%s PGUSER=%s PGPASSWORD=postgres psql -X -t -A -c \"%s\"", + pg_db, pg_user, newest_sql) + } + src_newest_raw <- system(sprintf("ssh '%s' '%s'", src$host, + gsub("'", "'\\''", src_newest_inner, fixed = TRUE)), intern = TRUE) + src_newest_raw <- strsplit(src_newest_raw[nzchar(src_newest_raw)], "|", + fixed = TRUE) + src_newest <- data.frame( + watershed_group_code = vapply(src_newest_raw, `[`, character(1), 1L), + newest = as.numeric(vapply(src_newest_raw, `[`, character(1), 2L))) + dest_newest$newest <- as.numeric(dest_newest$newest) + newer <- consolidate_newer_on_dest(dest_newest, src_newest) + if (length(newer) > 0L) { + log(src$host, " -> ERROR: the destination's newest log row is not ", + "this source's for ", paste(newer, collapse = ", "), + " (newer here, or none on the source) — nothing deleted or copied") + results[[src$host]] <- list(ok = FALSE, stage = "dest_log_newer", + rc = -1L, groups = newer) + next + } + } # Checked before anything is deleted: a log table whose key is not shared # cannot be replaced by key, and failing at its COPY would leave this # bucket's persist rows already deleted here. @@ -355,10 +449,8 @@ schema_consolidate <- function(schema, log(src$host, " -> DELETE bucket (", length(src$bucket), " WSGs) from ", length(split_tables$persist), " tables; ", length(split_tables$history), " log tables keep their history") - for (t in split_tables$persist) { - DBI::dbExecute(dest_conn, sprintf( - "DELETE FROM %s.%s WHERE watershed_group_code IN (%s)", - schema, t, wsg_list_sql)) + for (del in consolidate_dest_delete_sql(schema, wgc_tables, src$bucket)) { + DBI::dbExecute(dest_conn, del) } # 3.6. Snapshot pre-restore total row count across the schema. The @@ -436,21 +528,10 @@ schema_consolidate <- function(schema, # is by-name (shape-tolerant), not positional. A log table goes # through a one-transaction staged replace by primary key instead, so # its earlier rows for the bucket survive (link#354). - if (t %in% split_tables$history) { - sqlf <- tempfile(pattern = sprintf("consolidate_%s_", t), - fileext = ".sql") - writeLines(consolidate_history_sql(schema, t, shared_cols[[t]], - log_specs[[t]]$pk, tmpf), sqlf) - stage2 <- sprintf( - "PGHOST=localhost PGPORT=5432 PGDATABASE=fwapg PGUSER=postgres PGPASSWORD=postgres psql -1 -v ON_ERROR_STOP=1 -f %s", - shQuote(sqlf)) - } else { - sqlf <- NULL - dest_sql <- sprintf("COPY %s.%s (%s) FROM STDIN", schema, t, col_list) - stage2 <- sprintf( - "PGHOST=localhost PGPORT=5432 PGDATABASE=fwapg PGUSER=postgres PGPASSWORD=postgres psql -v ON_ERROR_STOP=1 -c \"%s\" < %s", - dest_sql, shQuote(tmpf)) - } + sqlf <- tempfile(pattern = sprintf("consolidate_%s_", t), + fileext = ".sql") + stage2 <- consolidate_stage2_cmd(schema, t, shared_cols[[t]], tmpf, sqlf, + log_specs) rc_dest <- system(stage2) unlink(c(tmpf, sqlf)) if (rc_dest != 0L) { diff --git a/data-raw/study_area_hosts.sh b/data-raw/study_area_hosts.sh index 74d9bedd..bb085433 100644 --- a/data-raw/study_area_hosts.sh +++ b/data-raw/study_area_hosts.sh @@ -466,7 +466,7 @@ burn_cyphers() { # group it was asked to run, finished, or logged done. That matters for # recovery: schema_consolidate DELETEs the bucket on the dispatcher before it # copies, so consolidating a group the host no longer holds destroys the -# dispatcher's copy. Where the host does not answer, `groups` is the +# dispatcher's copy of its persist rows (its log rows survive, #354). Where the host does not answer, `groups` is the # candidate set and `groups_measured` says no. # # No IP in the file: redact_log_addresses rewrites every ${TS}_* file, so it @@ -521,7 +521,8 @@ hosts_kept_report() { echo " schema_consolidate on this machine) and re-measure; where it says" echo " no, find out what is left first. Never groups_partial. schema_consolidate" echo " DELETEs its bucket on the dispatcher before it copies, so a group the" - echo " host no longer holds, or holds only in part, loses the dispatcher's copy." + echo " host no longer holds, or holds only in part, loses the dispatcher's copy" + echo " of its persist rows (its log rows survive)." if [ "${RECOMPUTE_RAN:-0}" = "1" ]; then echo " The run's recompute ran WITHOUT these groups: after consolidating," echo " recompute each one and each run group upstream of it" diff --git a/planning/active/progress.md b/planning/active/progress.md index aa435b98..ccc60c02 100644 --- a/planning/active/progress.md +++ b/planning/active/progress.md @@ -9,3 +9,5 @@ - Phases 1–3 (`0a1f8534`, `79a247e0`): `.lnk_log_specs()`; consolidate DELETEs persist tables only and replaces log rows by primary key in one transaction per table. Correction to the plan: full PK, not a `run_key` (findings). Restore-the-bug: both mutations go red. - Phase 4 (`7d85b244`): RUNBOOK. NEWS left to `/gh-pr-merge` (this repo writes NEWS in the release commit). - Phase 5: restored 5 `log` + 60 `log_input` rows into `fresh_default` from the 22:42 pre-consolidate dump (PINE was a fifth loss); both verify scripts pass at 241; `lnk_log_read()` newest is this run for all 241. +- Full suite (local): 3147 pass, 5 pre-existing failures (tunnel down ×3, ADMS dams pin ×2). The background wrapper reported done while R was still running; waited on the PID instead. +- Plan review arrived after Phase 5 (`review-1.md`); fixes landed as Phase 6. Mutations 3–5 (DELETE list unsplit, `-1` dropped, newer-guard off) all go red. diff --git a/planning/active/review-1.md b/planning/active/review-1.md new file mode 100644 index 00000000..d93ee4a7 --- /dev/null +++ b/planning/active/review-1.md @@ -0,0 +1,17 @@ +# Plan review 1 (Plan agent, 2026-10-17) — findings and dispositions + +Spawned after the baseline; it reported after Phases 1–5 had landed, so it reviewed code and plan. + +| # | Class | Finding | Disposition | +|---|---|---|---| +| 1 | Gap | With history kept, a dispatcher log row newer than the source's would name a build the persist tables no longer hold (persist carries no run id; readers take the newest row). | **Fixed.** `consolidate_newer_on_dest()`: a source is refused (`dest_log_newer`, nothing deleted) when any bucket group's newest destination log row is newer than the source's, or the source has none. RUNBOOK §8d. | +| 2 | Gap | The loop's use of the split and the `-1` flag were untested; reverting the loop to `wgc_tables` stayed green. | **Fixed.** `consolidate_dest_delete_sql()` and `consolidate_stage2_cmd()` carry both, tested; mutations go red. Dropping `-1` measured to fail closed (temp table gone before `\copy`), tested live and commented. | +| 3 | Gap | Tests run only under local `devtools::test()` (data-raw is build-ignored; no R-CMD-check workflow). | **Recorded** in Validation. | +| 4 | Gap | `ORDER BY date_end DESC` puts an open (NULL) row first in two diagnostic scripts once history keeps failed attempts. | **Fixed.** `NULLS LAST` in `habitat_diff_segments.R`, `query_habitat_thresholds_fiss.R`. | +| 5 | Gap | `.lnk_log_specs()` doc says "nowhere else", but `lnk_persist_init()` hard-codes a log list. | **Fixed (doc).** Softened; changing persist-init's drift check is outside #354. | +| – | Assumption | Stage 2 lacked `-X`. | **Fixed** for both branches. | +| – | Assumption | `restore_log_rows.sql` `SELECT *` is positional; record how scratch tables were made; state the rows were absent before. | **Fixed** in the script header and README. | +| – | Assumption | `dunc_default_328/verify.R` asserts one DUNC log row. | **Noted** in that README as a dated record. | +| – | Scope | `data-raw/README.md` described the pg_dump flow; `study_area_hosts.sh` banners. | **Fixed.** | +| – | Ordering | 23:28 dump checked too: nothing missing; the 22:42 restore was complete. | Recorded in the evidence README. | +| – | Blocker | `pg_restore -t` "into a copy" cannot remap a schema. | Already avoided by the landed route (sed on the COPY target). | diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md index ebcd4e66..c6a8375f 100644 --- a/planning/active/task_plan.md +++ b/planning/active/task_plan.md @@ -41,9 +41,14 @@ and `verify_extra.sql`; both pass; `lnk_log_read()` still returns this run's row per group - [x] CLAUDE.md #246 status: the four (five, with PINE) groups' earlier log rows restored +## Phase 6: Plan-review fixes (`review-1.md`) +- [x] Refuse a source whose group's destination log is newer (or absent on the source) +- [x] Helpers for the DELETE list and stage-2 command, tested; `-X`; `-1` fails closed, tested +- [x] `NULLS LAST` in two diagnostic readers; docstring; restore record; stale docs and banners + ## Validation -- [ ] Tests pass +- [x] Tests pass — local `devtools::test()` only (data-raw is build-ignored and there is no R-CMD-check workflow): 3147 pass; 5 fail, all pre-existing (3 need the `:63333` tunnel, 2 are the golden ADMS dams pin 8 → 7 in CLAUDE.md). Touched files re-run after review fixes: 0 fail - [ ] `/code-check` clean (each commit, or once over the branch with `/code-check branch`) - [ ] PWF checkboxes match landed work - [ ] `/planning-archive` on completion diff --git a/tests/testthat/test-schema_consolidate.R b/tests/testthat/test-schema_consolidate.R index b7776aa1..fe24dc31 100644 --- a/tests/testthat/test-schema_consolidate.R +++ b/tests/testthat/test-schema_consolidate.R @@ -35,11 +35,15 @@ test_that("consolidate_history_sql replaces by primary key, never by group", { sql <- e$consolidate_history_sql("s", "log_input", c("run_id", "watershed_group_code", "table_name", "row_count"), c("run_id", "table_name"), "/tmp/x.tsv") - joined <- paste(sql, collapse = "\n") - expect_match(joined, "CREATE TEMP TABLE consolidate_in ON COMMIT DROP AS SELECT run_id, watershed_group_code, table_name, row_count FROM s.log_input WITH NO DATA;", fixed = TRUE) - expect_match(joined, "\\copy consolidate_in (run_id, watershed_group_code, table_name, row_count) FROM '/tmp/x.tsv'", fixed = TRUE) - expect_match(joined, "DELETE FROM s.log_input d USING consolidate_in s WHERE (d.run_id, d.table_name) = (s.run_id, s.table_name);", fixed = TRUE) - expect_match(joined, "INSERT INTO s.log_input (run_id, watershed_group_code, table_name, row_count) SELECT run_id, watershed_group_code, table_name, row_count FROM consolidate_in;", fixed = TRUE) + cl <- "run_id, watershed_group_code, table_name, row_count" + expect_identical(sql, c( + paste0("CREATE TEMP TABLE consolidate_in ON COMMIT DROP AS SELECT ", cl, + " FROM s.log_input WITH NO DATA;"), + paste0("\\copy consolidate_in (", cl, ") FROM '/tmp/x.tsv'"), + paste0("DELETE FROM s.log_input d USING consolidate_in s", + " WHERE (d.run_id, d.table_name) = (s.run_id, s.table_name);"), + paste0("INSERT INTO s.log_input (", cl, ") SELECT ", cl, + " FROM consolidate_in;"))) delete <- grep("^DELETE", sql, value = TRUE) expect_no_match(delete, "watershed_group_code") }) @@ -55,6 +59,46 @@ test_that("consolidate_history_sql refuses an unshared key or a quoted path", { "single quote") }) +test_that("the bucket DELETE reaches persist tables only", { + e <- source_consolidate() + sql <- e$consolidate_dest_delete_sql("s", + c("log", "log_input", "log_recompute", "streams", "streams_habitat_bt"), + c("ADMS", "BULK")) + expect_identical(sql, c( + "DELETE FROM s.streams WHERE watershed_group_code IN ('ADMS', 'BULK')", + "DELETE FROM s.streams_habitat_bt WHERE watershed_group_code IN ('ADMS', 'BULK')")) + expect_identical(e$consolidate_dest_delete_sql("s", "log", "ADMS"), + character(0)) +}) + +test_that("stage 2 loads a log table in one transaction, a persist table by COPY", { + e <- source_consolidate() + sqlf <- withr::local_tempfile(fileext = ".sql") + cmd <- e$consolidate_stage2_cmd("s", "log", c("run_id", "host"), "/tmp/x.tsv", + sqlf) + expect_match(cmd, "psql -X -v ON_ERROR_STOP=1 -1 -f ", fixed = TRUE) + expect_match(paste(readLines(sqlf), collapse = "\n"), + "WHERE (d.run_id) = (s.run_id);", fixed = TRUE) + sqlf2 <- withr::local_tempfile(fileext = ".sql") + cmd <- e$consolidate_stage2_cmd("s", "streams", c("id_segment", "geom"), + "/tmp/x.tsv", sqlf2) + expect_match(cmd, "psql -X -v ON_ERROR_STOP=1 -c \"COPY s.streams (id_segment, geom) FROM STDIN\" < ", fixed = TRUE) + expect_false(file.exists(sqlf2)) +}) + +test_that("a group whose destination log is newer refuses the source", { + e <- source_consolidate() + dest <- data.frame(watershed_group_code = c("ADMS", "BULK", "DUNC", "KOTL"), + newest = c(100, 200, 300, 400)) + src <- data.frame(watershed_group_code = c("ADMS", "BULK", "DUNC", "NEWW"), + newest = c(100, 250, 299, 1)) + # ADMS same build, BULK older here: fine. DUNC newer here, KOTL logged here + # and not on the source: refused. NEWW has no row here: fine. + expect_identical(e$consolidate_newer_on_dest(dest, src), c("DUNC", "KOTL")) + expect_identical( + e$consolidate_newer_on_dest(dest[0, ], src), character(0)) +}) + test_that("the staged script keeps a group's earlier builds (live)", { e <- source_consolidate() conn <- skip_if_no_db() @@ -102,6 +146,8 @@ test_that("the staged script keeps a group's earlier builds (live)", { withr::with_envvar(env, suppressWarnings( system2(psql, args, stdout = TRUE, stderr = stderr))) } + cols_log <- c("run_id", "watershed_group_code", "date_start", "date_end") + cols_in <- c("run_id", "watershed_group_code", "table_name", "row_count") stage <- function(table, cols, pk, bad = FALSE) { tsv <- withr::local_tempfile(fileext = ".tsv") # stderr dropped: a NOTICE on it would land in the data file. @@ -114,8 +160,18 @@ test_that("the staged script keeps a group's earlier builds (live)", { res <- run_psql(c("-X", "-1", "-v", "ON_ERROR_STOP=1", "-f", shQuote(sqlf))) attr(res, "status") %||% 0L } - cols_log <- c("run_id", "watershed_group_code", "date_start", "date_end") - cols_in <- c("run_id", "watershed_group_code", "table_name", "row_count") + # Without -1 the script fails closed: ON COMMIT DROP under autocommit drops + # the temp table before \copy, so nothing is deleted or added. + tsv0 <- withr::local_tempfile(fileext = ".tsv") + writeLines("Z\tADMS\t2026-01-01\t\\N", tsv0) + sql0 <- withr::local_tempfile(fileext = ".sql") + writeLines(e$consolidate_history_sql(dst, "log", cols_log, "run_id", tsv0), + sql0) + res0 <- run_psql(c("-X", "-v", "ON_ERROR_STOP=1", "-f", shQuote(sql0))) + expect_false(is.null(attr(res0, "status"))) + expect_identical( + DBI::dbGetQuery(conn, sprintf("SELECT count(*)::int n FROM %s.log", dst))$n, + 3L) # A malformed file fails the whole transaction: nothing is deleted. expect_false(identical(stage("log", cols_log, "run_id", bad = TRUE), 0L)) From 1ccfafe50862e50a19cee8ca9d74b6b42b31619f Mon Sep 17 00:00:00 2001 From: almac2022 Date: Sat, 17 Oct 2026 21:01:20 -0700 Subject: [PATCH 07/11] Name a failed source log query; correct the recovery wording (#354) Code-check round 1: a group the host no longer holds is refused by the newest-log guard, not lost; only a partial one loses persist rows. A failed read of the source's log now refuses under its own stage rather than as missing rows. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx --- RUNBOOK.md | 14 ++++---- data-raw/schema_consolidate.R | 14 ++++++-- data-raw/study_area_hosts.sh | 9 ++--- planning/active/review-round1.md | 60 ++++++++++++++++++++++++++++++++ 4 files changed, 84 insertions(+), 13 deletions(-) create mode 100644 planning/active/review-round1.md diff --git a/RUNBOOK.md b/RUNBOOK.md index 48f8995f..281d237c 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -1272,15 +1272,15 @@ To recover a kept host: run its `ip_command`, consolidate its `groups` with `groups_unknown`, and only where `groups_measured` is `yes`**: `groups` is then the finished work still on the host. `unsettled` means wait until nothing runs there and re-measure. `schema_consolidate` deletes its bucket on the dispatcher before it -copies, so consolidating a group the host no longer holds (already -consolidated, then deleted from the source), or holds only in part, loses the -dispatcher's copy of its persist rows. Its log rows survive (link#354), and -consolidate refuses a source outright (`stage = "dest_log_newer"`, nothing +copies, so consolidating a group the host holds only in part loses the +dispatcher's copy of its persist rows (its log rows survive, link#354). +Consolidate refuses a source outright (`stage = "dest_log_newer"`, nothing deleted) when a group's newest log row on the dispatcher is newer than the source's, or the source has none: the persist tables carry no run id, so that -newest row is what says which build they hold. A group re-run on the -dispatcher during recovery is therefore never overwritten by the kept host's -older build. Where it says `no`, find out what is left first. If the +newest row is what says which build they hold. So a group the host no longer +holds at all (already consolidated, then deleted from the source, log rows +included) is refused rather than lost, and a group re-run on the dispatcher +during recovery is never overwritten by the kept host's older build. Where it says `no`, find out what is left first. If the run's recompute ran, it ran without the kept groups: recompute each one and each run group upstream of it (`wsg_recompute_one.R`). If it did not run (consolidate failed, coverage FATAL), recompute every group of the run; the diff --git a/data-raw/schema_consolidate.R b/data-raw/schema_consolidate.R index 82702b08..62e321cd 100644 --- a/data-raw/schema_consolidate.R +++ b/data-raw/schema_consolidate.R @@ -416,8 +416,18 @@ schema_consolidate <- function(schema, sprintf("PGHOST=localhost PGPORT=5432 PGDATABASE=%s PGUSER=%s PGPASSWORD=postgres psql -X -t -A -c \"%s\"", pg_db, pg_user, newest_sql) } - src_newest_raw <- system(sprintf("ssh '%s' '%s'", src$host, - gsub("'", "'\\''", src_newest_inner, fixed = TRUE)), intern = TRUE) + src_newest_raw <- suppressWarnings(system(sprintf("ssh '%s' '%s'", + src$host, gsub("'", "'\\''", src_newest_inner, fixed = TRUE)), + intern = TRUE)) + # A failed query would read as "no log rows on the source" and refuse + # under the wrong reason; name it instead. + if (!is.null(attr(src_newest_raw, "status"))) { + log(src$host, " -> ERROR: could not read the source's log (exit ", + attr(src_newest_raw, "status"), ") — nothing deleted or copied") + results[[src$host]] <- list(ok = FALSE, stage = "source_log_query", + rc = attr(src_newest_raw, "status")) + next + } src_newest_raw <- strsplit(src_newest_raw[nzchar(src_newest_raw)], "|", fixed = TRUE) src_newest <- data.frame( diff --git a/data-raw/study_area_hosts.sh b/data-raw/study_area_hosts.sh index bb085433..0915cd35 100644 --- a/data-raw/study_area_hosts.sh +++ b/data-raw/study_area_hosts.sh @@ -465,8 +465,9 @@ burn_cyphers() { # `groups` is MEASURED where the host answers: what is still on it, for every # group it was asked to run, finished, or logged done. That matters for # recovery: schema_consolidate DELETEs the bucket on the dispatcher before it -# copies, so consolidating a group the host no longer holds destroys the -# dispatcher's copy of its persist rows (its log rows survive, #354). Where the host does not answer, `groups` is the +# copies, so consolidating a group the host holds only in part destroys the +# dispatcher's copy of its persist rows (its log rows survive). One it no +# longer holds at all has no log row there, so consolidate refuses it (#354). Where the host does not answer, `groups` is the # candidate set and `groups_measured` says no. # # No IP in the file: redact_log_addresses rewrites every ${TS}_* file, so it @@ -521,8 +522,8 @@ hosts_kept_report() { echo " schema_consolidate on this machine) and re-measure; where it says" echo " no, find out what is left first. Never groups_partial. schema_consolidate" echo " DELETEs its bucket on the dispatcher before it copies, so a group the" - echo " host no longer holds, or holds only in part, loses the dispatcher's copy" - echo " of its persist rows (its log rows survive)." + echo " host holds only in part loses the dispatcher's copy of its persist rows" + echo " (one it no longer holds at all is refused, dest_log_newer)." if [ "${RECOMPUTE_RAN:-0}" = "1" ]; then echo " The run's recompute ran WITHOUT these groups: after consolidating," echo " recompute each one and each run group upstream of it" diff --git a/planning/active/review-round1.md b/planning/active/review-round1.md new file mode 100644 index 00000000..cd20b13e --- /dev/null +++ b/planning/active/review-round1.md @@ -0,0 +1,60 @@ +# Code-check round 1 — branch 354 (schema_consolidate keeps log history) + +Reviewer: unnamed subagent, 2026-10-17. Scope: `git diff main...HEAD` (branch.diff) plus full +reads of `data-raw/schema_consolidate.R`, `R/lnk_log.R` (specs/DDL), the three touched test +files, the callers in `data-raw/study_area_run.sh` / `wsgs_run_pipeline.sh` / +`study_area_hosts.sh`, and the log readers in `R/lnk_scenario_export.R` / `R/lnk_habitat_validate.R`. + +## Verdict: code clean. One low-severity documentation inaccuracy. + +## Findings + +- **[severity: fragile (doc only)]** RUNBOOK.md:1274-1277, data-raw/study_area_hosts.sh:466-469 and :523-525. + These say that consolidating a group the host "no longer holds (already consolidated, then + deleted from the source) ... loses the dispatcher's copy of its persist rows". With this branch + that case is now **refused**, not lossy. A successful consolidate copies the log tables and + then its source cleanup (step 5, `copied_tables`) deletes the bucket's log rows on the source + too. A re-consolidate of that group therefore finds a destination log row and no source row, + and `consolidate_newer_on_dest()` refuses the source (`dest_log_newer`) before any DELETE. Only + the "holds only in part" case (log row still on the source, persist rows partial) still loses + the dispatcher's persist rows. The RUNBOOK passage goes on to describe the refusal, so the + paragraph contradicts itself. The error is in the conservative direction and causes no data + loss. Suggested wording: "...a group the host holds only in part loses the dispatcher's copy + of its persist rows; one it no longer holds at all is refused (no log row on the source)." + +## Checked and found OK (so a later round need not re-derive) + +- The newest-log guard runs before the persist DELETE, and the `log_pk_unshared` guard too. + Each `next`s with ok = FALSE, so neither can leave a bucket half-deleted. +- Parsing the source's newest values: I simulated the exact ssh/psql string through a local + fake `ssh` (via = psql, read-only query on `fresh_default`). The `'\''` escaping comes through + intact and the result parses to the right `data.frame`. Over all 246 `fresh_default.log` rows, + RPostgres's double for `extract(epoch FROM date_start)` is identical to `as.numeric()` of + psql's text, so the same build on both sides never compares as "newer". `date_start` is + `timestamptz NOT NULL`, so `max()` is never NULL and the session timezone does not matter. +- If the source query fails (ssh or psql non-zero, exit status not checked), it reads as "no + source rows". The source is then refused if the destination holds a row for any bucket group, + and proceeds otherwise, when there is nothing to protect. So it fails closed; only the log + message would be misleading. +- A source whose schema has no `log` table skips the newest guard (`"log" %in% split_tables$history` + is FALSE). That is the pre-#354 behaviour, and every host that runs `lnk_persist_init()` has + the table, so I did not raise it as a finding. +- `consolidate_history_sql()` under `psql -X -1 -v ON_ERROR_STOP=1 -f` is one transaction: + temp table, `\copy`, DELETE by full PK, INSERT. The live test proves it fails closed both + without `-1` and on a malformed row. An empty TSV (e.g. a cypher's empty `log_recompute`) is a + clean no-op. +- `pre_rows`/`post_rows` semantics are unchanged in effect: persist tables are still deleted, + so a real copy still raises the count. +- Driver paths: `study_area_run.sh` runs `pkgload::load_all()` before sourcing, so + `link:::.lnk_log_specs` resolves. `wsgs_run_pipeline.sh` uses `library(link)`, so it needs + an installed link with the new helper; until then it errors at step 3.5, before any DELETE + (fails closed). +- The bucket fed in by `study_area_run.sh` (`host_consolidate_bucket` = database finished ∪ log + done) always has a source log row per group, so the guard does not refuse a normal run. +- Restore: `fresh_default.log` 246 rows / 241 groups, `log_input` 2,952, 0 orphan `log_input` + rows. The 5 restored rows are `config_name = default` and closed, so the newest-row readers + (`.lnk_export_runs`, `lnk_habitat_validate`) are unaffected. This matches + `restore_log_rows.out`. +- `NULLS LAST` edits in the two diagnostic scripts are correct. +- Tests (in a temp copy, `NOT_CRAN=true`, live DB): test-schema_consolidate.R 7/7 pass (live + block ran), test-lnk_pipeline_persist.R 9/9 pass, test-lnk_log.R 295 pass / 0 fail. From 0a2298525e871a80e2db0b9650b596b56e11b1b9 Mon Sep 17 00:00:00 2001 From: almac2022 Date: Sat, 17 Oct 2026 21:07:24 -0700 Subject: [PATCH 08/11] Code-check round 2: clean (#354) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx --- planning/active/review-round2.md | 42 ++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 planning/active/review-round2.md diff --git a/planning/active/review-round2.md b/planning/active/review-round2.md new file mode 100644 index 00000000..0ab2d336 --- /dev/null +++ b/planning/active/review-round2.md @@ -0,0 +1,42 @@ +# Code-check round 2 — branch `354-schema-consolidate-deletes-the-destinati` (#354) + +Reviewed 2026-10-17: the whole branch (`git diff merge-base...HEAD`, commits 0a1f8534..1ccfafe5), with attention to round 1's fixes in 1ccfafe5. + +## Clean + +No issues found. + +## What was checked, and how + +- **The round 1 fixes are correct.** + - The `source_log_query` stage: `system(intern = TRUE)` sets `attr(, "status")` only on a non-zero exit, and the branch reads it before parsing. I simulated the guard's quoting with `sh -c` in place of ssh, using the same `gsub("'", "'\\''")` escape. A good query returned `GROUP|epoch` lines with a NULL status. A query on a missing relation returned `character(0)` with status 1. + - The corrected docs (RUNBOOK §8d, the `study_area_hosts.sh` comment and `hosts_kept_report` echo) match the code. A group consolidated and then cleaned from the source has no source log row: step 5 deletes `copied_tables`, and the log tables are among them. The dispatcher still has that group's row, so the source is refused with `dest_log_newer`, and the refusal comes before step 3.5's DELETE. +- **End to end, in a temp copy of the repo with a fake `ssh` on PATH** (the fake rewrites the schema name and runs the command locally). Scratch schemas `zzr2_a` (destination) and `zzr2_b` (source), dropped afterwards, `keep_source = TRUE`: + - normal (destination row older): `ok`, persist replaced, BULK outside the bucket untouched, both log rows kept; + - destination row newer: `dest_log_newer`, persist and log untouched; + - source has no log row: `dest_log_newer`, untouched; + - destination has no log row: `ok`; + - source log query exits non-zero: `source_log_query`, untouched. +- **Precision of the guard's comparison.** The destination value comes through RPostgres as `numeric` and the source value is psql text. On all 241 `fresh_default` groups the two are equal, with 0 mismatches (read-only), so the same build never compares as newer. +- **Parsing edge cases.** An empty source result gives a 0-row frame, and `merge(all.x = TRUE)` then refuses every destination group. A line with a missing epoch (`"A|"`) parses to NA and is refused. A 0-row destination refuses nothing. `date_start` is NOT NULL, so `sort()` dropping NA cannot hide a group. +- **Stage 2 / the history SQL.** + - `-1` with `ON_ERROR_STOP` rolls the replace back as a whole. + - Without `-1`, `ON COMMIT DROP` makes the script fail closed. The live test covers both. + - The log tables have no unique index other than the primary key (the `idx` list in `.lnk_log_create_tables()`). + - `consolidate_dest_delete_sql()` returns `character(0)` when there are no persist tables, so the zero-length `sprintf` case is right. +- **Ordering.** The newest-log check, the source-query check and `pk_unshared` all `next` before any DELETE. A resumed consolidate after a partial failure has equal newest values on both sides (the log was copied, the source was not cleaned) and proceeds, as intended. +- **The driver path** (`study_area_run.sh`). The consolidate bucket is the database-finished set ∪ log done (`host_consolidate_bucket`). Every member therefore has a source log row from this run, so normal runs are not refused. Any `ok = FALSE` exits 1 and keeps every cypher (the existing behaviour). +- **Readers of a group's history.** The other `.log` readers pick the newest row by `date_start DESC` (`lnk_scenario_export`, `lnk_habitat_validate`, `habitat_variants_build.R`), or aggregate (`habitat_validate.R`), so the restored second rows do not change what they read. The two `date_end DESC` readers gained `NULLS LAST`. +- **The restore.** The live DB has `fresh_default.log` at 246 rows over 241 groups; DUNC, KOTL, LARL, PINE and SLOC have 2 rows each. `log_input` has 2,952 rows and no orphan `log_input` row. These match the README and CLAUDE.md. +- **Tests.** In a temp copy with `NOT_CRAN=true`: `test-schema_consolidate.R` 26 pass, `test-lnk_log.R` 295 pass, `test-lnk_pipeline_persist.R` 23 pass; 0 fail, 0 error, 0 skip. +- **Checklist items checked against the diff:** + - `on.exit` at top level: not used. + - `system2`/`system` status read off the attribute: done. + - psql `:'var'` / `\quit`: not used. + - `paste`/`sprintf` with zero-length input: handled. + - `$` partial matching: none. + - Escaping of the shell and SQL quoting: checked by running it. + - A guard that fails toward pass: none found, apart from the accepted no-`log`-table skip. + - The restore is additive, with `ON CONFLICT DO NOTHING`. + +The repo was not modified, apart from this file. From 6dfcddf3abfedf7da82715ce37d1e789bac8ca07 Mon Sep 17 00:00:00 2001 From: almac2022 Date: Sat, 17 Oct 2026 21:17:10 -0700 Subject: [PATCH 09/11] Say which log tables travel and when a group is refused (#354) Code-check round 3, closed by enumerating every added sentence that restates the history rule or the guard: seven dropped a condition the code has (only three log tables are copied; never deleted by group on the destination; refusal needs a destination log row). RUNBOOK names the pre-log case the guard does not protect. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx --- CLAUDE.md | 2 +- R/lnk_log.R | 10 +++-- RUNBOOK.md | 23 ++++++---- data-raw/README.md | 2 +- data-raw/schema_consolidate.R | 13 +++--- data-raw/study_area_hosts.sh | 7 ++- planning/active/findings.md | 26 +++++++++++ planning/active/review-round3.md | 76 ++++++++++++++++++++++++++++++++ 8 files changed, 137 insertions(+), 22 deletions(-) create mode 100644 planning/active/review-round3.md diff --git a/CLAUDE.md b/CLAUDE.md index 9d3e9413..cf4956eb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -22,7 +22,7 @@ Experimental package — breaking all the time and loving the learning curve. St - **The recompute takes 154 min at 241 groups and `-j4`** (median 2.0 min per group, longest 16 min), not the ~1 h the plan assumed. - **A present species with no barrier in a group is accessible** (`access_` 1 or 2) **and has `has_barriers__dnstr` FALSE**, so its mapping code is built. Zero barrier rows mean NA ("no data") only when nothing says the species is present. Before `91a968ab` the flag stayed NA and UPET RB shipped with `mapping_code_rb` "" on every segment; UPET RB is the only such case in the 241. - **`fresh_default` is 31 GB** (14 GB tables, 17 GB indexes; 11.39 M segments), up from 7.3 GB. Summing `pg_total_relation_size()` over every `pg_class` row counts each index twice (48 GB). The fwapg volume is 600 GiB with 253 G free after the run. -- **The run replaced the August KOTL, LARL and SLOC builds and the 2026-10-12 DUNC build** (status below), which published floodplains items had come from. Those four old builds survive in a machine-local `pg_dump`, the driver's pre-consolidate backup: it was taken after m1 had rebuilt its own 76 groups but before the cloud hosts' groups (all four were on job3) were copied in, so it is **not** the whole pre-run schema. The pre-run state of the other pre-run groups m1 rebuilt is gone. Operator, 2026-10-17: no floodplains item needs the old builds restored. **Their log rows were restored from that dump (#354):** DUNC, KOTL, LARL, SLOC and PINE (also cypher-built, not named before) each hold their earlier `log` row again, and their `log_input` rows (5 + 60; `data-raw/logs/consolidate_history_354/`). Since #354 consolidate keeps the log tables' history, replacing their rows by primary key. +- **The run replaced the August KOTL, LARL and SLOC builds and the 2026-10-12 DUNC build** (status below), which published floodplains items had come from. Those four old builds survive in a machine-local `pg_dump`, the driver's pre-consolidate backup: it was taken after m1 had rebuilt its own 76 groups but before the cloud hosts' groups (all four were on job3) were copied in, so it is **not** the whole pre-run schema. The pre-run state of the other pre-run groups m1 rebuilt is gone. Operator, 2026-10-17: no floodplains item needs the old builds restored. **Their log rows were restored from that dump (#354):** DUNC, KOTL, LARL, SLOC and PINE (also cypher-built, not named before) each hold their earlier `log` row again, and their `log_input` rows (5 + 60; `data-raw/logs/consolidate_history_354/`). Since #354 consolidate keeps the dispatcher's history in the three log tables it copies (`log`, `log_input`, `log_recompute`), replacing their rows there by primary key. ## Status (2026-10-16) — `default_tuned` reads wet's discharge; BT's MAD minima 0.043 / 0.043 (#331) diff --git a/R/lnk_log.R b/R/lnk_log.R index f44f83de..b4b3fb7b 100644 --- a/R/lnk_log.R +++ b/R/lnk_log.R @@ -495,10 +495,12 @@ cols_log_input <- c( #' The provenance tables: name, columns and primary key. #' #' These tables are history, not per-group state, and no primary key here -#' includes `watershed_group_code`. `data-raw/schema_consolidate.R` replaces -#' their rows by primary key and never deletes them by watershed group, so a -#' group copied home from another host keeps the rows of its earlier builds -#' (link#354). +#' includes `watershed_group_code`. Of them, `data-raw/schema_consolidate.R` +#' copies the three that carry `watershed_group_code` (`log`, `log_input`, +#' `log_recompute`; the config-snapshot tables never leave a host). It +#' replaces their rows on the destination by primary key and never deletes +#' them there by watershed group, so a group copied home from another host +#' keeps the rows of its earlier builds (link#354). #' #' Read by [.lnk_log_create_tables()] and by `schema_consolidate.R`, so the two #' agree on which tables are history. (`lnk_persist_init()`'s drift check keeps diff --git a/RUNBOOK.md b/RUNBOOK.md index 281d237c..5f8cf83d 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -506,8 +506,9 @@ EXCEPT SELECT watershed_group_code FROM .log; loader rows that cannot be re-created (link#270's FWA rows) survive it. - `log`, `log_input` and `log_recompute` carry `watershed_group_code` so `schema_consolidate.R` auto-discovers them. They are history, not per-group - state: consolidate replaces their rows by primary key (one key is one build) - and never deletes them by group, so a group copied home from a cloud host + state: consolidate replaces their rows on the dispatcher by primary key (one + key is one build) and never deletes them there by group (the source's copy + is still cleared once copied), so a group copied home from a cloud host keeps its earlier builds' rows (link#354; before it, every cloud-built group kept only its newest row). The set is `.lnk_log_specs()`, read by the DDL and by consolidate. `log_parameters_fresh` / `log_parameters_habitat_thresholds` / @@ -1274,13 +1275,17 @@ then the finished work still on the host. `unsettled` means wait until nothing runs there and re-measure. `schema_consolidate` deletes its bucket on the dispatcher before it copies, so consolidating a group the host holds only in part loses the dispatcher's copy of its persist rows (its log rows survive, link#354). -Consolidate refuses a source outright (`stage = "dest_log_newer"`, nothing -deleted) when a group's newest log row on the dispatcher is newer than the -source's, or the source has none: the persist tables carry no run id, so that -newest row is what says which build they hold. So a group the host no longer -holds at all (already consolidated, then deleted from the source, log rows -included) is refused rather than lost, and a group re-run on the dispatcher -during recovery is never overwritten by the kept host's older build. Where it says `no`, find out what is left first. If the +Where both sides have a `log` table, consolidate refuses a source outright +(`stage = "dest_log_newer"`, nothing deleted) when a group has a log row on +the dispatcher and the source's newest is older or absent: the persist tables +carry no run id, so that newest row is what says which build they hold. So a +group the host no longer holds at all (already consolidated, then deleted from +the source, log rows included) is refused rather than lost, and a group re-run +on the dispatcher during recovery is never overwritten by the kept host's +older build. A group with persist rows but **no** log row on the dispatcher +(built before the log tables, link#127: 59 of `fresh`'s 98 groups, 0 in +`fresh_default`, 2026-10-17) is not protected: it is deleted and, if the host +no longer holds it, not replaced. Where it says `no`, find out what is left first. If the run's recompute ran, it ran without the kept groups: recompute each one and each run group upstream of it (`wsg_recompute_one.R`). If it did not run (consolidate failed, coverage FATAL), recompute every group of the run; the diff --git a/data-raw/README.md b/data-raw/README.md index be4b8d79..bcbc06dd 100644 --- a/data-raw/README.md +++ b/data-raw/README.md @@ -90,7 +90,7 @@ Run-adjacent helpers (planning, consolidation across hosts). | Script | Purpose | |--------|---------| | `buckets_balance.R` | Standalone LPT planner for the 3-host case. Reads per-host wall times from prior runs and prints buckets ready to paste into `wsgs_dispatch.sh --m4-bucket=…`. **Superseded for the N-host orchestrator** — `wsgs_dispatch.sh` now computes the LPT plan inline at dispatch time using the same algorithm. Kept here for one-off planning + cross-checks. Dedups `(wsg, host)` and across hosts before LPT so multi-run CSV accumulation doesn't double-assign WSGs. | -| `schema_consolidate.R` | Per-table, bucket-filtered `COPY` from each source host over ssh into the dispatcher. Deletes each source's bucket from the destination's persist tables before copying; the log tables are history, so their rows are replaced by primary key and never deleted by group, and a source is refused if a group's destination log is newer than its own (#354). `ok = TRUE` requires every table's COPY to succeed and the row count to rise. | +| `schema_consolidate.R` | Per-table, bucket-filtered `COPY` from each source host over ssh into the dispatcher. Deletes each source's bucket from the destination's persist tables before copying; the three log tables it copies (`log`, `log_input`, `log_recompute`) are history, so on the destination their rows are replaced by primary key and never deleted by group, and a source is refused if a group that has a destination log row has an older or no log row on the source (#354). `ok = TRUE` requires every table's COPY to succeed and the row count to rise. | | `runs_archive.sh` | Moves the current top-level `_per_wsg_times.csv` + `*.rds` + `*_annotated.csv` artifacts in `provincial_/` to `archive//`. Operator cadence: run between provincial runs when you want the LPT planner to use the most recent run only. Skip to median-over multiple recent runs. | | `trifecta_smoke.sh` | Thin shim over `wsgs_dispatch.sh` — one small WSG per host (m4→DEAD, m1→ELKR, cyN→ADMS/BABL/BULL). ~3 min wall. Exercises every orchestrator code path (preflight, dispatch, tunnel, RDS pull-back, annotation) before committing to a 200-WSG run. All flags pass through (e.g. `--cy-workspaces=`, `--mapping-code`). | diff --git a/data-raw/schema_consolidate.R b/data-raw/schema_consolidate.R index 62e321cd..948c1f0e 100644 --- a/data-raw/schema_consolidate.R +++ b/data-raw/schema_consolidate.R @@ -31,11 +31,14 @@ # COPY-streaming with `WHERE wsg IN (bucket)` at source eliminates the # over-fetch class entirely. # -# The log tables are the exception to "replace the bucket" (link#354). They -# carry `watershed_group_code`, so they are discovered and copied like the -# rest, but they are history: a group's earlier builds must survive a later -# build copied home from another host. Their rows are replaced by primary key -# (`.lnk_log_specs()`), never deleted by group. +# The log tables are the exception to "replace the bucket" (link#354). The +# three that carry `watershed_group_code` (`log`, `log_input`, +# `log_recompute`) are discovered and copied like the rest, but they are +# history: a group's earlier builds must survive a later build copied home +# from another host. On the destination their rows are replaced by primary +# key (`.lnk_log_specs()`), never deleted by group; step 5 still clears them +# from the source once copied. The config-snapshot log tables carry no +# `watershed_group_code`, so they are never discovered and never travel. # # Cross-refs: rtj#94 (general orchestrator); link#112 (first usage); # link#180 (additive Step 0 unblocked by this source-filter fix). diff --git a/data-raw/study_area_hosts.sh b/data-raw/study_area_hosts.sh index 0915cd35..dffccca9 100644 --- a/data-raw/study_area_hosts.sh +++ b/data-raw/study_area_hosts.sh @@ -467,7 +467,9 @@ burn_cyphers() { # recovery: schema_consolidate DELETEs the bucket on the dispatcher before it # copies, so consolidating a group the host holds only in part destroys the # dispatcher's copy of its persist rows (its log rows survive). One it no -# longer holds at all has no log row there, so consolidate refuses it (#354). Where the host does not answer, `groups` is the +# longer holds at all has no log row there, so consolidate refuses it when the +# dispatcher has a log row for it; one with only pre-log rows on the +# dispatcher is still deleted and not replaced (#354). Where the host does not answer, `groups` is the # candidate set and `groups_measured` says no. # # No IP in the file: redact_log_addresses rewrites every ${TS}_* file, so it @@ -523,7 +525,8 @@ hosts_kept_report() { echo " no, find out what is left first. Never groups_partial. schema_consolidate" echo " DELETEs its bucket on the dispatcher before it copies, so a group the" echo " host holds only in part loses the dispatcher's copy of its persist rows" - echo " (one it no longer holds at all is refused, dest_log_newer)." + echo " (one it no longer holds at all is refused, dest_log_newer, where the" + echo " dispatcher has a log row for it; with pre-log rows only, it is lost)." if [ "${RECOMPUTE_RAN:-0}" = "1" ]; then echo " The run's recompute ran WITHOUT these groups: after consolidating," echo " recompute each one and each run group upstream of it" diff --git a/planning/active/findings.md b/planning/active/findings.md index fd70f6fd..614e468c 100644 --- a/planning/active/findings.md +++ b/planning/active/findings.md @@ -57,6 +57,32 @@ Run on copies of the test and script under the scratchpad: expectations fail. - Every table treated as persist (`is_history` all FALSE): 3 split expectations fail. +## Code-check: how the loop ended (enumeration, 2026-10-17) + +Round 1: one doc error (§8d / host banners said a group the host no longer holds is lost; the +guard refuses it), fixed with a status check on the source log query. Round 2: clean. Round 3: +two wording findings, one **inside round 1's fix** (the "is refused" sentence lacked the +condition that the dispatcher holds a log row for the group; 59 of `fresh`'s 98 groups have +persist rows and no log row). + +Mechanism (round 3): prose that restates the guard or the history rule drops a condition the +code has. Three conditions can be dropped: (a) only the three log tables with +`watershed_group_code` are copied, the config-snapshot ones never travel; (b) "never deleted by +group" holds on the destination, step 5 still clears the source; (c) the refusal needs a +destination log row and a `log` table on both sides. + +Enumeration: every added line on the branch matching +`refus|never delet|not .*delet|history|survive|by primary key|discovered|copied like|travel|keeps? (its|their)` +(47 lines, plus CLAUDE.md's new status sentence). Seven dropped a condition and were fixed: +`R/lnk_log.R` specs doc (a, b); consolidate header (a, b); RUNBOOK §6b (b); RUNBOOK §8d (c, and +now names the unprotected pre-log case); `data-raw/README.md` (a, b, c); `study_area_hosts.sh` +comment and banner (c); CLAUDE.md (a, b). The rest restate the destination DELETE, a test, or a +helper's own scope, and hold as written. + +Accepted: over Tailscale SSH to a Mac, ssh exits 0 on a failed remote command, so the +`source_log_query` stage could read as `dest_log_newer` there. Sources are Linux droplets, and +the source is refused either way. + ## Errors Encountered | Error | Resolution | diff --git a/planning/active/review-round3.md b/planning/active/review-round3.md new file mode 100644 index 00000000..057a03f0 --- /dev/null +++ b/planning/active/review-round3.md @@ -0,0 +1,76 @@ +# Code-check round 3 — #354 (branch `354-schema-consolidate-deletes-the-destinati`) + +Diff: `git diff 3b6e591e...HEAD` (excluding `*.out` evidence and `review-*.md`). + +## Mechanism + +The fact "which build a group's persist rows hold" now has one producer in code: the +newest-log guard in step 3.5 (`max(date_start)` per group, destination against source). +Rounds 1 and the plan review both found the same thing: prose that restates that code path +or the history rule, and drops a condition the code has. Round 1's instance was "a group the +host no longer holds loses the dispatcher's rows", which the guard actually refuses. This +round enumerated every restatement and every reader of "newest log row": + +| where | restates | verdict | +|---|---|---| +| `schema_consolidate.R` step 3.5 guard | newest by `max(date_start)` | the producer | +| `lnk_log_read()`, `.lnk_export_runs()`, `.lnk_hv_check_log()`, `habitat_variants_build.R:530` | newest by `date_start DESC` | consistent with the guard | +| `habitat_diff_segments.R:164`, `query_habitat_thresholds_fiss.R:226` | `date_end DESC NULLS LAST` | stamps only, decide nothing; fine | +| guard's epoch compare, DBI double vs `as.numeric()` of psql text | equality on a re-copied row | measured: 0 mismatches over 241 real `fresh_default` groups and 20,000 synthetic 16-digit epochs | +| PK replace by `run_id` / `recompute_id` | one key = one build | host-prefixed ids, distinct hostnames in `fresh_default.log` (m1, cypher-job1..3), no `set.seed` in R/ or data-raw; holds | +| `dunc_default_328/README.md` | verify.R now sees 2 DUNC rows | measured: DUNC 2 rows; documented | +| header comment, `.lnk_log_specs()` roxygen, data-raw/README | which tables are history, and that they are never deleted by group | finding 1 | +| RUNBOOK §8d, `study_area_hosts.sh:467-470` and `:525-526` | when the guard refuses | finding 2 | + +## Findings + +- **[severity: fragile]** `data-raw/schema_consolidate.R:34-38`, `R/lnk_log.R:497-501`, + `data-raw/README.md:93`: they say more about the log tables than the code does. + - The header says "The log tables ... carry `watershed_group_code`, so they are discovered + and copied like the rest". The roxygen says consolidate "replaces their rows by primary + key". Both refer to the six `.lnk_log_specs()` tables. + - In fact only `log`, `log_input` and `log_recompute` carry the column. I checked + `information_schema` for `fresh` and `fresh_default`. The three config-snapshot tables + (`log_parameters_fresh`, `log_parameters_habitat_thresholds`, `log_dimensions`) are never + discovered and never leave a cloud host. RUNBOOK §6b says so correctly ("they do not yet + travel between hosts"), and the test comment in `test-schema_consolidate.R:27-28` agrees. + - The header, the roxygen and data-raw/README also say the log rows are "never deleted by + group". Step 5 (`schema_consolidate.R:622-626`) does delete the log tables by + `watershed_group_code` on the source, because they are in `copied_tables`. This loses no + data, since the rows were copied first. The RUNBOOK's "already consolidated, then deleted + from the source, log rows included" relies on that deletion. + - Effect: a reader of the code comments would conclude that a config snapshot built only on + a cloud host survives the burn. It does not. + - Fix: qualify the three sentences. Say "the three that carry `watershed_group_code`", and + "never deleted by group on the destination". + +- **[severity: fragile]** `data-raw/study_area_hosts.sh:469-470` and `:525-526`, RUNBOOK.md + §8d (around line 1280): the refusal is stated without its precondition. + - The docs say a group the host no longer holds at all "has no log row there, so + consolidate refuses it". `consolidate_newer_on_dest()` refuses only a group that **has a + log row on the destination** (`merge(..., all.x = TRUE)` on `dest`). + - Where the dispatcher holds a group's persist rows with no log row (builds from before the + log tables existed), the guard passes. The bucket DELETE then removes those rows, and + nothing replaces them. The coverage check in `study_area_run.sh` detects this only + afterwards. + - Today `fresh_default` has 0 such groups. `fresh` has **59 of 98** (measured: + `fresh.streams` groups minus `fresh.log` groups). + - So in a `bcfishpass`-config recovery, the operator text printed by `hosts_kept_report` + promises a refusal that the code does not make. + - Fix: add the condition, e.g. "is refused when the dispatcher has a log row for it (any + build since the log tables, #127); a group with only pre-log rows here is still deleted + and not replaced". + +No bugs, security issues or data-loss paths found in the code itself. Checked: + +- The staged replace runs as one transaction (`-1` and `ON COMMIT DROP`). +- The guard and the `log_pk_unshared` check run before any DELETE. +- `copy_empty` still behaves correctly on a re-consolidate, because the persist tables are + re-added. +- A refusal makes the driver exit non-zero and keep the cyphers + (`study_area_run.sh:1352-1358`). +- An empty source result parses to a zero-row frame. NA epochs fail closed. + +Not raised: the source-log status check cannot see a failed remote query over a transport +that loses the exit status (Tailscale SSH to a Mac). It then refuses under the wrong stage +name. That fails closed, and every other ssh call in the file has the same pattern. From edc7146c89ae05ae2339b286dd777c0977b9e833 Mon Sep 17 00:00:00 2001 From: almac2022 Date: Sat, 17 Oct 2026 21:17:35 -0700 Subject: [PATCH 10/11] Archive planning files for issue #354 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx --- .../README.md | 38 +++++++++++++++++++ .../findings.md | 0 .../progress.md | 0 .../review-1.md | 0 .../review-round1.md | 0 .../review-round2.md | 0 .../review-round3.md | 0 .../task_plan.md | 6 +-- 8 files changed, 41 insertions(+), 3 deletions(-) create mode 100644 planning/archive/2026-10-issue-354-consolidate-log-history/README.md rename planning/{active => archive/2026-10-issue-354-consolidate-log-history}/findings.md (100%) rename planning/{active => archive/2026-10-issue-354-consolidate-log-history}/progress.md (100%) rename planning/{active => archive/2026-10-issue-354-consolidate-log-history}/review-1.md (100%) rename planning/{active => archive/2026-10-issue-354-consolidate-log-history}/review-round1.md (100%) rename planning/{active => archive/2026-10-issue-354-consolidate-log-history}/review-round2.md (100%) rename planning/{active => archive/2026-10-issue-354-consolidate-log-history}/review-round3.md (100%) rename planning/{active => archive/2026-10-issue-354-consolidate-log-history}/task_plan.md (93%) diff --git a/planning/archive/2026-10-issue-354-consolidate-log-history/README.md b/planning/archive/2026-10-issue-354-consolidate-log-history/README.md new file mode 100644 index 00000000..0685fc83 --- /dev/null +++ b/planning/archive/2026-10-issue-354-consolidate-log-history/README.md @@ -0,0 +1,38 @@ +## Outcome + +`schema_consolidate()` deleted the destination's bucket from every table with a +`watershed_group_code` column, so the three log tables that carry one (`log`, `log_input`, +`log_recompute`) lost every earlier build of each cloud-built group. The log tables are now +named once in `.lnk_log_specs()`; consolidate deletes the bucket from persist tables only and +loads a log table through a one-transaction staged replace by primary key, so earlier builds +survive and a re-consolidate changes nothing. The plan's `run_key` became the full primary key, +which deletes only rows being rewritten. The plan review found the consequence that mattered: +with history kept, a group's newest log row is what names the build its persist rows hold, so +consolidate now refuses a source (nothing deleted) when a group has a destination log row and +the source's is older or absent. `lnk_pipeline_persist()` never touched the log tables; a test +pins it. The rows #246 had lost were restored from the driver's pre-consolidate dump. + +## Measurement + +- Lost in run `20261017_004619-3f4c713b`: 5 `log` rows (DUNC, KOTL, LARL, SLOC, and PINE, which + the issue did not name) and their 60 `log_input` rows, 0 `log_recompute`. Restored: + `fresh_default.log` 241 → 246, `log_input` 2,892 → 2,952. Both verify scripts pass at 241; + `lnk_log_read()` still gives this run as newest for all 241 groups. +- Readers already handled history: `fresh.log` held 2–4 rows for 20 groups. +- Unprotected by the guard: groups with persist rows and no log row, 59 of `fresh`'s 98, 0 of + `fresh_default`'s 241 (RUNBOOK §8d). +- Dropping `psql -1` fails closed (`relation "consolidate_in" does not exist`), measured. +- Restore-the-bug: five mutations (group-keyed DELETE, unsplit tables, unsplit DELETE list, no + `-1`, guard off) each turn the tests red. +- Suite (local `devtools::test()`): 3,147 pass; 5 failures, all predating the branch + (tunnel `:63333` down ×3, golden ADMS dams 8 → 7 ×2). +- Review: a plan review, then three `/code-check` rounds. Round 3 found a defect inside round 1's + fix, and the loop ended by enumerating the 47 added lines that restate the rule (7 fixed; + `findings.md`). Wrong turn kept: the background test wrapper reported done while R was still + running, so the wait was redone on the PID. + +## Evidence + +`data-raw/logs/consolidate_history_354/*` + +Closed by: PR for #354 diff --git a/planning/active/findings.md b/planning/archive/2026-10-issue-354-consolidate-log-history/findings.md similarity index 100% rename from planning/active/findings.md rename to planning/archive/2026-10-issue-354-consolidate-log-history/findings.md diff --git a/planning/active/progress.md b/planning/archive/2026-10-issue-354-consolidate-log-history/progress.md similarity index 100% rename from planning/active/progress.md rename to planning/archive/2026-10-issue-354-consolidate-log-history/progress.md diff --git a/planning/active/review-1.md b/planning/archive/2026-10-issue-354-consolidate-log-history/review-1.md similarity index 100% rename from planning/active/review-1.md rename to planning/archive/2026-10-issue-354-consolidate-log-history/review-1.md diff --git a/planning/active/review-round1.md b/planning/archive/2026-10-issue-354-consolidate-log-history/review-round1.md similarity index 100% rename from planning/active/review-round1.md rename to planning/archive/2026-10-issue-354-consolidate-log-history/review-round1.md diff --git a/planning/active/review-round2.md b/planning/archive/2026-10-issue-354-consolidate-log-history/review-round2.md similarity index 100% rename from planning/active/review-round2.md rename to planning/archive/2026-10-issue-354-consolidate-log-history/review-round2.md diff --git a/planning/active/review-round3.md b/planning/archive/2026-10-issue-354-consolidate-log-history/review-round3.md similarity index 100% rename from planning/active/review-round3.md rename to planning/archive/2026-10-issue-354-consolidate-log-history/review-round3.md diff --git a/planning/active/task_plan.md b/planning/archive/2026-10-issue-354-consolidate-log-history/task_plan.md similarity index 93% rename from planning/active/task_plan.md rename to planning/archive/2026-10-issue-354-consolidate-log-history/task_plan.md index c6a8375f..0e73af43 100644 --- a/planning/active/task_plan.md +++ b/planning/archive/2026-10-issue-354-consolidate-log-history/task_plan.md @@ -49,6 +49,6 @@ ## Validation - [x] Tests pass — local `devtools::test()` only (data-raw is build-ignored and there is no R-CMD-check workflow): 3147 pass; 5 fail, all pre-existing (3 need the `:63333` tunnel, 2 are the golden ADMS dams pin 8 → 7 in CLAUDE.md). Touched files re-run after review fixes: 0 fail -- [ ] `/code-check` clean (each commit, or once over the branch with `/code-check branch`) -- [ ] PWF checkboxes match landed work -- [ ] `/planning-archive` on completion +- [x] `/code-check branch` — 3 rounds (1 doc finding; clean; 2 wording findings, one inside round 1's fix), ended by enumerating the 47 restating lines (findings.md) +- [x] PWF checkboxes match landed work +- [x] `/planning-archive` on completion From 4e63c4c2fcef1be588959dda7f3df7dc61c3a771 Mon Sep 17 00:00:00 2001 From: almac2022 Date: Sat, 17 Oct 2026 21:18:19 -0700 Subject: [PATCH 11/11] Archive README: name PR #355 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx --- .../archive/2026-10-issue-354-consolidate-log-history/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/planning/archive/2026-10-issue-354-consolidate-log-history/README.md b/planning/archive/2026-10-issue-354-consolidate-log-history/README.md index 0685fc83..2f164b16 100644 --- a/planning/archive/2026-10-issue-354-consolidate-log-history/README.md +++ b/planning/archive/2026-10-issue-354-consolidate-log-history/README.md @@ -35,4 +35,4 @@ pins it. The rows #246 had lost were restored from the driver's pre-consolidate `data-raw/logs/consolidate_history_354/*` -Closed by: PR for #354 +Closed by: PR #355