Skip to content

Build the bundled and vignette stream data from link's tables (#87) - #91

Open
NewGraphEnvironment wants to merge 17 commits into
mainfrom
87-build-the-bundled-and-vignette-data-from
Open

Build the bundled and vignette stream data from link's tables (#87)#91
NewGraphEnvironment wants to merge 17 commits into
mainfrom
87-build-the-bundled-and-vignette-data-from

Conversation

@NewGraphEnvironment

@NewGraphEnvironment NewGraphEnvironment commented Oct 18, 2026

Copy link
Copy Markdown
Owner

Summary

  • The bundled test streams (inst/testdata/streams.gpkg) and the Parsnip vignette streams (inst/vignette-data/pars.gpkg) now come from link's default configuration (fresh_default), with upstream area and precipitation joined from fwapg. bcfishpass's views are no longer read.
  • data-raw/network_extract.R and data-raw/wsg_vignette_data.R pin both the link modelling run and the recompute that later rewrote access and mapping codes, and stop before writing when either has moved. pars_meta.rds records the link run, the recompute, the fwapg state read, and the source of each context layer.
  • inst/NOTICE.md now credits each attribute to its source. It keeps a scoped bcfishpass ODbL credit, because link's default config applies bcfishpass's barrier and habitat override tables. Present-tense numbers in the README, vignettes, roxygen and notes were re-measured; dated measurements stay as records.

Breaking change to shipped data (for NEWS)

inst/testdata/streams.gpkg and the streams layer of pars.gpkg:

  • segmented_stream_id (character) is replaced by id_segment (integer); id_segment is unique only together with the new watershed_group_code column.
  • spawning and rearing are logical (TRUE/FALSE), not integer codes.
  • linear_feature_id is added, and rows are ordered by blue_line_key and downstream_route_measure.

fl_valley_join(cols = NULL) therefore now picks up spawning and rearing on the bundled tile.

What moved

  • Bulkley test reach: same 50 segments and geometry. Precipitation differs on 28 segments (−17 to +1 mm) and one access value changed. The floodplain moves by one cell (28,727 → 28,726); the README example output is unchanged.
  • Parsnip: 9,386 segments, against 8,436 before. The floodplain goes from 40,888.2 ha to 40,969.3 ha (+0.2%) on an unchanged DEM.
  • Upstream area: 8 Parsnip segments have no upstream area, because fwapg's watershed lookup has no row for them (fwapg#33). The vignette says so.
  • Corrected along the way, all stale before this change: the vignette's reach length (~5 km, not ~8), the overlap ratio (about 2.2–2.3x, not 2.4x) and the fallback share (5.4%, not 3.0%).

Related Issues

Test plan

  • devtools::test(): 584 passed, 0 failed, 0 skipped
  • R CMD check: 0 errors, 0 warnings, 2 notes, both pre-existing (CITATION.cff and pkgdown/ at the top level)
  • pkgdown::check_pkgdown() clean; lintr finds 145 lints against 149 on main
  • Both changed vignettes render; the Parsnip vignette states its source from pars_meta.rds
  • /code-check branch: 4 rounds, ended by an enumeration of every write and provenance field in the data-raw scripts. Each fix is proven by a harness (planning/archive/2026-10-issue-87-link-sourced-data/findings.md)

Notes

  • The shared database behind the context layers (railways, roads, reserves, parks) was unreachable, so those layers are carried over unchanged from the previous pars.gpkg, which was built at 2ef5c89 on 2026-10-13. pars_meta.rds records this per layer.
  • Durable facts are in research/bundled_stream_data.md; the investigation, including the reviews, is in the planning archive.

🤖 Generated with Claude Code

https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW

NewGraphEnvironment and others added 17 commits October 17, 2026 18:41
@NewGraphEnvironment @claude
Initialize PWF baseline for #87
5228948 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
Build the bundled test streams from link's fresh_default tables (#87)
5dc6e8e 
network_extract.R reads link's default-config BULK run and joins upstream
area and precipitation from fwapg. Same 50 segments and geometry; precip
moves on 28 segments, one access value 1 -> 2, spawning/rearing are now
logical and the id column is id_segment. testdata.R no longer writes
streams.gpkg.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
Read the Parsnip vignette streams from link's fresh_default tables (#87)
80cf2d1 
wsg_vignette_data.R takes streams, access, habitat and mapping code from
link's default-config run, upstream area and precipitation from fwapg,
and stamps the link run in pars_meta.rds in place of bcfishpass.log.
Context layers carry over from the previous gpkg when the shared
database is unreachable, and the meta file records which.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
Pin the link run and guard the database in network_extract.R (#87)
95b0d5b 
Plan-review fixes: stop unless fresh_default holds the expected BULK
run, has rows for it, and the connection reaches a database with the
schema. streams.gpkg gains watershed_group_code, since id_segment is
unique only with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
Re-measure the bundled-tile claims on the link-sourced streams (#87)
d367030 
The join test expects the shipped logical spawning/rearing under
cols = NULL. Channel width is named as fwapg's, its real source. Claims
that were already stale are corrected from measurement: the overlap
ratio (2.28 / 2.21, not 2.4), the fallback share (5.4%, not 3.0%) and
the reach length (5 km, not 8). Timing figures stay as dated records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
Rebuild the Parsnip vignette data from link's tables (#87)
09a5eeb 
9,386 order 3+ bull-trout-accessible segments from link's default run
(8,436 from bcfishpass before); floodplain 40,888.2 -> 40,969.3 ha
(+0.2%) on the same DEM. pars_meta.rds records the link run and where
the context layers came from (carried over from 2ef5c89: the shared
database was unreachable). The vignette states the source from it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
Credit link and fwapg for the bundled stream data in NOTICE and README (
5b1e45a 
#87)

NOTICE goes per attribute: upstream area from fwapg, precipitation and
modelled widths from ClimateBC through fwapg (#85), access and habitat
from link's default config. That config applies bcfishpass's override
tables, so a scoped ODbL credit stays. The README caption no longer
points at link's bcfishpass-config article.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
Fall back per context layer when the shared database lacks it (#87)
407ee26 
Code-check round 1: with PG_*_SHARE unset, frs_db_conn() reached fwapg,
every context query failed inside try(), and the script dropped all four
layers while recording them as fetched. Each layer now falls back to the
carried copy on a query error, or stops when there is none, and
pars_meta.rds records the source per layer. README no longer says fwapg
has upstream area on every segment (fwapg#33).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
Rebuild the stac-dem cache on the link-sourced test streams (#87)
2d5644d 
The 10 m guard count moves 28,727 -> 28,726 with the fwapg precipitation;
the lidar runs move by 0.04%. test-vignette_data.R passes on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
Keep the previous Parsnip gpkg until the new one is complete (#87)
875ae1d 
Code-check round 2: a run that stopped at a guard had already deleted
pars.gpkg, losing the context layers it would carry over. The gpkg is
now built in a tempfile and copied into place after the last layer, and
the carried layers' origin is tracked per layer, inherited from the
previous metadata, never taken from an uncommitted file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
Rebuild the Parsnip gpkg and meta with per-layer context provenance (#87
541cd59 
)

The final run from a clean tree records each carried context layer's
origin (2ef5c89, where they were last fetched). Both scripts now point
from their pinned run_id to the copy of it in inst/NOTICE.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
Pin link's recompute runs and stop trusting proxies for provenance (#87)
9f929ac 
Code-check round 3. link's recompute rewrites access and mapping codes
after the modelling run, so both scripts pin it too and NOTICE names it.
The fwapg tables read live are recorded by row count. The carried
layers' origin needs the gpkg and meta clean at one commit. A named
streams query error stops the run, the copy refuses to drop a previous
layer, and the gpkg is renamed into place rather than copied over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
Keep a rejected Parsnip gpkg where the error message says it is (#87)
0823eba 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
Rebuild the Parsnip meta with the pinned recompute and fwapg state (#87)
99a1817 
Final run at 0823eba: pars_meta.rds names the recompute that wrote
access and mapping codes (m1-...1b04b0, link a17e917) and the fwapg row
counts read; floodplain unchanged at 455,215 cells.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
Order the recompute after the run, and keep unknown origins unknown (#87
0ee7c55 
)

Code-check round 4. The recompute guard now requires the pinned
recompute to have started after the pinned modelling run. A layer the
previous run carried with an unknown origin stays NA rather than being
credited to the current commit. A rejected gpkg is kept outside inst/,
and a failed move cleans up its partial file. The meta comment no
longer credits habitat flags to the recompute.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
research: bundled stream data provenance (#87)
8f7d00e 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW
@NewGraphEnvironment @claude
Archive planning files for issue #87
f051bf7 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186ZefY3qMPiW8pwnwwMuxW

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Build the bundled and vignette data from our link-built tables, not bcfishpass

1 participant

@NewGraphEnvironment