Repository navigation
Root ID updates: dense missing-root lookup, repairs that keep last_updated, and an update log - #263
Merged
Merged
Conversation
…dated
The dense lookup queued one chord over the whole span between the lowest and highest
row with a NULL root, 500 ids per task, all at once. A large span filled Redis; a span
of one row crashed every task ('int' object is not subscriptable) and any longer span
could not be sent at all (range is not JSON serializable). Each task also re-looked-up
every row in its range, not only the NULL ones, and the chain ended with
update_metadata, moving the table's last_updated to now: the next update_root_ids run
then skipped the roots of other rows that expired in between, leaving them stale.
It now streams the ids of rows with a supervoxel id and a NULL root, and queues small
batches as the process queue drains (batch_missing_root_ids_query, throttled), like the
sparse lookup but without its "table modified recently" check. Roots are looked up at
the table's last update, which stays as it is.
monitor_task_states polls only the tasks still pending, instead of every task every
0.2 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ar update Two tools, both through update_root_ids (so both write only what changed and keep the table's last_updated right): - Full re-lookup, for a table whose roots may be stale without knowing which: every row is looked up from its supervoxel id (lookup_all_root_ids). New per-table endpoint /materialize/run/lookup_all_root_ids/datastack/<ds>/table/<table>; the existing update_roots?lookup_all_root_ids=true does it for a whole datastack. - NULL roots: every update now looks up the rows with a supervoxel id and a NULL root in one chunk of their own (they used to be added to every expired-root chunk, and were skipped entirely when nothing had expired). The dense and sparse missing-root endpoints now run that update right away; the hourly and build chains no longer need the separate (gated) missing-root step. update_root_ids_workflow queues its chunk tasks with at most UPDATE_ROOT_IDS_MAX_IN_FLIGHT (default 20) unfinished at a time (run_with_limit) instead of one chord of all of them, so a full re-lookup of a large table, in 50k-row chunks, cannot flood Redis. get_new_root_ids writes only rows whose root changed. update_metadata runs only after every chunk succeeded, as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…of it A full re-lookup (lookup_all_root_ids) or a NULL fill (only_missing_roots, from the dense and sparse missing-root endpoints) is a repair: it looks roots up as of the table's last_updated (now if it has none) and leaves last_updated as it is, so the next regular update still covers everything that expired since. The NULL fill no longer searches for expired roots. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…y Delta table
One append-only Delta table per annotation table, root_id_updates/{datastack}/{table}/:
one commit per run (partitioned by run_id), rows (id, root_column, supervoxel_id,
old_root_id, new_root_id, start_time_stamp, end_time_stamp) sorted by (root_column, id),
id delta-encoded, zstd, 16k-row row groups; plus a _manifest table with one row per run,
including runs that changed nothing or failed, so the windows chain. Off unless
ROOT_ID_UPDATE_LOG names the datastack / tables. Only rows whose root changed are
recorded.
(The root log history up to vRootLog12, squashed onto the throttled root id lookups.)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The root ID update work that minnie and ltv7 have been running as
vRootLog*images, on current master.Changes
last_updated.last_updatedand look up roots as of it.update_root_idsrun is recorded as an append-only Delta table.Relation to other branches
root-id-update-log(deployed asvRootLog17) without that branch's earlier copies of tests: stop test_ingest_new_annotations leaking a mock cloudvolume #258, Upload: wait up to 6 h for the Cloud SQL CSV import (was 20 min) #259 and Upload: run the long steps in orchestration job pods, acked late and resumable #260. Master already has those as squash merges, so merging the branch directly conflicts.root-id-update-log, this branch differs only by Spatial lookup: 2000-point sub-batches (was 500); rate shown per minute #261.Conflict resolution
Every commit conflicted in
tests/test_ingest_new_annotations.py: each edits the import list that #258 moved into a block that mocks modules only during the import. In each case I kept master's block and used the commit's import names.Testing
Full suite on a fresh PostGIS: 575 passed, 2 skipped.
🤖 Generated with Claude Code