[fcollman]

Problem

• The dense missing root id lookup (/materialize/run/dense_lookup_root_ids/...) could not run:
  • one NULL row crashed every task ('int' object is not subscriptable);
  • more than one could not be sent at all (range is not JSON serializable);
  • over a wide id span, it queued one chord with a task per 500 ids, enough to run Redis out of memory.
• It could also leave other rows stale. It re-looked-up every row in the span, then moved the table's last_updated to now. The next update then skipped roots that had expired in between.
• There was no throttled way to re-look-up every root of a table, for when roots may be stale without knowing which.
• Each hourly update looked up NULL roots repeatedly. The rows with a NULL root were added to every expired-root chunk query. When nothing had expired, they were skipped entirely.

Change

Two repair tools, both through update_root_ids, so both write only rows that changed.

Both repairs leave last_updated alone. They look roots up as of the table's last_updated (or as of now if the table has none) and never move it. The next regular update therefore still covers everything that expired since then.

• Full re-lookup (lookup_all_root_ids): every row of a table is looked up from its supervoxel id.
  • New per-table endpoint: POST /materialize/run/lookup_all_root_ids/datastack/<ds>/table/<table>.
  • The existing POST /materialize/run/update_roots/datastack/<ds>?lookup_all_root_ids=true does the same for a whole datastack.
  • Chunks are 50,000 rows, and each chunk task streams its rows in chunk_size batches.
• NULL roots: every regular update now looks up the rows with a supervoxel id and a NULL root in one chunk of their own, MISSING_ROOTS, alongside the expired roots, even when nothing expired.
  • The existing dense (per datastack) and sparse (per table) missing-root endpoints run only that chunk, as a repair (only_missing_roots). They do not search for expired roots.
  • The hourly and build chains no longer need the separate, gated find_missing_root_ids_workflow step.
• Throttling: update_root_ids_workflow sends its chunk tasks through a new run_with_limit, with at most UPDATE_ROOT_IDS_MAX_IN_FLIGHT (default 20) unfinished at a time, instead of one chord of all of them. Each chunk task still checks the process queue length before queuing each batch.
• get_new_root_ids writes only rows whose root changed. It no longer writes the supervoxel column back.
• update_metadata runs only after every chunk of a regular update succeeded, as before. A repair never runs it.
• monitor_task_states polls only the tasks still pending.

Testing

New tests:

• the chunk queries: expired roots without OR IS NULL, the NULL-root chunk, and full-lookup id ranges, including the open last chunk;
• the workflow's chunk sequence in each mode, including that NULL roots are filled when nothing expired, and that a failed chunk leaves last_updated alone;
• that repairs look up as of last_updated, fall back to now when there is none, and never call update_metadata;
• that only changed rows are written;
• that run_with_limit never exceeds its limit;
• that the dense and sparse entry points run the update.

Full suite on fresh containers: 449 passed, 2 skipped. Ruff is clean.

🤖 Generated with Claude Code

[fcollman]

@copilot resolve the merge conflicts in this pull request

[Copilot]

@copilot resolve the merge conflicts in this pull request

Resolved in 77531ff by merging current master, preserving both its scoped test mocks and the root-ID repair tests. Syntax, whitespace, and secret checks passed. Pytest/Ruff were unavailable locally; CodeQL reported four existing upload API findings unchanged from master.