[fcollman]

On minnie on 2026-10-25, a 337M-row upload was lost partway through upload_to_db, which then ran on the KEDA-scaled workflow queue:

• Scale-down: 8 minutes into the Cloud SQL import, the workflow queue was empty, so KEDA scaled the producer down.
• Kill: the preStop hook failed, and the 6000 s grace period expired in the middle of the second of six index builds.
• Lost task: the task had been acked on receipt, so nothing ran the remaining indexes, the spatial lookup or the transfer.

Neither keeping a minimum replica count nor a preStop hook can protect a task that runs for hours.

Changes

• The long steps run in orchestration job pods. upload_to_db, cluster_staging_tables and transfer_to_production move to the orchestration queue. Each runs in a KEDA job pod of its own, on a non-preemptible node, that runs the one task and exits. Nothing scales it down mid-task.
• They are acked late and resume on redelivery. A pod can still be lost (node failure, the job deadline), so its task must be safe to run again:
  • upload_to_db records its Cloud SQL import (operation name, running or done) per job in Redis.
  • A redelivery skips creating the table, truncating it and dropping its indexes.
  • It waits on the recorded import, which keeps running without its client, or skips it if the import is done.
  • It then waits for any CREATE INDEX the dead attempt left running before building the indexes, which use IF NOT EXISTS.
  • A failed import fails the job.
  • Cluster and transfer were already safe to rerun.
• A job pod stops consuming when its last allowed task starts. It never claims a second message and holds it unacked until the visibility timeout restores it. This is one explanation for the Oct 22–23 duplicate process_csv runs.
• The old workflow:* names stay registered for one release, for messages queued under them during a deploy.

Depends on the chart

cave-helm-charts #12 raises the broker visibility timeout (materialize.celery.visibilityTimeout) to 90000 s by default, longer than the 86400 s job deadline. Without it, a late-acked task running longer than 6 h would be handed to a second pod while it still runs. The orchestration pods keep the default unacked key on purpose: the metrics exporter consumes on that key and restores a dead pod's message, and KEDA then starts a job pod for it. Deploy both together.

Tradeoff

If a job pod's container crashes mid-task (OOM), its message is restored only after the visibility timeout (~25 h), and then it resumes. Today such a task is simply lost.

Tests

• test_upload_long_tasks.py:
  • a fresh run records its import;
  • a redelivery waits on a running import instead of importing again, and skips the table setup and index drop;
  • a redelivery after a finished import goes straight to the indexes;
  • a failed import fails the job;
  • the wait for leftover index builds;
  • autoshutdown stops consuming when the last task starts.
• Routing tests updated: the three steps go to orchestration with late ack, and the aliases go to workflow.
• The full suite passes locally on fresh PostGIS 18 and Redis: 523 passed, 2 skipped.

🤖 Generated with Claude Code