Repository navigation
[local-fix-worker-pause-injected-and-retry-drai] fix(worker): pause injected and retry drains while the harness chain is exhausted - #368
Conversation
…worker): pause injected and retry drains while the harness chain is exhausted
Make the injected-task and retry-queue acquirer tests hermetic by resetting the process-wide worker failover before and after every test, so a leaked/preloaded exhausted controller can no longer silently pause their drains. Add production wiring coverage: assert the exported isHarnessChainPaused tracks live WorkerFailover state, and drive buildInjectedTaskAcquirer/buildRetryQueueAcquirer with a real exhausted failover to prove rows are not claimed until the window elapses. The executor's own pause guard is disabled in those tests so only the acquirer-level wiring can keep the rows idle.
Artifacts & verificationRun: 39 · Commit: Change explanationProduced by Pause injected and retry drains while the harness chain is exhaustedWhy this change is neededThe task states that when every harness in the chain is usage-limited, What changed
Validation actually performedObserved, from the pipeline evidence:
Not observed: no daemon run, no before/after log-line measurement, and no reproduction of the original 1,200-line flood. Material risks
Where reviewers should focus
Observed evidence vs inference
Visual explanationProduced by Show artifactSummaryWhen every harness in the chain is usage-limited, the injected and retry drains now leave their rows untouched instead of claiming, running, and requeueing the same row on every 5-second tick. This removes the repeated EvidenceThe change is one gate per acquirer, wired to the existing failover singleton: The retry acquirer also stops setting Validation observed in the supplied pipeline evidence:
Before/after results for the actual daemon log flood were not captured. No live usage-limit window was reproduced, and the 1,200-line counts come from the task report, not from this change's validation. Merge DangerReverting any one of the three source files restores the old per-tick claim/requeue loop and the log flood, but nothing else breaks: the rows are durable in SQLite, the executor guard is unchanged, and there is no schema, config, or API change. Blast radius is small and contained — only The real risk is not the revert, it is the pause state itself: if Verification reportProduced by Show artifactAuto-review summaryProduced by Show artifact |
Why
Task local-fix-worker-pause-injected-and-retry-drai: fix(worker): pause injected and retry drains while the harness chain is exhausted
When every harness in the failover chain was usage-limited or cooling down, the injected-task and dashboard-retry drains still claimed their rows every 5 seconds, immediately got
"deferred"back from the fleet executor, requeued the rows, and logged arunning/deferredpair each cycle. One injected task produced over a thousand identical log pairs per pause window. The fix makes both short-interval drains idle while the chain is exhausted, leaving the existingonPause/onResumelines as the only per-window output.What changed
packages/code/src/lib/workspace/injected-acquirer.ts: added an injectableisPaused?: () => booleanoption, defaulting togetWorkerFailover()?.allExhausted() ?? false.tick()now returns before claiming when paused, so it neither runs, requeues, nor logs anything. The existing per-task"deferred"handling (repo capacity/concurrency) is unchanged.packages/code/src/lib/workspace/retry-acquirer.ts: sameisPausedoption, default, and early return at the top oftick().packages/code/src/lib/workspace/workspace-worker.ts: addedisHarnessChainPaused()resolving the live failover state and wired it into both drains. Extracted the inline retry-acquirer construction intobuildRetryQueueAcquirer()(matchingbuildInjectedTaskAcquirer) so production wiring is explicit andrunWorkspaceWorkerstays under the lint line limit.packages/code/tests/injected-acquirer.test.tsandpackages/code/tests/retry-queue-acquirer.test.ts: each gained a test that runs 50 paused ticks, asserts the executor is never called,console.logis never called, the row stays pending/queued, then verifies the first unpaused tick drains it to done.fleet-executor.tswas intentionally left untouched: its pre-gitisHarnessExhausted()guard remains defense in depth.Scope
Included: the two acquirers' tick gates, failover predicate plumbing in
workspace-worker.ts, and the unit tests above.Excluded: failover/usage-limit semantics, drain interval changes, suppressing the repo-capacity
"deferred"log, worktree retention, and dashboard changes.Tradeoffs
tick(). Work that starts between that check and the executor call is still caught by the executor's existing guard and may emit onerunning/deferredpair, which is the intended defense in depth rather than a second check per claimed row.workspace-worker.tsalso wires the shared helper explicitly.buildRetryQueueAcquireradds an exported builder and spreadsFleetExecutorDepsinstead of listing fields, but keeps the executor configuration identical and gives the worker function headroom undermax-lines-per-function.Blast Radius
runWorkspaceWorkernow builds both drains through the failover-aware builders; behavior changes only whileallExhausted()is true.nullfailover →falsewhen failover has not been started.buildInjectedTaskAcquirercallers (workspace worker andworkspace-worker.test.ts) gain the pause gate; existing tests still pass."deferred"contract, supervisor capacity handling, injected-store requeue semantics, and retry-store queue semantics are unchanged.Verification
cd packages/code && bun run test injected-acquirer.test.ts retry-queue-acquirer.test.ts→ 22 pass, 0 fail (63 expect calls).cd packages/code && bun run test→ 2527 pass, 0 fail across 168 files.cd packages/code && bun run typecheck→ passed.bun run typecheck→ Turbo reported 12/12 packages successful. The chainedtypecheck:examplesstep initially hit a transient nono sandboxIO error: Permission denied/ sandbox-state drift; rerunningbunx tsc --noEmit -p examples/pipeline-pluginssucceeded with no output.bunx oxfmt --checkon the five changed files → all correctly formatted.bunx oxlint packages/code/src/lib/workspace packages/code/tests→ no findings.This PR was automatically created by @devintern/code