Skip to content

substituter: back off and time out stalled pack reads - #189

Draft
lassulus wants to merge 1 commit into
Mic92:mainfrom
lassulus:substituter-stall-retry
Draft

lassulus wants to merge 1 commit into
Mic92:mainfrom
lassulus:substituter-stall-retry

Conversation

@lassulus

Copy link
Copy Markdown

On a superconfig CI run, one pack Range read got a 503 from the GHA cache storage (ServerBusy: Egress is over the account limit). The daemon logged transient error reading pack ..., retrying once and then nothing for 8 minutes, while every Nix request for that NAR got an empty reply until the build failed:

13:26:28  hestia substituter: transient error reading pack pack-b67b497f..., retrying: unexpected HTTP status 503 ... ServerBusy: Egress is over the account limit
13:27:42 .. 13:32:32  warning/error: unable to download 'http://127.0.0.1:49170/nar/1d528r60....nar' (empty reply)

read_pack_range retries immediately, and the serve client has only a connect timeout, so the retried read could hang forever while holding the per-path lock.

This PR:

  • retries transient pack read failures with the same doubling backoff the blob helpers use (TRANSIENT_RETRY_DELAY, now pub)
  • gives the substituter its own client with a 60 s read_timeout (an idle timeout), so a stalled read becomes a transient error and gets retried. The shared client is unchanged, because reqwest's read timer already runs while the request body is being sent, which would cut off slow pack uploads.

cargo clippy --all-targets -D warnings and cargo test pass. The stall path itself isn't covered by a test.

A throttled GHA cache storage account (Azure "503 ServerBusy: Egress is
over the account limit") answered one pack Range read with 503. The
substituter retried the same URL immediately, and that retry never
returned: the client has only a connect timeout. The hung read held the
per-path lock, so every later Nix request for that NAR got an empty
reply until the build gave up.

Retry transient pack read failures with the same doubling backoff the
blob helpers use, and give the substituter its own HTTP client with a
60 s read (idle) timeout so a stalled read becomes a retried transient
error. The shared client stays without it: there the timer would also
run while a large pack upload is still sending.

Assisted-by: Hermes:claude-opus-5-5
@lassulus
lassulus marked this pull request as draft November 12, 2026 13:58
@lassulus

Copy link
Copy Markdown
Author

Status update, keeping this as a draft: we ran this build in our CI, and it did not fix the failure that prompted it. The macOS job still gets Empty reply from server for the same large NAR roughly every 40 s. Two things from that run:

  • Some of the failing Nix attempts had no 503 at all.
  • The 60 s stall timeout never fired, because the connection is closed earlier.

So the 503 is not what causes that failure. We're collecting more data (NAR size, daemon RSS, memory pressure) and will update or close this PR based on what we find.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant