Repository navigation
Conversation
A throttled GHA cache storage account (Azure "503 ServerBusy: Egress is over the account limit") answered one pack Range read with 503. The substituter retried the same URL immediately, and that retry never returned: the client has only a connect timeout. The hung read held the per-path lock, so every later Nix request for that NAR got an empty reply until the build gave up. Retry transient pack read failures with the same doubling backoff the blob helpers use, and give the substituter its own HTTP client with a 60 s read (idle) timeout so a stalled read becomes a retried transient error. The shared client stays without it: there the timer would also run while a large pack upload is still sending. Assisted-by: Hermes:claude-opus-5-5
lassulus
marked this pull request as draft
November 12, 2026 13:58
Author
|
Status update, keeping this as a draft: we ran this build in our CI, and it did not fix the failure that prompted it. The macOS job still gets
So the 503 is not what causes that failure. We're collecting more data (NAR size, daemon RSS, memory pressure) and will update or close this PR based on what we find. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On a superconfig CI run, one pack Range read got a 503 from the GHA cache storage (
ServerBusy: Egress is over the account limit). The daemon loggedtransient error reading pack ..., retryingonce and then nothing for 8 minutes, while every Nix request for that NAR got an empty reply until the build failed:read_pack_rangeretries immediately, and the serve client has only a connect timeout, so the retried read could hang forever while holding the per-path lock.This PR:
TRANSIENT_RETRY_DELAY, nowpub)read_timeout(an idle timeout), so a stalled read becomes a transient error and gets retried. The shared client is unchanged, because reqwest's read timer already runs while the request body is being sent, which would cut off slow pack uploads.cargo clippy --all-targets -D warningsandcargo testpass. The stall path itself isn't covered by a test.