Skip to content

feat: bound statements with a default command_timeout - #86

Merged
lesnik512 merged 3 commits into
mainfrom
fix/opt-in-command-timeout
Oct 28, 2026
Merged

lesnik512 merged 3 commits into
mainfrom
fix/opt-in-command-timeout

Conversation

@lesnik512

@lesnik512 lesnik512 commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Closes #85.

build_connection_factory used to bound only the connect. A statement on a pooled connection to a primary that stopped answering (a paused VM, a partition that drops packets) waited for TCP keepalive, which can take hours, so failover never reached the caller.

The factory now takes a keyword-only command_timeout: float | None = 60.0 and hands it to asyncpg. A command_timeout in the DSN query wins over the keyword, and None turns the bound off.

Behaviour change

This is on by default. Any statement that runs longer than 60 seconds now raises TimeoutError, and postgres_retry does not retry it, because the statement may have completed. Engines that run migrations or long reports need a higher value or command_timeout=None. With None, a frozen primary hangs the statement as before.

The asyncpg floor rises from 0.28 to 0.32. On asyncpg 0.30 and 0.31 the bound fires, but SQLAlchemy then discards the connection through asyncpg's close(), which waits with no bound for the server to acknowledge the cancel. A frozen primary never does, so the caller still hangs, only one step later. The floors CI jobs caught this on the first push (3.13 on 0.30, 3.14 and 3.14t on 0.31), and it reproduces locally with the hang lasting past 30s. asyncpg 0.32 puts the cancel wait and the disconnect under one deadline.

Changelog note: build_connection_factory now bounds each statement at 60s by default (command_timeout). Pass a larger value or None for long-running statements; a command_timeout in the DSN still takes precedence. db-retry now requires asyncpg>=0.32.

Why a default and not opt-in

An opt-in keyword defaulting to None was the other option. It changes nothing for existing users, but the people it misses are the ones who never read the failover docs, and they are exactly the ones who hang. faststream-outbox and faststream-redis-timers also put a default bound inside the library, using their lease. db-retry has no lease to borrow, so 60s is a chosen number: well above ordinary OLTP statements, and a minute instead of hours on a frozen primary. The reasoning is recorded in docs/adr/0003-statements-get-a-default-bound.md.

Tests

  • tests/test_command_timeout.py puts a small TCP proxy in front of Postgres. Its freeze() stops forwarding on the connections already open but lets new ones through, which is how a paused primary looks to a pool. With command_timeout=0.5, the statement on the frozen pooled connection raises TimeoutError and the next statement gets a fresh connection and succeeds. With command_timeout=None, the same test hits its 5s outer budget, which is the hang.
  • A mocked asyncpg.connect checks the precedence for single-host and multi-host DSNs, and that leaving the keyword out sends 60.0.
  • just test --cov=.: 73 passed, 100% coverage. Ruff, uvx ty@latest check and just adr-check pass.

Failover drill

tests_dr now passes command_timeout=3 as the keyword instead of the DSN param. Two just test-dr runs:

Scenario No bound command_timeout=3 Attempts Failed outer calls
kill not measured (a killed host fails fast, no frozen connection) 1.6s, 1.7s 2 0
pause hangs until the 180s test budget 5.7s, 5.7s 2 1 TimeoutError

With the 60s default and no keyword, the pause drill would recover after about a minute instead of hanging.

A statement on a pooled connection to a primary that stops answering hung
until TCP keepalive gave up. The keyword bounds each statement; a
command_timeout in the DSN query wins. Default stays None.
The bound is now on by default. None turns it off; a command_timeout in
the DSN still wins. Adds ADR 0003.
Before 0.32, close() after a command_timeout waits with no bound for a
cancel a frozen server never answers, so the caller still hangs.
@lesnik512
lesnik512 merged commit 846a7e1 into main Oct 28, 2026
16 checks passed
@lesnik512
lesnik512 deleted the fix/opt-in-command-timeout branch October 28, 2026 16:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Statements on a pooled connection to a frozen primary hang without command_timeout

1 participant