Skip to content

Statements on a pooled connection to a frozen primary hang without command_timeout #85

Closed

Description

@lesnik512

build_connection_factory(url, timeout) bounds the connect, but db-retry sets nothing that bounds a statement. When the primary freezes (process stopped, host unreachable, packets dropped) while the pool holds a connection to it, the next statement on that connection never returns. The caller waits until TCP keepalive closes the socket, about two hours with Linux defaults. postgres_retry never sees an exception, so it cannot help.

The failover drill from #84 shows this. With command_timeout in the DSN the pause scenario passes: the first statement times out after 3s, and the caller's next call reaches the promoted replica in 6 to 10s. Without it the scenario hangs.

Reproduction

  1. In tests_dr/test_failover.py, change
    dsn = f"{DB_DSN}&command_timeout={COMMAND_TIMEOUT_SECONDS}" to dsn = DB_DSN.
  2. Run:
    docker compose -f docker-compose.dr.yml run --rm dr uv run pytest tests_dr -s -k pause
    docker compose -f docker-compose.dr.yml down --remove-orphans --volumes
  3. The INSERT in write(1), the first write after docker pause, blocks on the pooled connection until the 180s test budget cancels it.

What a bounded statement then does

With command_timeout set, asyncpg raises TimeoutError. postgres_retry does not retry it, which matches CONTEXT.md: the server may have run the statement, so it is not a retriable error. The caller has to decide whether to call again.

Options

  • Document that a multi-host setup needs command_timeout (in the DSN query or connect_args), next to build_connection_factory in the README.
  • Have build_connection_factory set a default command_timeout when the URL has none, for example equal to timeout or a separate argument.

The second option changes behaviour for anyone running long statements, so it needs a decision before any code. A fix starts with a failing test, and the drill already gives one.

Related: #67 covers connect-side timeouts in asyncpg's multi-host connect. This issue is about statements on a connection that is already open.

Activity

  1. added
    needs-triageMaintainer needs to evaluate this issue
    enhancementNew feature or request
    and removed
    needs-triageMaintainer needs to evaluate this issue
    on Oct 28, 2026
  2. lesnik512 commented on Oct 28, 2026

    @lesnik512
    MemberAuthor

    Decision: a default bound. build_connection_factory gets a keyword-only command_timeout: float | None = 60.0. A command_timeout in the DSN query takes precedence over it, and None turns the bound off. This is a behaviour change: a statement that runs longer than 60s now raises TimeoutError, which postgres_retry does not retry. An opt-in default of None was considered and declined, because the callers it misses are exactly the ones who hang. The reasoning is in docs/adr/0003.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions