Skip to content

test: add opt-in failover drills (just test-dr) - #84

Merged
lesnik512 merged 1 commit into
mainfrom
test/dr-failover-drills
Oct 28, 2026
Merged

lesnik512 merged 1 commit into
mainfrom
test/dr-failover-drills

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Adds just test-dr, an opt-in suite that runs build_connection_factory through a real failover. It follows faststream-outbox/tests_dr and faststream-redis-timers/tests_dr. Every failover test in tests/ monkeypatches asyncpg.connect, so this is the first run of the multi-host path against real servers.

docker-compose.dr.yml starts a primary, a streaming replica, and a second standby, and gives the dr container one multi-host DSN covering all three. The drill builds a write engine from build_db_dsn(...) and a read engine from build_db_dsn(..., use_replica=True), both with build_connection_factory as async_creator, and writes inside Transaction under postgres_retry. It then kills or pauses the primary, promotes the replica with pg_promote(), and repeats the write until it lands. It checks that:

  • the write reaches the promoted replica within 60s;
  • reads with use_replica=True succeed before and after the promotion;
  • 10 fresh connections from the factory all land on the promoted replica and none on the standby, which is still in recovery afterwards.

The justfile starts a fresh cluster for each scenario. tests_dr/ is outside testpaths and omitted from coverage, and the ruff test ignores now include it. CI does not run it.

Results

Three runs of just test-dr on Docker Desktop (macOS, arm64):

Scenario Time to new primary postgres_retry attempts Failed outer calls Host-by-host fallbacks
kill 2.2s / 1.6s / 1.3s 2 / 2 / 2 0 0 / 0 / 0
pause 7.7s / 5.7s / 9.7s 2 / 2 / 2 1 TimeoutError each run 7 / 5 / 3

With the primary killed, the first attempt fails on the pooled connection to the dead host. SQLAlchemy marks the connection invalidated, and postgres_retry reruns the body on a new connection that reaches the promoted replica.

With the primary paused, the first statement on the pooled connection hits command_timeout=3 and raises TimeoutError. postgres_retry does not retry it, as CONTEXT.md says, since the server may have committed. The drill's own loop calls again and the next call connects. A host-by-host fallback happens whenever the shuffled bulk attempt puts the frozen host first and spends the whole connect timeout on it.

Found by the drill, not fixed here

  • Without command_timeout in the DSN, the pause drill hangs. db-retry bounds the connect but not statements, so the first statement on a pooled connection to a frozen primary waits until the test budget cancels it (in production, until TCP keepalive gives up, about two hours by default). To reproduce, change dsn = f"{DB_DSN}&command_timeout={COMMAND_TIMEOUT_SECONDS}" to dsn = DB_DSN in tests_dr/test_failover.py and run docker compose -f docker-compose.dr.yml run --rm dr uv run pytest tests_dr -s -k pause. Whether db-retry should set a default or document the requirement belongs in its own issue, and a fix would start with a failing test.
  • The primary runs with wal_keep_size=256MB. Without it, the two standbys' concurrent pg_basebackup runs sometimes failed with "requested WAL segment ... has already been removed" (1 in 5 cluster starts). faststream-outbox's DR compose has the same layout.

Test plan

  • just test-dr: three runs, both scenarios passed each time
  • just test --cov=.: 63 passed, 100% coverage
  • just lint-ci: clean

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant