Skip to content

Keep WAL on the DR primary so concurrent standby basebackups don't race - #267

Merged
lesnik512 merged 1 commit into
mainfrom
dr-wal-keep-size
Oct 28, 2026
Merged

lesnik512 merged 1 commit into
mainfrom
dr-wal-keep-size

Conversation

@lesnik512

Copy link
Copy Markdown
Member

docker-compose.dr.yml starts two standbys, pg-replica and pg-standby, and each runs pg_basebackup -X stream against the primary at the same time. The second backup forces a checkpoint, and with no wal_keep_size the primary is free to recycle a WAL segment the first backup is still streaming. That backup then dies and the standby container exits 1:

outbox-dr-replica  | pg_basebackup: error: unexpected termination of replication stream: ERROR:  requested WAL segment 000000010000000000000002 has already been removed
outbox-dr-replica  | pg_basebackup: error: background process terminated unexpectedly

This PR starts the primary with wal_keep_size=256MB, so it holds enough WAL for both backups to finish. db-retry hit the same race in modern-python/db-retry#84 and fixed it with the same setting.

Each start below ran docker compose -f docker-compose.dr.yml down --remove-orphans --volumes, then up -d --wait pg-replica pg-standby, and grepped the standby logs for "has already been removed".

Starts Failed WAL segment removed
main 15 7 7
this PR 15 0 0

Every failure on main was this error. With the fix, just test-dr passes both scenarios (kill and blackhole), and just lint is clean.

@github-actions

Copy link
Copy Markdown

Benchmark gate

✅ gate passed

scenario msg/s delete/msg WALrec/msg WALB/msg fpi upd del dead_tup
consumer/w1/b10 1313 1.000 6.72 922 242 5000 5000 10000
consumer/w1/b100 1434 1.000 7.11 1154 243 5000 5000 10000
consumer/w2/b10 1574 1.000 6.71 922 242 5000 5000 10000
consumer/w2/b100 1926 1.000 7.11 1154 243 5000 5000 10000
consumer/w4/b10 1621 1.000 6.76 950 242 5000 5000 10000
consumer/w4/b100 1957 1.000 7.11 1153 243 5000 5000 10000
consumer/w1/b100/tfbs100 5651 0.010 6.11 1130 243 5000 5000 10000
consumer/w1/b100/interval1-10 1445 1.000 7.17 1196 265 5000 5000 10000
producer/w1/b100 3021 0.000 3.04 597 1 0 0 0
producer/w1/b100/ppt1 1015 0.000 4.04 635 0 0 0 0

Gated (fails the build): delete_calls + tuple counters (upd/del/ins) + the producer's insert_calls, exact; select_calls and other_calls within +2; wal_records within a 10% band; fetch_calls within 2x. msg/s, WAL bytes and total calls are informational (timing/FPI noise).

@lesnik512
lesnik512 merged commit 0e8c173 into main Oct 28, 2026
15 checks passed
@lesnik512
lesnik512 deleted the dr-wal-keep-size branch October 28, 2026 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant