Skip to content

feat: queue a rearm when extending a timer whose handler is running - #256

Merged
lesnik512 merged 1 commit into
feat/timer-extendfrom
feat/timer-rearm
Oct 26, 2026
Merged

lesnik512 merged 1 commit into
feat/timer-extendfrom
feat/timer-rearm

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Stacked on #254. With this change, publish(..., timer_conflict="extend") also works on a timer whose handler is already running. In #254 that case returned None and the heartbeat was lost. That broke the operator-timeout case, where a ping that lands while the expiry runs can't arm the next timer, and it meant a handler couldn't rearm its own timer either.

Contract

When the extend hits a leased row, it queues a rearm in three new nullable columns: rearm_at, rearm_payload and rearm_headers. It returns the row id and leaves the running instance alone: the lease, the claimed deadline and the payload are unchanged. A rearm only moves later, using the same rule as a pending extend, and an extend whose deadline isn't later than the running deadline queues nothing.

How each terminal write handles a queued rearm:

Run ends with Result
Success (single or batched delete) The row becomes a pending timer at the rearm deadline. Retry history and deliveries_count are reset.
Retry The rearm replaces the retry. deliveries_count is kept, so max_deliveries still stops a handler that fails on every run.
Terminal failure with a DLQ The failed run is archived and the rearm is applied, in one statement.
Expired lease Redelivered as before. The rearm applies when that run ends.

cancel_timer now locks the row first. It deletes an unleased row, or drops the rearm queued on a leased one, and still never deletes a leased row. Locking first closes a race where a run could finish and apply its rearm between the cancel's check and its write.

Cost

The common path is unchanged. A completion is still one DELETE ... WHERE rearm_at IS NULL, and only when it matches nothing does a follow-up UPDATE apply the rearm. That fallback runs only for rows with a queued rearm or a lost lease. just bench-check passes against the existing baseline. The same fallback pattern covers the batched delete and the DLQ path. cancel_timer now takes two statements instead of one.

An extend and a completion are always ordered. The completion's DELETE waits on the row lock the extend holds, then re-checks rearm_at and falls through to applying it. An integration test exercises that order.

Schema migration

This change needs a migration: three nullable columns plus outbox_rearm_ck (rearm_at IS NULL OR acquired_token IS NOT NULL). docs/operations/alembic.md gains an "Upgrading to 0.19" section. It adds the columns without rewriting the table and adds the CHECK as NOT VALID before validating it. I ran that Alembic snippet against a table in the 0.18 shape, and validate_schema() passed afterwards. Run the migration before deploying, because every completion reads rearm_at. Older versions ignore the columns, and validate_schema() doesn't flag extra columns. A missing rearm column error now links to the upgrade section. The release number 0.19 is my assumption.

Tests

  • Contract tests on both adapters:
    • rearm queued on a fresh lease and on an expired one;
    • a rearm never moves earlier, and nothing is queued behind a later claimed deadline;
    • rearm applied by single delete, batched delete and DLQ delete;
    • a lost lease keeps the rearm;
    • a retry yields to the rearm and keeps deliveries_count;
    • cancel drops a queued rearm and returns False when there is none.
  • Real Postgres:
    • the completion waits on an uncommitted extend and applies it;
    • the claim/extend race now queues a rearm;
    • a ping during a running expiry fires the timer again;
    • a handler rearms its own timer three times;
    • missing rearm CHECK or columns are reported.
  • Fake broker: a handler that extends its own timer leaves the row pending, and sync mode does not dispatch it again.
  • just test: 817 passed with the coverage gate on. just bench-check: OK.

What private-users still owns

A rearm doesn't undo what the running handler did. The expiry handler still has to re-check last activity in the same transaction as the OFFLINE transition. With this PR, a ping that lands during the expiry is no longer lost: the timer fires again for the newer deadline.

@github-actions

Copy link
Copy Markdown

Benchmark gate

✅ gate passed

scenario msg/s delete/msg WALrec/msg WALB/msg fpi upd del dead_tup
consumer/w1/b10 2045 1.000 6.71 921 242 5000 5000 10000
consumer/w1/b100 2148 1.000 7.11 1154 243 5000 5000 10000
consumer/w2/b10 2771 1.000 6.72 923 242 5000 5000 10000
consumer/w2/b100 3261 1.000 7.11 1154 243 5000 5000 10000
consumer/w4/b10 2774 1.000 6.74 946 242 5000 5000 10000
consumer/w4/b100 3366 1.000 7.11 1152 243 5000 5000 10000
consumer/w1/b100/tfbs100 11250 0.010 6.11 1130 243 5000 5000 10000
consumer/w1/b100/interval1-10 2401 1.000 7.11 1154 243 5000 5000 10000
producer/w1/b100 6312 0.000 3.04 597 1 0 0 0
producer/w1/b100/ppt1 1809 0.000 4.04 635 0 0 0 0

Gated (fails the build): delete_calls + tuple counters (upd/del/ins) + the producer's insert_calls, exact; select_calls and other_calls within +2; wal_records within a 10% band; fetch_calls within 2x. msg/s, WAL bytes and total calls are informational (timing/FPI noise).

@lesnik512
lesnik512 merged commit 7d05f82 into feat/timer-extend Oct 26, 2026
15 checks passed
@lesnik512
lesnik512 deleted the feat/timer-rearm branch October 26, 2026 14:56
@lesnik512

Copy link
Copy Markdown
Member Author

Folded into #254: feat/timer-extend was fast-forwarded to this branch's head (7d05f82), so GitHub marked this PR merged into that branch. Review continues on #254.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant