gateway: concurrent ACME rotation is only best-effort serialized #1008

Follow-up to #935.

RotateAcmeCredentials is serialized across nodes by a TTL lock in WaveKV (global/acme_rotation_lock). WaveKV is last-writer-wins without compare-and-swap, so the lock is acquired by read-then-write: two nodes calling the RPC within a replication gap can both acquire it and rotate concurrently.

Impact

If two rotations interleave, CAA records end up pinned to one node's new account while LWW keeps the other node's credential in KV. Since #935 the state is recoverable — the published credential wins LWW and one SetCaa run re-pins every domain to it — but issuance is broken until an operator notices and intervenes.

The constraint "rotate through one gateway at a time" is advisory only: the admin endpoint uses a shared bearer token with no per-method authorization, so nothing enforces it.

Possible directions

• Add CAS (or a fenced-lock primitive) to WaveKV and make the rotation lock a real mutex.
• Route rotation to a designated leader node instead of accepting it on any gateway.
• At minimum: automatic post-rotation verification that the published credential's accounturi matches every domain's CAA, alerting on divergence (see the CAA reconciliation issue).