Skip to content

Finish competing-body dedup (#134): review fixes for #130 - #135

Open
zmanian wants to merge 35 commits into
mainfrom
claude/project-thread-erte2i
Open

zmanian wants to merge 35 commits into
mainfrom
claude/project-thread-erte2i

Conversation

@zmanian

@zmanian zmanian commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This finishes #134 (competing-body dedup for #130). It is #134's branch at 19bad0c plus fixes for the review of that head. #134 itself is untouched.

Before: a replayed or failed copy of a block could spend the legacy gate's 8-call budget and shut the real block out. The new gate also replaced main's gate for everyone, with no switch. On the relay side, one key could hold every anchor for a block, a set still arriving could be evicted, and held copies were retried inline on the sidecar's receive loop.

After: with every switch off, the gate behaves exactly as main's does. The competing-body gate is opt-in ([submit_gate] competing_bodies = true, and it is forced on with the bound raw switch). Replays and transient errors can't lock a block out. All of the relay and sidecar review items below are fixed and each has a test.

How: the gate work is in its own commits (f2ace79, plus the release wiring in c0c28d9), so it can be split into a separate PR if you'd rather land it first. The relay fixes and the sidecar and CI fixes were merged in from two branches (d27f8bb and the following merge).

Changes

Must-fix

  • 1. Legacy 8-call lockout. The attempt counter is gone. Only distinct bodies that Zebra answered, or that are in flight, count against strict_bodies_per_block (default 4). Transient errors count for nothing. A body Zebra refused is remembered for the dedup window (300 s), so a replay is turned away without another Zebra call. This replaces the PR's test that pinned the 8-call lockout.
  • 2. Unswitched gate change. The gate now sits behind competing_bodies, which is off by default. With it off, it is main's gate: dedup is recorded on accept and the lock is held across the Zebra call. There is a new end-to-end test on the real binary with every switch off (tests/legacy_gate_e2e.rs). The ingress legacy wire keeps main's repeat rule for a copy that arrives while another is in flight.
  • 3. Anchor quota. Each key can hold at most 2 bodies of a block. Headers more than 30 min older than the newest header that passed PoW are refused. A receiver now tells its relays when Zebra refuses a bound body, so that body's place is freed for a rival body.
  • 4. Receiver set eviction. A set that is still arriving (a segment within the last 5 s) is never evicted. A newcomer is shelved and retried instead.
  • 5. Held-copy retries on the receive loop. Released copies now start as bound tasks from a bounded retry queue. There is a stall test for both legacy and compact copies.

Should-fix

  • 6. Capability negotiation. The client sends a bound hello and sends bound objects only to relays that answer it. A bound send with no capable relay is refused at the sender.
  • 7. The node now records a bound object as sent only after it has actually gone out.
  • 8. The node no longer sends a bound object back to the connection it came from.
  • 9. dedup_capacity = 0 is refused when the config loads.
  • 10. Dry runs are kept apart from the live gate. A dry run is no longer counted as delivered.
  • 11. Dual mode. In dual mode, the legacy and bound sends share one egress pace. If the legacy copy leaves and the bound copy fails, the claim is committed, so later deliveries don't resend the full legacy copy.
  • 12. Cheap pool holding. A node keeps an open object for 5 s (it was 30 s). Objects waiting for their first segment give way to a session holding less.
  • 13. A full spare chunk is decoded once room frees.

Before merge

Nits

  • Zebra's inconclusive result gets its own rejection class and metric.
  • BoundTaskLimits can be set in config or on the command line (bound_tasks_running, bound_tasks_budget_secs, bound_tasks_held; defaults unchanged).
  • Bound usage is exported as Prometheus metrics.
  • BlockSender::for_tests is behind test-support.
  • send_bound_raw_block times out after 10 s.
  • contain logs when it ends a client's bound path.

Missing tests added

  • the legacy loop end to end
  • the strict limit with transient errors
  • dedup_capacity = 0
  • the held-copy stall
  • dual mode with a failing bound send
  • a switch-on node with a legacy-only receiver

Defaults picked (each is a const or config value and easy to change)

Setting Value
strict_bodies_per_block 4
Refused-body memory the dedup window (300 s)
MAX_BODIES_PER_KEY_PER_BLOCK 2
ANCHOR_HEADER_WINDOW 30 min
ACTIVE_SET 5 s
NODE_ASSEMBLY_TTL 5 s
BOUND_SEND_TIMEOUT 10 s
SHARED_EGRESS_BURST 64
A released body is refused for anchor_ttl (90 s)

Behaviour changes to review

  • With the competing gate on, a body Zebra refused stays refused for about 5 min. The PR used to retry it.
  • In dual mode, a failed bound copy is not retried once the legacy copy has gone.
  • Nine ingress tests and a few sidecar tests that pinned the old behaviour now pin the new one. Each is named in its commit message.

Merged with main after #148

Still open

  • Legacy blocks that no bound task is working on still call Zebra on the receive loop, as on main. known_gap_a_held_legacy_zebra_call_still_stops_this_task pins this.
  • Copies dropped from a full retry queue are logged but not counted in a metric.
  • The full matrix.py matrix was not run here.

Refs #134, #130.

Checklist

  • cargo fmt --all passes
  • cargo clippy --workspace --all-targets -- -D warnings passes
  • cargo test --workspace passes (115 suites, 2374 passed, 0 failed, 10 ignored, after merging main at 127a30a)
  • cargo vet --locked and cargo deny pass (no new third-party crate versions)
  • New functionality includes tests
  • Commit messages are descriptive

🤖 Generated with Claude Code

erenyegit and others added 30 commits November 1, 2026 15:47
Draft for design and code review, not for merge or deployment. Part of #130.

Adds a candidate raw path behind switches that are off by default. Chunks
are bound to an object, frames to one body, and the delivery record ends
with what the consumer did with the block. The format, the limits and every
number are research values.

The gate fix is not behind a switch and also changes the legacy path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Diagnostic for the Linux failure on #134. A near-maximum body timed out with
nothing to say where it stopped.

The test now reports, for sender to node and node to client apart, what each
side saw, the receive buffer each socket was given, what the kernel queued and
dropped for it, capacity refusals, evictions, what was still open, and a
timeline each second. New counters are relaxed atomics, so counting takes no
lock on the measured path. A read is field by field and is no single instant.

A temporary CI job keeps that report as an artifact whether the test passes or
fails, and keeps cargo's exit code. The existing test job is unchanged.

No change to pacing, timeouts, FEC, buffer sizes or ignored tests. This names
no cause and fixes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 41311208629 on #134 showed the largest body losing datagrams at the
node's UDP receive buffer in all three runs. Nothing reads the socket while
the receive loop verifies, decodes, encodes again and sends on.

With the bound path on, a node now empties its socket into a queue between
datagrams, in order and one at a time. The queue is limited to 8192 datagrams
and 16 MiB of payload, research values. It comes ahead of every check and is
no part of the pool, so its memory adds to the pool's and one source can fill
it. It takes up a burst and adds no capacity.

Bound sends now pause after each object, 13 datagrams and 1 ms, research
values. This is a new default on this path, so it changes how the test sends
too. It costs time, and in dual mode the legacy copy waits behind it. Legacy
pacing is unchanged, and with the bound path off the loop is as it was.

The temporary CI job compares what each part does alone, keeps every log, and
tells a lost block from a build or harness failure. Timeouts, FEC and receive
buffer sizes are unchanged.

Not compiled or run before this commit. Whether this carries the body on
Linux is not known yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g state

Review 5728419600 on #134 found two faults on the bound path.

A spare could push itself off a full shelf while making room for its own
decode. Nine chunks of an object, thirty-nine others and its tenth chunk
panicked a node holding the forwarder lock, which ended its receive loop,
legacy path included. The client and sweep paths had the same fault. A spare
now keeps its place until it has a normal one, and a look over the shelf
passes over one that went.

A receive-only key reached shared state. Its segment 0 released what a Full
session left waiting, recorded those objects as sent, and they were dropped.
It also held a place in the pool. The role check now comes first, with the MAC
still checked and the keepalive kept. `receive_only_chunks_dropped` now counts
each authenticated datagram.

Panic containment is a second layer, and it covers what was tested. Panics
forced at set points in a node's steps drop the session that step works on. A
panic between those steps, or a poisoned lock, resets the path. That costs
every session what it had open, and nothing limits how often it can happen. No
datagram is known to reach it, which rests on reading the code. On a client, a
panic in a step restarts the bound path. `Verifier::verify`, `seal`, sending
and the rest of the receive loop are not covered, and no real panic was tested
inside a relay-node binary.

A build with panic = "abort" now refuses the path at start with an error, on
node, client, sidecar and ingress. Legacy still runs there.

Tests wait for exact datagrams by source and bytes, through a bounded test-only
`handled-log` feature that is off in release builds. A `research-crates` CI job
runs tests and Clippy for both research crates.

Local results on macOS with Rust 1.98.1: relay, sidecar and ingress 1040
passed, chunk-proof 153, delivery-lease 109, Clippy clean. Rust 1.91.0 cargo
check is clean. Workspace tests, Linux and the new CI job have not run.

Known limit, tracked separately: invalid limits still turn a client's bound
path off with only a warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…for room

Review 5728419600 on #134 found three faults in what the bound path allows.

One key on many source ports could pass any limit. Sessions, objects, held
bytes, work and the shelf now also count in a budget of the verified key_id,
between the session and the pool, so more ports buy no more room. Past its
limit a session gets no place, and a new source of that key gets no session in
the node's table, whatever its role. All a session held comes back to its key
and the pool when it goes. The values in `key_budget` are for research and were
not chosen for production: about a quarter of the pool. The fleet's key layout
is not assumed, and with one key the limit binds everybody as one.

Declared shapes could reserve far more than they cost to send. Bound segments
now take only the shape senders use, a raw segment of 13 shards with 10 data
shards, on the node, the client and the key trial. Any other shape reserves
nothing and starts no session, and a frame has one encoding, so one id. A shape
whose shortest content is past `max_content` is refused before any reservation.
At the research values the datagram size already stops that, so this check acts
alone only at a lower limit.

Making room could delete what was here. When the shelf cannot take an object
now, nothing moves and the new request is refused. What moved to the shelf
unfinished and decoded content are the only copies, so they stay, and only
leftovers of objects someone owns can go. Decoded objects waiting for segment 0
no longer give way. An object here in full needs bytes and no open place to
decode, so the nine, thirty-nine and one case of the last commit still delivers
without deleting anything.

A session holding nothing can no longer ask another to give way. Share counts
all sessions of a key and their shelves. A refusal from the pool moves an object
of a heavier key, then of a heavier session of the same key. A refusal from the
key moves only its own sessions. `Refusal::Outer` says which budget refused.

The two old tests that fixed the replaced behaviour are gone, and tests of the
new rules stand in their place. The panic tests now ask from a session holding
a little, and check that their panic points are reached.

Local results on macOS with Rust 1.98.1: relay, sidecar and ingress 1067
passed, chunk-proof 172, Clippy clean. Rust 1.91.0 cargo check is clean.
Workspace tests, Linux, CI, Rust 1.99.1 Clippy, release tests and the binary
matrix have not run. delivery-lease did not change and was not run.

38 hand-made mutants ran on a copy and none survived. In 36, every named test
failed. In one, the named test's assertion failed and its process then aborted.
In one, a named test still passes, as two checks cover it, and removing both
fails it. Two tests were strengthened after a first round showed them weak.

Known limits. A source refused at its key's limit has no session, so each of
its datagrams spends the per-IP allowance for unknown sources, and new sessions
from that IP, under any key, can be refused until it refills. Established
sessions do not reach that limiter, and the limiter did not change, but this
refusal makes it far easier to reach. This is no complete isolation between
keys. A session holding nothing waits while the pool is full, and a session
that fills its table and shelf has new objects refused until the old ones
finish or expire.
Review 5728419600 on #134 found that a body could lose its anchor to a flood
of headers, and that nothing limited anchors by key or by block.

Anchors were a first in, first out list of 256, so segments 0 under any valid
header could push out the anchor of a body under way. They are now grouped by
block hash, and only the exact block hash and body id opens the way, so a rival
body inherits nothing. Nothing pushes an anchor out. With no room a new one is
refused ahead of the PoW check and leaves no record. A block takes 4 bodies, the
number the gate takes, and a test ties the two together. Objects that wait for
segment 0 give their room back after 30 s.

An anchor stands only while at least one key pays for it. The key that opens it
pays, and a key that uses it and has room pays too, each for 90 s from its own
last use. Another key's use neither extends a payment nor ends it, and neither
does a session closing or a key being revoked, so leaving and coming back frees
nothing. In a test with a table of 8 and a quota of 2, the earlier rule let two
keys fill the table and refuse a third. Now they hold 4 and the third gets
through. That test uses injected time and a stand-in validator.

A key at its quota can use an existing anchor. It extends nothing and is counted
in `anchor_uses_without_room`. When the last payment ends the anchor goes, even
with such a key still using it, and that body's later objects wait for segment
0. Recovery needs quota to free up, the waiting records to still be valid and a
copy of segment 0 to arrive. The other choices were an anchor nobody pays for, a
key past its quota, or pushing out that key's own body. This wait is an open
resource and availability limit of the draft prototype and not a production
policy.

Two tests of the bound path could wait on the runtime they had blocked: with the
gate shut, a new socket's first send needed a driver nobody was polling. They
now set up before the gate shuts and run on one worker, so a wrong await fails
every time. A gate that gives up now tells its test. Production code did not
change for this.

The values are for research: 256 anchors, 64 per key, 4 bodies per block, 90 s
for a payment and 30 s for waiting. The fleet's key layout is not assumed.

Local results on macOS: relay, sidecar and ingress 1114 passed with Rust 1.98.1,
Clippy with `--all-targets -D warnings` on each exited 0, and Rust 1.91.0 cargo
check exited 0. 32 hand-made mutants ran on a copy and none survived. In 31
every named test failed. In one a named test still passed, and two other tests
caught that mutant. Linux, CI, workspace tests, Rust 1.99.1 Clippy, release
tests and the binary matrix have not run.

Known limits. `max_anchors` is a separate limit, and at the research values four
keys inside their quota can fill the table. A key inside its limits can take all
4 bodies of a block, and the right body is refused for as long as those anchors
are paid for. A valid old header still opens an anchor, as the node has no chain
source. L1 and L2 from the review are not fixed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review 5728419600 on #134 found three faults in how the bound path leaves a
client.

The ingress counted a bound copy as sent once it was queued. The client encoded
and sent later and only logged a failure, so a local failure stopped any retry
and no bound copy went out. `send_bound_raw_block` now waits for the local
result and hands it back. `Ok` says the block was encoded and that one relay's
socket took every datagram of it. It says nothing of whether anything arrived.
A body that did not fit, no relay taking it all, a full queue and a client that
went each come back as an error, and the forwarding claim stays open for the
body's next delivery.

The send ran on the client's loop, so a full encode and send held up legacy
traffic, receiving and keepalives. It now has a task of its own on the same
socket, one body at a time, with encoding on the blocking pool and four sends
waiting behind the one going out. A full queue answers at once and is no reason
for anything else to wait. Dropping the caller does not stop a send the client
has taken. `RelayPayload::BoundRawBlock` is gone, and the bound send has its own
queue.

In dual the ingress sends the two copies side by side, so neither waits for the
other and a bound failure does not stop legacy. A body counts under its header
from the moment any copy has left, and its own repeats cost it nothing. After a
failed bound copy, a cancel or `relay_timeout` the body stays open, and its next
delivery sends both copies again, which repeats the legacy copy.

Bodies under the emergency bypass now keep the same three things apart with
Sending, Owed and Sent. A legacy copy counts and is reported once, the body
holds its place, and it can still come again for the copy it is owed. A copy
that arrives during a send of the same body waits for how that ends instead of
being recorded as a rejection, and a different body past the limit is still
refused. On the legacy wire with the bypass on, that waiting is a change. Also
on the legacy wire, a dedup capacity of 0 no longer refuses the fifth copy of a
header, which was a fault of the earlier per-body limit.

`SOVRIGHT_RELAY_BOUND_RAW` used to turn the path on if it existed at all, so 0
and false enabled it. It is now an explicit boolean with the words the ingress
takes, read before the config on `relay-node` and the sidecar, and a value that
cannot be read stops the start. An empty value means off with a warning, so that
`VAR=` is a way back. That is this prototype's choice and no production
decision.

The bound path diagnostics no longer assume the send returns when queued.

`relay_timeout` did not change. It now also covers the local send, measured at
about 870 ms for a 2 MB body in a debug build on macOS, so a large body can pass
the 1 s default. The caller then gives up, Zebra's answer stands, the legacy
copy has gone and the send goes on.

Local results on this tree, macOS, Rust 1.98.1: relay, sidecar and ingress 1170
passed, 4 ignored. Clippy with `--all-targets -D warnings` on each of the three
exited 0, and Rust 1.91.0 cargo check exited 0. The switch was run on real
binaries with real bodies against a stand-in Zebra. Timeout tests use a paused
clock and a held send. 49 hand-made mutants ran on a copy and all failed; in one
a named test still passed, as two checks cover it, and two others caught it.
Linux, CI, workspace tests, Rust 1.99.1 Clippy, release tests and the binary
matrix have not run.

Known limits. A UDP send is not delivery. The claim waits only for the bound
copy, and a legacy copy counts once queued. A peer loop waits for the bound
result with no time limit, and so do copies of the same body: a full queue at
the measured speed comes to about 4.4 s in a debug build, as an example and no
upper bound. A pool block shares the queue with peer bodies, is refused when it
is full, and nothing on the pool side retries. A panic in the bound task is
untested. Legacy still sends on the client loop. Review item M2 is not closed:
concurrent traffic was shown on a clean loopback only, with no netem run, no
sustained load and no bandwidth or hop latency measurement, and dual still
doubles raw bandwidth. L1 and L2 are not fixed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review 5728419600 on #134, item 8, found that a slow Zebra call stopped the
sidecar's receive loop. A bound block was awaited inline through its parent
lookup and its submit, so a different block, legacy traffic and cleanup all
waited for the answer. This fixes that for the bound path and for no other.

A bound block's parent lookup and submit now run on a task, at most four at
once. With all four busy the loop takes no more bound blocks: they stay in their
channel, and behind that in the client, within the limits they had. The loop
waits for no place and goes on with legacy traffic and cleanup. It records a
delivered block in the dedup ring when the task ends, and asks the ring before
it starts one.

A copy that arrives during a task needs somewhere to wait, because the gate
turns away the bytes it has at Zebra. A copy with the bytes of a running task
has a place of its own, one for each task, and nothing can crowd it out. If the
task fails that copy is the retry, and if Zebra took the block it is a repeat. A
second copy of those bytes is dropped, as they are waiting already. Any other
copy of a block under way shares a room of four. With no room it goes on as it
came: the gate lets a different body run alongside, at the price of a Zebra call
on the receive loop. A compact block that needs the mempool cannot say its
bytes, so it counts as other bytes. At most eight copies wait, and each is one
block. These counts are research values. The gate did not change.

A task that ends without Zebra taking the block gives up what it held. Its gate
slot, the block's ownership and its reservation go with it, on an error, at its
budget or when the handler is cancelled, and nothing says the block is dealt
with, so the same body can come again. After the budget that is shown on the
live loop, where the body reaches Zebra a second time. A cancelled handler
leaves no receive task to take it, so that test reads the gate and the ledger.

The budget is 120 s, also a research value, and it is a timer. It acts where a
task waits: synchronous work runs to its end first, and work a task hands on
goes on under its own limits. It is no upper bound on CPU or run time. When the
client goes, the loop ends and waits for running tasks up to their budgets.

`tokio`'s `test-util` is a dev-dependency here, for a paused clock.

Local results on this tree, macOS arm64, Rust 1.98.1: the sidecar package 335
passed, 1 ignored. Clippy with `--all-targets -D warnings` on it exited 0, and
Rust 1.91.0 cargo check on it exited 0. Twelve regressions failed on 2555daf
with the tasks absent. Five more failed before the reserved place, four of them
with the copy never reaching Zebra. 12 hand-made mutants of the reserved place
and of cancellation ran on a copy of this tree and all failed, with both
baselines green. In one, a test named at first still passed, as the parent
lookup has a 3 s limit of its own, and eight other tests caught it. The test
binary ran 60 times with no failure, which proves nothing alone. Zebra is a
stand-in throughout.

The relay and ingress code did not change. Their tests and Clippy, the 1193
passed with the sidecar and 16 earlier mutants all ran on the tree before the
reserved place, and not on this one. Linux, CI, workspace tests, Rust 1.99.1
Clippy, release tests and the binary matrix have not run.

Known limits. The slow Zebra item is not closed for every path. Legacy paths
still run on the receive loop, as does a held raw copy once released, so a slow
legacy call still stops new bound blocks, cleanup and other legacy traffic, and
`known_gap_a_held_legacy_zebra_call_still_stops_this_task` shows that. Work
handed to the emergency bypass leaves the task with its gate slot, so a copy
released then can still be turned away. That is read from the code, untested,
and older than this change. Nothing asks for a copy, so with none a failed block
has no retry. The gate's budgets can refuse a held copy, and held copies drop at
shutdown; neither was tested. A stalled Zebra can fill all four tasks for up to
their budgets, with new bound blocks waiting, and that delay was not measured. A
child can reach Zebra before its parent, untested. A panic in a task was tried
only with a stand-in future. What hashing copies costs the receive loop was not
measured. Review item M2 is not closed, and L1 and L2 are not fixed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`BoundTaskLimits` was hard-coded in main.rs. The running, budget and held
limits are now `bound_tasks_running`, `bound_tasks_budget_secs` and
`bound_tasks_held` in the sidecar TOML, with matching `--bound-tasks-*`
CLI flags, the same way the other settings are read. The defaults are
unchanged (4 tasks, 120 s, 4 held).

The config is checked at load: running must be at least 1 and the budget
above 0 (held is unsigned, so it can't go below 0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…loop

When a bound task ended without delivering, each copy held for its block
was retried inline: `finish_raw_segments` (and the compact path, through
the `again` queue) awaited Zebra on the receive loop, one copy after
another. With Zebra hanging that stopped the loop for every held copy.
A copy that found no room to wait also ran inline beside the task.

Copies whose wait is over, and copies of a block under way with no room
to wait, now go to a bounded `Retries` queue. The loop starts each one as
a bound task once there is room under the running limit (bound blocks
and retries share it) and never awaits Zebra for a copy. The queue holds
at most `running + held` copies and keeps one copy per known body; past
that a copy is dropped with a warning. Before a copy starts, the loop
checks the dedup ring as the legacy paths do and re-holds it if another
task has taken its block up.

The legacy paths for blocks no task is on are unchanged: a compact block
or reassembled raw block with no bound work under way still runs here as
before.

The gated compact path, the compact fast path and the raw segment path
are split so the inline and task versions share the same code.

Adds the held-copy stall test: with Zebra holding every call, a retried
legacy or compact copy waits at Zebra while the receive loop buffers
other payloads and another bound block reaches Zebra. It fails on the
old code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… sent

Review items 7 and 8 on #134.

Self fan-out: objects let go by a segment 0 went back to the session that
sent them, because the node only left out the source of the current
datagram. Each object now carries the session it came from, and the node
sends it to every session but that one.

Record before send: the forwarded record was written inside take, before
the send. With no peer, or a socket that refused, the next copy was
dropped for resend_after and counted against max_resends. The node now
records a send only once every datagram of the object went out to at
least one peer. What take lets go is marked for RELEASE_LEFTOVER (500 ms)
so the unneeded shards of the same burst do not open a new object; that
mark is no send and counts as none.

objects_sent_on now counts only objects that went out; the new
objects_released counts what the node let go. Tests that check anchors
with no receiver attached now read objects_released, and the unit test
for a panic while recording now expects the object to have gone (the
record comes after the send).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d copy in flight

On RawWire::Legacy a copy of an unverified body that is still being sent is a
repeat again, as it was on main. Waiting on the send and retrying when it fails
is kept for the body-keyed wires (Dual, Bound) only.

Review item 2 on #134.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… out a block

Review items 1, 2, 9 and 10 on #134, and the inconclusive nit.

- New submit_gate.competing_bodies switch, off by default. Off, the gate is the
  legacy one exactly: it marks a block when a candidate passes and holds its
  lock across the Zebra call. The bound raw switch turns it on.
- Competing-body gate: Zebra's refusal of a body is remembered per header for
  the dedup window (new RejectReason::RefusedBody), so a replayed body never
  reaches Zebra again. Only distinct bodies that got an answer, or are at Zebra
  now, count against strict_bodies_per_block. Repeats and calls that failed in
  transit, timed out or were cancelled cost nothing. strict_attempts_per_block
  is gone.
- Dry runs read and write only their own marks, keyed by block and body, and
  spend no live budget. A dry run no longer counts as delivered on the bound
  path.
- dedup_capacity = 0 is refused at config load; the gate also keeps at least
  one entry.
- Zebra's 'inconclusive' gets its own rejection class instead of 'invalid'.
- Tests: legacy loop end to end on a real sidecar with every switch off,
  transient errors never exhausting the limit, a replayed refused body, dry-run
  isolation, capacity 0, and the legacy gate's unchanged behaviour. Tests that
  pinned retrying a refused body now pin that it is turned away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`sovright-relay` depends on both research crates by path, but each had
its own `[workspace]` table and lockfile and sat in the root `exclude`
list. Their exact pins (`sha2 =0.10.9`, `reed-solomon-erasure =6.0.0`,
`hmac =0.12.1` and the dev-dependency pins) constrained resolution, and
CI tested them only against their own locks.

They are now workspace members: the `[workspace]` tables and their
Cargo.lock files are gone, shared dependencies use the workspace
versions (`sha2`, `serde_json`, `tokio`, `blake2b_simd`), and the rest
use caret versions matching what the root lockfile already resolves
(`hmac 0.12`, `reed-solomon-erasure 6.0`, `tokio-test 0.4`). The root
Cargo.lock only gains their dev-dependency edges. The "Not a workspace
member" descriptions are fixed.

The separate `research-crates` CI job is removed: the workspace test and
clippy jobs now build, test and lint them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The delivery-lease test vectors were named in Turkish. The file is now
`frames.json` and its `cerceveler` / `cerceve` / `cerceve_0` keys are
`frames` / `frame` / `frame_0`, with the one test that reads them
updated. Every value is unchanged (checked by comparing the parsed JSON
with the keys mapped).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`free_port` bound port 0, read the port and closed the socket, and the
child bound that port later, so another process could take it in
between (and the ingress RPC probe could then connect to the wrong
listener).

The relay and the ingress RPC now bind 127.0.0.1:0 themselves, and the
script reads the port each bound from the `local_addr=` field of the
line it already logs once listening ("Sovright relay listening",
"Pool submitblock relay gateway started"; both predate #134). A child
that exits before that line fails at once, with its log path. The
first relay's port is kept for the clients and for later relays in a
rollout or swap, which bind it explicitly. `free_port` is gone.

Checked against the relay-node and sovright-p2p-ingress binaries built
here: both report their port, and a restarted relay comes back on the
same one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ci.yml had no `permissions:` block, so its jobs ran with the repository's
default token scope. It now sets `contents: read` at the top level; no
job needs more (upload-artifact needs nothing extra). cargo-fmt.yml and
fuzz-regression.yml get the same read-only block, as they only check out
and build. images.yml and release-linux.yml already declare theirs.

Every third-party action in .github/workflows is pinned to the full
commit SHA its tag points to today, with the version in a trailing
comment. dtolnay/rust-toolchain is pinned to the head of its `stable` /
`nightly` branch, and since the ref no longer names the toolchain, each
use passes `toolchain: stable` / `toolchain: nightly` explicitly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A node sent bound datagrams to every session, and a client's bound send
said Ok after handing them to relays that might drop every one.

A client with bound work now sends a hello (an authenticated keepalive
with a marker hash and a flags word) beside its keepalives. A node with
the path on answers it with an ack and marks the session as taking bound
objects if the hello says it receives; a node without the path, or from
before it, rejects it as it rejects any odd keepalive. The node forwards
bound objects only to sessions that said they take them, so a legacy
receiver gets the block the legacy way and no bound datagrams. A bound
send goes only to relays whose ack still stands (BOUND_ACK_TTL, 180 s),
waits up to BOUND_HELLO_WAIT (2 s) for one, and fails with
ConnectionRefused when none answered.

Tests changed to pin the new behaviour: a raw sender that never sent a
hello no longer gets forwards (a_session_sends_an_object_on_once_...),
a send through a node with the switch off now fails instead of Ok
(zero_leaves_the_path_shut_on_a_real_node), and the socket-error tests
mark their refusing relay as having answered so they still reach the
socket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Config::validate now runs the submit gate checks and the bound task limit checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The competing-body gate's Zebra call is a sixth site and goes through the
shared classifier like the others; the test now checks every site does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any old header with good PoW opened an anchor, one key could take
every body a block has, and nothing freed a place when a body was
refused downstream. So a key could hold the bound path shut.

- Recency: a segment 0 whose header time is more than
  ANCHOR_HEADER_WINDOW (30 min) behind the newest header that passed
  here is refused before its PoW check. The newest is capped at this
  node's clock, so a future-dated header cannot make every other look
  old. A header that passes ages out anchors that far behind it, and a
  segment 0 refused for room is PoW-checked when it would age some out,
  so a table old headers filled at a cold start clears once a newer one
  comes.
- Per-key share: a key pays for at most MAX_BODIES_PER_KEY_PER_BLOCK
  (2) of one block's max_bodies_per_block (4) bodies.
- Release: BoundBlock now names its body (block_hash, body_id) and
  BoundBlock::refuse gives it back and sends an authenticated
  BOUND_RELEASE keepalive to the client's relays. A node with the path
  on takes the anchor away when the release comes from a full-role
  key and keeps that body out for anchor_ttl. RelayNode has
  release_bound_body for a caller beside the node.
- New counters in BoundNodeUsage: anchors_refused_for_key_bodies,
  anchors_refused_for_age, anchors_refused_as_released,
  anchors_released.

Tests that pinned the old behaviour now pin the new one:
known_gap_one_key_inside_its_limits_can_use_up_a_blocks_bodies became
one_key_cannot_use_up_a_blocks_bodies, and
known_gap_an_old_header_still_opens_an_anchor became
an_old_header_opens_no_anchor_once_a_newer_one_has.
a_block_takes_only_so_many_bodies_at_a_real_node now uses three keys.
Test bodies of different fills get one header time, since a fill byte
in the time field made them decades apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The body buffer made room for a new set by dropping the oldest set,
and a block's least advanced one, whether or not it was still
arriving. Eight forged segment 0s could push out an honest 2 MB set
mid-way, and decoded segments keep no spares, so the body was gone
until a resend.

A set now counts as arriving for ACTIVE_SET (5 s) after its last
segment. BodyBuffer::room_for picks the victims a segment needs, from
quiet sets only (least advanced, then newest), or says there are none.
Intake asks it before the ledger and answers Refused(Refusal::Sets),
which the bridge treats as a lack of room and shelves for a retry, so
the newcomer costs nothing and comes back. The ledger's capacity path
offers only quiet sets for a segment too; the copy of a whole body at
assembly may still take any set, since refusing it would lose a body
that is all there.

Tests that pinned the old order now let earlier sets go quiet first.
the_older_of_two_equal_sets_goes_first became
the_newer_of_two_equal_quiet_sets_goes_first.
known_gap_a_consistent_body_of_the_header_can_evict_the_honest_one now
pins that it cannot while the honest set arrives, only once it is
quiet. New: a_set_still_arriving_is_never_given_up_for_a_newcomer and
forged_newcomers_cannot_push_out_an_honest_set_under_way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The legacy pacer and the bound sender ran independently on the same
socket, so with both under way a client could put out far more than
either pace allows. While a bound send is under way, every datagram of
the client now takes its turn from one token bucket: the bound path's
own pace to every relay, with SHARED_EGRESS_BURST datagrams allowed at
once. The bound path alone stays under that pace, and legacy traffic
alone is held only by its own pacing, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In dual, a bound copy that failed left the claim uncommitted, so a bound
path that kept failing had every later P2P delivery send the whole
legacy copy again. Once the legacy copy has left, a failed bound copy is
now logged and the body counts as sent; that bound copy is not tried
again. The bound wire is unchanged, and so is a caller that gives up
with the bound copy still under way.

Tests that pinned the old dual behaviour now pin the new one:
- a_bound_failure_lets_the_legacy_copy_go_in_dual: the body is sent, and
  repeated deliveries send no legacy copy (the missing "dual mode with a
  failing bound send" test).
- the every-end helper takes whether a failure sends; in dual, Failed,
  NobodyAnswered and ClientWent leave the waiting copy a repeat.
- after_a_failed_bound_copy_{a_strict,an_unverified}_body_* and
  a_copy_during_an_unverified_send_* expect the body sent in dual.
- a_full_bound_queue_* is split: the bound wire still comes back to the
  caller; in dual the body is sent by its legacy copy.
- an_owed_body_can_come_again_with_its_header_at_its_limit makes the body
  owed by a caller that gives up, which still leaves it open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An object's first datagram reserves the room of its whole object, about
eight times its own size, and on a node it kept that for 30 s. The
reservation stays, since it is what lets an object that was let in
finish, but a node now keeps an open object for NODE_ASSEMBLY_TTL (5 s):
a sender sends an object in one burst and a second copy comes after
resend_after (2 s), so holding the pool with first datagrams now takes
one every 5 s per object. The receiver keeps its own 30 s.

Decoded objects waiting for their segment 0 never gave way. They now do
when the pool or their key is short of bytes, oldest first, to a session
that holds less, after any open object of theirs that can move. With the
pool out of open places they would make no room, so they stay. Counted
as waiting_gave_way.

what_waits_for_its_segment_0_is_not_given_up pinned the old behaviour;
it is now what_waits_for_its_segment_0_gives_way_to_a_lighter_session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A spare that had all it needed but was refused its decode while the pool
was full sat on the node until its time ran out, room or not: only the
receiver ever called revive. The forwarder now marks a session whose
chunk was refused while it holds such a spare (Assembler gains
has_ready_spares), and BoundForwarder::revive tries those again, admits
what decodes and sends it on like any other. The node calls it after a
bound send has gone out, which gives room back, and on its cleanup
timer. Counted as spares_revived.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bound_usage() was only for tests and logs. render_bound_prometheus_text
renders BoundNodeUsage under sovright_relay_bound_*: what the pool holds
now as gauges, the rest as counters, and arrivals as one counter labelled
by outcome. relay-node appends it to its metrics textfile when the bound
path is on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It was public in every build. It is now behind cfg(test) or the
test-support feature, which the one downstream user's dev-dependency
already turns on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A caller of send_bound_raw_block waited for as long as the client took.
It now hears Timeout after BOUND_SEND_TIMEOUT (10 s), which covers the
sends queued ahead, a hello's wait, the encoding and the paced datagrams
of the largest body. The send goes on, as it does when a caller stops
waiting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
contain dropped the bound receiver with no word when its consumer had
gone. It now warns, so an operator can tell why bound blocks stopped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relay side of item 3 lets one key hold only two bodies of a block
and frees a place when a receiver says a body was refused, but nothing
in the sidecar said so: a bound block Zebra refused was only given back.
One key sending two wrong bodies first then shut the right one out at
the node. A bound block that Zebra calls invalid, or that the gate
turns away as a body Zebra refused, now goes back with BoundBlock::refuse.

Sidecar tests brought in line with the merged relay fixes:
- a bound send through a legacy relay is now refused at the sender (item 6),
  and a dual send still delivers its legacy copy
- a refused body's later copy stops at the relay or the gate
- side-by-side bodies of one block come from keys of their own
- the evicted-body test dates both headers alike and waits for the set
  to go quiet (items 3 and 4)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main (#137 to #142) hardened every workflow on its own. The workflow
conflicts take main's version, which is stricter than this branch's:
per-job permissions, persist-credentials: false, newer pins, and no
build cache on release builds. The bound-path diagnostic job from #134
is kept in ci.yml with the same hardening, and its matrix test args now
go through an env var instead of being expanded into the script.

Cargo.lock is main's, plus the two research crates that are now
workspace members. No third-party crate version changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main's cargo-deny job (#139) refuses a crate with no license field, and
chunk-proof and delivery-lease became workspace members in this PR.
They now take the workspace's MIT OR Apache-2.0, like every other crate
here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zmanian
zmanian marked this pull request as ready for review November 13, 2026 01:10
…-body branch

Four conflicts, each kept from both sides:
- ci.yml: the bound-path diagnostic job and #148's zakura-native job are both kept.
- sovright-p2p-ingress dev-dependencies: both sets kept.
- nu7_transport_tests.rs: uses #148's LoopbackRelay helper; its RelayBridge
  now sets this branch's `raw_wire: RawWire::Legacy`.
- relay-sidecar main.rs: #145's per-block reconstruction log and metric go
  into this branch's compact_fast_path_outcome, so bound tasks log it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants