Repository navigation
Conversation
Draft for design and code review, not for merge or deployment. Part of #130. Adds a candidate raw path behind switches that are off by default. Chunks are bound to an object, frames to one body, and the delivery record ends with what the consumer did with the block. The format, the limits and every number are research values. The gate fix is not behind a switch and also changes the legacy path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Diagnostic for the Linux failure on #134. A near-maximum body timed out with nothing to say where it stopped. The test now reports, for sender to node and node to client apart, what each side saw, the receive buffer each socket was given, what the kernel queued and dropped for it, capacity refusals, evictions, what was still open, and a timeline each second. New counters are relaxed atomics, so counting takes no lock on the measured path. A read is field by field and is no single instant. A temporary CI job keeps that report as an artifact whether the test passes or fails, and keeps cargo's exit code. The existing test job is unchanged. No change to pacing, timeouts, FEC, buffer sizes or ignored tests. This names no cause and fixes nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 41311208629 on #134 showed the largest body losing datagrams at the node's UDP receive buffer in all three runs. Nothing reads the socket while the receive loop verifies, decodes, encodes again and sends on. With the bound path on, a node now empties its socket into a queue between datagrams, in order and one at a time. The queue is limited to 8192 datagrams and 16 MiB of payload, research values. It comes ahead of every check and is no part of the pool, so its memory adds to the pool's and one source can fill it. It takes up a burst and adds no capacity. Bound sends now pause after each object, 13 datagrams and 1 ms, research values. This is a new default on this path, so it changes how the test sends too. It costs time, and in dual mode the legacy copy waits behind it. Legacy pacing is unchanged, and with the bound path off the loop is as it was. The temporary CI job compares what each part does alone, keeps every log, and tells a lost block from a build or harness failure. Timeouts, FEC and receive buffer sizes are unchanged. Not compiled or run before this commit. Whether this carries the body on Linux is not known yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g state Review 5728419600 on #134 found two faults on the bound path. A spare could push itself off a full shelf while making room for its own decode. Nine chunks of an object, thirty-nine others and its tenth chunk panicked a node holding the forwarder lock, which ended its receive loop, legacy path included. The client and sweep paths had the same fault. A spare now keeps its place until it has a normal one, and a look over the shelf passes over one that went. A receive-only key reached shared state. Its segment 0 released what a Full session left waiting, recorded those objects as sent, and they were dropped. It also held a place in the pool. The role check now comes first, with the MAC still checked and the keepalive kept. `receive_only_chunks_dropped` now counts each authenticated datagram. Panic containment is a second layer, and it covers what was tested. Panics forced at set points in a node's steps drop the session that step works on. A panic between those steps, or a poisoned lock, resets the path. That costs every session what it had open, and nothing limits how often it can happen. No datagram is known to reach it, which rests on reading the code. On a client, a panic in a step restarts the bound path. `Verifier::verify`, `seal`, sending and the rest of the receive loop are not covered, and no real panic was tested inside a relay-node binary. A build with panic = "abort" now refuses the path at start with an error, on node, client, sidecar and ingress. Legacy still runs there. Tests wait for exact datagrams by source and bytes, through a bounded test-only `handled-log` feature that is off in release builds. A `research-crates` CI job runs tests and Clippy for both research crates. Local results on macOS with Rust 1.98.1: relay, sidecar and ingress 1040 passed, chunk-proof 153, delivery-lease 109, Clippy clean. Rust 1.91.0 cargo check is clean. Workspace tests, Linux and the new CI job have not run. Known limit, tracked separately: invalid limits still turn a client's bound path off with only a warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…for room Review 5728419600 on #134 found three faults in what the bound path allows. One key on many source ports could pass any limit. Sessions, objects, held bytes, work and the shelf now also count in a budget of the verified key_id, between the session and the pool, so more ports buy no more room. Past its limit a session gets no place, and a new source of that key gets no session in the node's table, whatever its role. All a session held comes back to its key and the pool when it goes. The values in `key_budget` are for research and were not chosen for production: about a quarter of the pool. The fleet's key layout is not assumed, and with one key the limit binds everybody as one. Declared shapes could reserve far more than they cost to send. Bound segments now take only the shape senders use, a raw segment of 13 shards with 10 data shards, on the node, the client and the key trial. Any other shape reserves nothing and starts no session, and a frame has one encoding, so one id. A shape whose shortest content is past `max_content` is refused before any reservation. At the research values the datagram size already stops that, so this check acts alone only at a lower limit. Making room could delete what was here. When the shelf cannot take an object now, nothing moves and the new request is refused. What moved to the shelf unfinished and decoded content are the only copies, so they stay, and only leftovers of objects someone owns can go. Decoded objects waiting for segment 0 no longer give way. An object here in full needs bytes and no open place to decode, so the nine, thirty-nine and one case of the last commit still delivers without deleting anything. A session holding nothing can no longer ask another to give way. Share counts all sessions of a key and their shelves. A refusal from the pool moves an object of a heavier key, then of a heavier session of the same key. A refusal from the key moves only its own sessions. `Refusal::Outer` says which budget refused. The two old tests that fixed the replaced behaviour are gone, and tests of the new rules stand in their place. The panic tests now ask from a session holding a little, and check that their panic points are reached. Local results on macOS with Rust 1.98.1: relay, sidecar and ingress 1067 passed, chunk-proof 172, Clippy clean. Rust 1.91.0 cargo check is clean. Workspace tests, Linux, CI, Rust 1.99.1 Clippy, release tests and the binary matrix have not run. delivery-lease did not change and was not run. 38 hand-made mutants ran on a copy and none survived. In 36, every named test failed. In one, the named test's assertion failed and its process then aborted. In one, a named test still passes, as two checks cover it, and removing both fails it. Two tests were strengthened after a first round showed them weak. Known limits. A source refused at its key's limit has no session, so each of its datagrams spends the per-IP allowance for unknown sources, and new sessions from that IP, under any key, can be refused until it refills. Established sessions do not reach that limiter, and the limiter did not change, but this refusal makes it far easier to reach. This is no complete isolation between keys. A session holding nothing waits while the pool is full, and a session that fills its table and shelf has new objects refused until the old ones finish or expire.
Review 5728419600 on #134 found that a body could lose its anchor to a flood of headers, and that nothing limited anchors by key or by block. Anchors were a first in, first out list of 256, so segments 0 under any valid header could push out the anchor of a body under way. They are now grouped by block hash, and only the exact block hash and body id opens the way, so a rival body inherits nothing. Nothing pushes an anchor out. With no room a new one is refused ahead of the PoW check and leaves no record. A block takes 4 bodies, the number the gate takes, and a test ties the two together. Objects that wait for segment 0 give their room back after 30 s. An anchor stands only while at least one key pays for it. The key that opens it pays, and a key that uses it and has room pays too, each for 90 s from its own last use. Another key's use neither extends a payment nor ends it, and neither does a session closing or a key being revoked, so leaving and coming back frees nothing. In a test with a table of 8 and a quota of 2, the earlier rule let two keys fill the table and refuse a third. Now they hold 4 and the third gets through. That test uses injected time and a stand-in validator. A key at its quota can use an existing anchor. It extends nothing and is counted in `anchor_uses_without_room`. When the last payment ends the anchor goes, even with such a key still using it, and that body's later objects wait for segment 0. Recovery needs quota to free up, the waiting records to still be valid and a copy of segment 0 to arrive. The other choices were an anchor nobody pays for, a key past its quota, or pushing out that key's own body. This wait is an open resource and availability limit of the draft prototype and not a production policy. Two tests of the bound path could wait on the runtime they had blocked: with the gate shut, a new socket's first send needed a driver nobody was polling. They now set up before the gate shuts and run on one worker, so a wrong await fails every time. A gate that gives up now tells its test. Production code did not change for this. The values are for research: 256 anchors, 64 per key, 4 bodies per block, 90 s for a payment and 30 s for waiting. The fleet's key layout is not assumed. Local results on macOS: relay, sidecar and ingress 1114 passed with Rust 1.98.1, Clippy with `--all-targets -D warnings` on each exited 0, and Rust 1.91.0 cargo check exited 0. 32 hand-made mutants ran on a copy and none survived. In 31 every named test failed. In one a named test still passed, and two other tests caught that mutant. Linux, CI, workspace tests, Rust 1.99.1 Clippy, release tests and the binary matrix have not run. Known limits. `max_anchors` is a separate limit, and at the research values four keys inside their quota can fill the table. A key inside its limits can take all 4 bodies of a block, and the right body is refused for as long as those anchors are paid for. A valid old header still opens an anchor, as the node has no chain source. L1 and L2 from the review are not fixed here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review 5728419600 on #134 found three faults in how the bound path leaves a client. The ingress counted a bound copy as sent once it was queued. The client encoded and sent later and only logged a failure, so a local failure stopped any retry and no bound copy went out. `send_bound_raw_block` now waits for the local result and hands it back. `Ok` says the block was encoded and that one relay's socket took every datagram of it. It says nothing of whether anything arrived. A body that did not fit, no relay taking it all, a full queue and a client that went each come back as an error, and the forwarding claim stays open for the body's next delivery. The send ran on the client's loop, so a full encode and send held up legacy traffic, receiving and keepalives. It now has a task of its own on the same socket, one body at a time, with encoding on the blocking pool and four sends waiting behind the one going out. A full queue answers at once and is no reason for anything else to wait. Dropping the caller does not stop a send the client has taken. `RelayPayload::BoundRawBlock` is gone, and the bound send has its own queue. In dual the ingress sends the two copies side by side, so neither waits for the other and a bound failure does not stop legacy. A body counts under its header from the moment any copy has left, and its own repeats cost it nothing. After a failed bound copy, a cancel or `relay_timeout` the body stays open, and its next delivery sends both copies again, which repeats the legacy copy. Bodies under the emergency bypass now keep the same three things apart with Sending, Owed and Sent. A legacy copy counts and is reported once, the body holds its place, and it can still come again for the copy it is owed. A copy that arrives during a send of the same body waits for how that ends instead of being recorded as a rejection, and a different body past the limit is still refused. On the legacy wire with the bypass on, that waiting is a change. Also on the legacy wire, a dedup capacity of 0 no longer refuses the fifth copy of a header, which was a fault of the earlier per-body limit. `SOVRIGHT_RELAY_BOUND_RAW` used to turn the path on if it existed at all, so 0 and false enabled it. It is now an explicit boolean with the words the ingress takes, read before the config on `relay-node` and the sidecar, and a value that cannot be read stops the start. An empty value means off with a warning, so that `VAR=` is a way back. That is this prototype's choice and no production decision. The bound path diagnostics no longer assume the send returns when queued. `relay_timeout` did not change. It now also covers the local send, measured at about 870 ms for a 2 MB body in a debug build on macOS, so a large body can pass the 1 s default. The caller then gives up, Zebra's answer stands, the legacy copy has gone and the send goes on. Local results on this tree, macOS, Rust 1.98.1: relay, sidecar and ingress 1170 passed, 4 ignored. Clippy with `--all-targets -D warnings` on each of the three exited 0, and Rust 1.91.0 cargo check exited 0. The switch was run on real binaries with real bodies against a stand-in Zebra. Timeout tests use a paused clock and a held send. 49 hand-made mutants ran on a copy and all failed; in one a named test still passed, as two checks cover it, and two others caught it. Linux, CI, workspace tests, Rust 1.99.1 Clippy, release tests and the binary matrix have not run. Known limits. A UDP send is not delivery. The claim waits only for the bound copy, and a legacy copy counts once queued. A peer loop waits for the bound result with no time limit, and so do copies of the same body: a full queue at the measured speed comes to about 4.4 s in a debug build, as an example and no upper bound. A pool block shares the queue with peer bodies, is refused when it is full, and nothing on the pool side retries. A panic in the bound task is untested. Legacy still sends on the client loop. Review item M2 is not closed: concurrent traffic was shown on a clean loopback only, with no netem run, no sustained load and no bandwidth or hop latency measurement, and dual still doubles raw bandwidth. L1 and L2 are not fixed here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review 5728419600 on #134, item 8, found that a slow Zebra call stopped the sidecar's receive loop. A bound block was awaited inline through its parent lookup and its submit, so a different block, legacy traffic and cleanup all waited for the answer. This fixes that for the bound path and for no other. A bound block's parent lookup and submit now run on a task, at most four at once. With all four busy the loop takes no more bound blocks: they stay in their channel, and behind that in the client, within the limits they had. The loop waits for no place and goes on with legacy traffic and cleanup. It records a delivered block in the dedup ring when the task ends, and asks the ring before it starts one. A copy that arrives during a task needs somewhere to wait, because the gate turns away the bytes it has at Zebra. A copy with the bytes of a running task has a place of its own, one for each task, and nothing can crowd it out. If the task fails that copy is the retry, and if Zebra took the block it is a repeat. A second copy of those bytes is dropped, as they are waiting already. Any other copy of a block under way shares a room of four. With no room it goes on as it came: the gate lets a different body run alongside, at the price of a Zebra call on the receive loop. A compact block that needs the mempool cannot say its bytes, so it counts as other bytes. At most eight copies wait, and each is one block. These counts are research values. The gate did not change. A task that ends without Zebra taking the block gives up what it held. Its gate slot, the block's ownership and its reservation go with it, on an error, at its budget or when the handler is cancelled, and nothing says the block is dealt with, so the same body can come again. After the budget that is shown on the live loop, where the body reaches Zebra a second time. A cancelled handler leaves no receive task to take it, so that test reads the gate and the ledger. The budget is 120 s, also a research value, and it is a timer. It acts where a task waits: synchronous work runs to its end first, and work a task hands on goes on under its own limits. It is no upper bound on CPU or run time. When the client goes, the loop ends and waits for running tasks up to their budgets. `tokio`'s `test-util` is a dev-dependency here, for a paused clock. Local results on this tree, macOS arm64, Rust 1.98.1: the sidecar package 335 passed, 1 ignored. Clippy with `--all-targets -D warnings` on it exited 0, and Rust 1.91.0 cargo check on it exited 0. Twelve regressions failed on 2555daf with the tasks absent. Five more failed before the reserved place, four of them with the copy never reaching Zebra. 12 hand-made mutants of the reserved place and of cancellation ran on a copy of this tree and all failed, with both baselines green. In one, a test named at first still passed, as the parent lookup has a 3 s limit of its own, and eight other tests caught it. The test binary ran 60 times with no failure, which proves nothing alone. Zebra is a stand-in throughout. The relay and ingress code did not change. Their tests and Clippy, the 1193 passed with the sidecar and 16 earlier mutants all ran on the tree before the reserved place, and not on this one. Linux, CI, workspace tests, Rust 1.99.1 Clippy, release tests and the binary matrix have not run. Known limits. The slow Zebra item is not closed for every path. Legacy paths still run on the receive loop, as does a held raw copy once released, so a slow legacy call still stops new bound blocks, cleanup and other legacy traffic, and `known_gap_a_held_legacy_zebra_call_still_stops_this_task` shows that. Work handed to the emergency bypass leaves the task with its gate slot, so a copy released then can still be turned away. That is read from the code, untested, and older than this change. Nothing asks for a copy, so with none a failed block has no retry. The gate's budgets can refuse a held copy, and held copies drop at shutdown; neither was tested. A stalled Zebra can fill all four tasks for up to their budgets, with new bound blocks waiting, and that delay was not measured. A child can reach Zebra before its parent, untested. A panic in a task was tried only with a stand-in future. What hashing copies costs the receive loop was not measured. Review item M2 is not closed, and L1 and L2 are not fixed here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`BoundTaskLimits` was hard-coded in main.rs. The running, budget and held limits are now `bound_tasks_running`, `bound_tasks_budget_secs` and `bound_tasks_held` in the sidecar TOML, with matching `--bound-tasks-*` CLI flags, the same way the other settings are read. The defaults are unchanged (4 tasks, 120 s, 4 held). The config is checked at load: running must be at least 1 and the budget above 0 (held is unsigned, so it can't go below 0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…loop When a bound task ended without delivering, each copy held for its block was retried inline: `finish_raw_segments` (and the compact path, through the `again` queue) awaited Zebra on the receive loop, one copy after another. With Zebra hanging that stopped the loop for every held copy. A copy that found no room to wait also ran inline beside the task. Copies whose wait is over, and copies of a block under way with no room to wait, now go to a bounded `Retries` queue. The loop starts each one as a bound task once there is room under the running limit (bound blocks and retries share it) and never awaits Zebra for a copy. The queue holds at most `running + held` copies and keeps one copy per known body; past that a copy is dropped with a warning. Before a copy starts, the loop checks the dedup ring as the legacy paths do and re-holds it if another task has taken its block up. The legacy paths for blocks no task is on are unchanged: a compact block or reassembled raw block with no bound work under way still runs here as before. The gated compact path, the compact fast path and the raw segment path are split so the inline and task versions share the same code. Adds the held-copy stall test: with Zebra holding every call, a retried legacy or compact copy waits at Zebra while the receive loop buffers other payloads and another bound block reaches Zebra. It fails on the old code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… sent Review items 7 and 8 on #134. Self fan-out: objects let go by a segment 0 went back to the session that sent them, because the node only left out the source of the current datagram. Each object now carries the session it came from, and the node sends it to every session but that one. Record before send: the forwarded record was written inside take, before the send. With no peer, or a socket that refused, the next copy was dropped for resend_after and counted against max_resends. The node now records a send only once every datagram of the object went out to at least one peer. What take lets go is marked for RELEASE_LEFTOVER (500 ms) so the unneeded shards of the same burst do not open a new object; that mark is no send and counts as none. objects_sent_on now counts only objects that went out; the new objects_released counts what the node let go. Tests that check anchors with no receiver attached now read objects_released, and the unit test for a panic while recording now expects the object to have gone (the record comes after the send). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d copy in flight On RawWire::Legacy a copy of an unverified body that is still being sent is a repeat again, as it was on main. Waiting on the send and retrying when it fails is kept for the body-keyed wires (Dual, Bound) only. Review item 2 on #134. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… out a block Review items 1, 2, 9 and 10 on #134, and the inconclusive nit. - New submit_gate.competing_bodies switch, off by default. Off, the gate is the legacy one exactly: it marks a block when a candidate passes and holds its lock across the Zebra call. The bound raw switch turns it on. - Competing-body gate: Zebra's refusal of a body is remembered per header for the dedup window (new RejectReason::RefusedBody), so a replayed body never reaches Zebra again. Only distinct bodies that got an answer, or are at Zebra now, count against strict_bodies_per_block. Repeats and calls that failed in transit, timed out or were cancelled cost nothing. strict_attempts_per_block is gone. - Dry runs read and write only their own marks, keyed by block and body, and spend no live budget. A dry run no longer counts as delivered on the bound path. - dedup_capacity = 0 is refused at config load; the gate also keeps at least one entry. - Zebra's 'inconclusive' gets its own rejection class instead of 'invalid'. - Tests: legacy loop end to end on a real sidecar with every switch off, transient errors never exhausting the limit, a replayed refused body, dry-run isolation, capacity 0, and the legacy gate's unchanged behaviour. Tests that pinned retrying a refused body now pin that it is turned away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`sovright-relay` depends on both research crates by path, but each had its own `[workspace]` table and lockfile and sat in the root `exclude` list. Their exact pins (`sha2 =0.10.9`, `reed-solomon-erasure =6.0.0`, `hmac =0.12.1` and the dev-dependency pins) constrained resolution, and CI tested them only against their own locks. They are now workspace members: the `[workspace]` tables and their Cargo.lock files are gone, shared dependencies use the workspace versions (`sha2`, `serde_json`, `tokio`, `blake2b_simd`), and the rest use caret versions matching what the root lockfile already resolves (`hmac 0.12`, `reed-solomon-erasure 6.0`, `tokio-test 0.4`). The root Cargo.lock only gains their dev-dependency edges. The "Not a workspace member" descriptions are fixed. The separate `research-crates` CI job is removed: the workspace test and clippy jobs now build, test and lint them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The delivery-lease test vectors were named in Turkish. The file is now `frames.json` and its `cerceveler` / `cerceve` / `cerceve_0` keys are `frames` / `frame` / `frame_0`, with the one test that reads them updated. Every value is unchanged (checked by comparing the parsed JSON with the keys mapped). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`free_port` bound port 0, read the port and closed the socket, and the
child bound that port later, so another process could take it in
between (and the ingress RPC probe could then connect to the wrong
listener).
The relay and the ingress RPC now bind 127.0.0.1:0 themselves, and the
script reads the port each bound from the `local_addr=` field of the
line it already logs once listening ("Sovright relay listening",
"Pool submitblock relay gateway started"; both predate #134). A child
that exits before that line fails at once, with its log path. The
first relay's port is kept for the clients and for later relays in a
rollout or swap, which bind it explicitly. `free_port` is gone.
Checked against the relay-node and sovright-p2p-ingress binaries built
here: both report their port, and a restarted relay comes back on the
same one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ci.yml had no `permissions:` block, so its jobs ran with the repository's default token scope. It now sets `contents: read` at the top level; no job needs more (upload-artifact needs nothing extra). cargo-fmt.yml and fuzz-regression.yml get the same read-only block, as they only check out and build. images.yml and release-linux.yml already declare theirs. Every third-party action in .github/workflows is pinned to the full commit SHA its tag points to today, with the version in a trailing comment. dtolnay/rust-toolchain is pinned to the head of its `stable` / `nightly` branch, and since the ref no longer names the toolchain, each use passes `toolchain: stable` / `toolchain: nightly` explicitly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A node sent bound datagrams to every session, and a client's bound send said Ok after handing them to relays that might drop every one. A client with bound work now sends a hello (an authenticated keepalive with a marker hash and a flags word) beside its keepalives. A node with the path on answers it with an ack and marks the session as taking bound objects if the hello says it receives; a node without the path, or from before it, rejects it as it rejects any odd keepalive. The node forwards bound objects only to sessions that said they take them, so a legacy receiver gets the block the legacy way and no bound datagrams. A bound send goes only to relays whose ack still stands (BOUND_ACK_TTL, 180 s), waits up to BOUND_HELLO_WAIT (2 s) for one, and fails with ConnectionRefused when none answered. Tests changed to pin the new behaviour: a raw sender that never sent a hello no longer gets forwards (a_session_sends_an_object_on_once_...), a send through a node with the switch off now fails instead of Ok (zero_leaves_the_path_shut_on_a_real_node), and the socket-error tests mark their refusing relay as having answered so they still reach the socket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Config::validate now runs the submit gate checks and the bound task limit checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The competing-body gate's Zebra call is a sixth site and goes through the shared classifier like the others; the test now checks every site does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any old header with good PoW opened an anchor, one key could take every body a block has, and nothing freed a place when a body was refused downstream. So a key could hold the bound path shut. - Recency: a segment 0 whose header time is more than ANCHOR_HEADER_WINDOW (30 min) behind the newest header that passed here is refused before its PoW check. The newest is capped at this node's clock, so a future-dated header cannot make every other look old. A header that passes ages out anchors that far behind it, and a segment 0 refused for room is PoW-checked when it would age some out, so a table old headers filled at a cold start clears once a newer one comes. - Per-key share: a key pays for at most MAX_BODIES_PER_KEY_PER_BLOCK (2) of one block's max_bodies_per_block (4) bodies. - Release: BoundBlock now names its body (block_hash, body_id) and BoundBlock::refuse gives it back and sends an authenticated BOUND_RELEASE keepalive to the client's relays. A node with the path on takes the anchor away when the release comes from a full-role key and keeps that body out for anchor_ttl. RelayNode has release_bound_body for a caller beside the node. - New counters in BoundNodeUsage: anchors_refused_for_key_bodies, anchors_refused_for_age, anchors_refused_as_released, anchors_released. Tests that pinned the old behaviour now pin the new one: known_gap_one_key_inside_its_limits_can_use_up_a_blocks_bodies became one_key_cannot_use_up_a_blocks_bodies, and known_gap_an_old_header_still_opens_an_anchor became an_old_header_opens_no_anchor_once_a_newer_one_has. a_block_takes_only_so_many_bodies_at_a_real_node now uses three keys. Test bodies of different fills get one header time, since a fill byte in the time field made them decades apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The body buffer made room for a new set by dropping the oldest set, and a block's least advanced one, whether or not it was still arriving. Eight forged segment 0s could push out an honest 2 MB set mid-way, and decoded segments keep no spares, so the body was gone until a resend. A set now counts as arriving for ACTIVE_SET (5 s) after its last segment. BodyBuffer::room_for picks the victims a segment needs, from quiet sets only (least advanced, then newest), or says there are none. Intake asks it before the ledger and answers Refused(Refusal::Sets), which the bridge treats as a lack of room and shelves for a retry, so the newcomer costs nothing and comes back. The ledger's capacity path offers only quiet sets for a segment too; the copy of a whole body at assembly may still take any set, since refusing it would lose a body that is all there. Tests that pinned the old order now let earlier sets go quiet first. the_older_of_two_equal_sets_goes_first became the_newer_of_two_equal_quiet_sets_goes_first. known_gap_a_consistent_body_of_the_header_can_evict_the_honest_one now pins that it cannot while the honest set arrives, only once it is quiet. New: a_set_still_arriving_is_never_given_up_for_a_newcomer and forged_newcomers_cannot_push_out_an_honest_set_under_way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The legacy pacer and the bound sender ran independently on the same socket, so with both under way a client could put out far more than either pace allows. While a bound send is under way, every datagram of the client now takes its turn from one token bucket: the bound path's own pace to every relay, with SHARED_EGRESS_BURST datagrams allowed at once. The bound path alone stays under that pace, and legacy traffic alone is held only by its own pacing, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In dual, a bound copy that failed left the claim uncommitted, so a bound
path that kept failing had every later P2P delivery send the whole
legacy copy again. Once the legacy copy has left, a failed bound copy is
now logged and the body counts as sent; that bound copy is not tried
again. The bound wire is unchanged, and so is a caller that gives up
with the bound copy still under way.
Tests that pinned the old dual behaviour now pin the new one:
- a_bound_failure_lets_the_legacy_copy_go_in_dual: the body is sent, and
repeated deliveries send no legacy copy (the missing "dual mode with a
failing bound send" test).
- the every-end helper takes whether a failure sends; in dual, Failed,
NobodyAnswered and ClientWent leave the waiting copy a repeat.
- after_a_failed_bound_copy_{a_strict,an_unverified}_body_* and
a_copy_during_an_unverified_send_* expect the body sent in dual.
- a_full_bound_queue_* is split: the bound wire still comes back to the
caller; in dual the body is sent by its legacy copy.
- an_owed_body_can_come_again_with_its_header_at_its_limit makes the body
owed by a caller that gives up, which still leaves it open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An object's first datagram reserves the room of its whole object, about eight times its own size, and on a node it kept that for 30 s. The reservation stays, since it is what lets an object that was let in finish, but a node now keeps an open object for NODE_ASSEMBLY_TTL (5 s): a sender sends an object in one burst and a second copy comes after resend_after (2 s), so holding the pool with first datagrams now takes one every 5 s per object. The receiver keeps its own 30 s. Decoded objects waiting for their segment 0 never gave way. They now do when the pool or their key is short of bytes, oldest first, to a session that holds less, after any open object of theirs that can move. With the pool out of open places they would make no room, so they stay. Counted as waiting_gave_way. what_waits_for_its_segment_0_is_not_given_up pinned the old behaviour; it is now what_waits_for_its_segment_0_gives_way_to_a_lighter_session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A spare that had all it needed but was refused its decode while the pool was full sat on the node until its time ran out, room or not: only the receiver ever called revive. The forwarder now marks a session whose chunk was refused while it holds such a spare (Assembler gains has_ready_spares), and BoundForwarder::revive tries those again, admits what decodes and sends it on like any other. The node calls it after a bound send has gone out, which gives room back, and on its cleanup timer. Counted as spares_revived. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bound_usage() was only for tests and logs. render_bound_prometheus_text renders BoundNodeUsage under sovright_relay_bound_*: what the pool holds now as gauges, the rest as counters, and arrivals as one counter labelled by outcome. relay-node appends it to its metrics textfile when the bound path is on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It was public in every build. It is now behind cfg(test) or the test-support feature, which the one downstream user's dev-dependency already turns on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A caller of send_bound_raw_block waited for as long as the client took. It now hears Timeout after BOUND_SEND_TIMEOUT (10 s), which covers the sends queued ahead, a hello's wait, the encoding and the paced datagrams of the largest body. The send goes on, as it does when a caller stops waiting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
contain dropped the bound receiver with no word when its consumer had gone. It now warns, so an operator can tell why bound blocks stopped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relay side of item 3 lets one key hold only two bodies of a block and frees a place when a receiver says a body was refused, but nothing in the sidecar said so: a bound block Zebra refused was only given back. One key sending two wrong bodies first then shut the right one out at the node. A bound block that Zebra calls invalid, or that the gate turns away as a body Zebra refused, now goes back with BoundBlock::refuse. Sidecar tests brought in line with the merged relay fixes: - a bound send through a legacy relay is now refused at the sender (item 6), and a dual send still delivers its legacy copy - a refused body's later copy stops at the relay or the gate - side-by-side bodies of one block come from keys of their own - the evicted-body test dates both headers alike and waits for the set to go quiet (items 3 and 4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main (#137 to #142) hardened every workflow on its own. The workflow conflicts take main's version, which is stricter than this branch's: per-job permissions, persist-credentials: false, newer pins, and no build cache on release builds. The bound-path diagnostic job from #134 is kept in ci.yml with the same hardening, and its matrix test args now go through an env var instead of being expanded into the script. Cargo.lock is main's, plus the two research crates that are now workspace members. No third-party crate version changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main's cargo-deny job (#139) refuses a crate with no license field, and chunk-proof and delivery-lease became workspace members in this PR. They now take the workspace's MIT OR Apache-2.0, like every other crate here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zmanian
marked this pull request as ready for review
November 13, 2026 01:10
5 tasks done
…-body branch Four conflicts, each kept from both sides: - ci.yml: the bound-path diagnostic job and #148's zakura-native job are both kept. - sovright-p2p-ingress dev-dependencies: both sets kept. - nu7_transport_tests.rs: uses #148's LoopbackRelay helper; its RelayBridge now sets this branch's `raw_wire: RawWire::Legacy`. - relay-sidecar main.rs: #145's per-block reconstruction log and metric go into this branch's compact_fast_path_outcome, so bound tasks log it too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This finishes #134 (competing-body dedup for #130). It is #134's branch at 19bad0c plus fixes for the review of that head. #134 itself is untouched.
Before: a replayed or failed copy of a block could spend the legacy gate's 8-call budget and shut the real block out. The new gate also replaced main's gate for everyone, with no switch. On the relay side, one key could hold every anchor for a block, a set still arriving could be evicted, and held copies were retried inline on the sidecar's receive loop.
After: with every switch off, the gate behaves exactly as main's does. The competing-body gate is opt-in (
[submit_gate] competing_bodies = true, and it is forced on with the bound raw switch). Replays and transient errors can't lock a block out. All of the relay and sidecar review items below are fixed and each has a test.How: the gate work is in its own commits (f2ace79, plus the release wiring in c0c28d9), so it can be split into a separate PR if you'd rather land it first. The relay fixes and the sidecar and CI fixes were merged in from two branches (d27f8bb and the following merge).
Changes
Must-fix
strict_bodies_per_block(default 4). Transient errors count for nothing. A body Zebra refused is remembered for the dedup window (300 s), so a replay is turned away without another Zebra call. This replaces the PR's test that pinned the 8-call lockout.competing_bodies, which is off by default. With it off, it is main's gate: dedup is recorded on accept and the lock is held across the Zebra call. There is a new end-to-end test on the real binary with every switch off (tests/legacy_gate_e2e.rs). The ingress legacy wire keeps main's repeat rule for a copy that arrives while another is in flight.Should-fix
dedup_capacity = 0is refused when the config loads.Before merge
zakura-nativejob.chunk-proofanddelivery-leaseare now workspace members, and the separate research CI job is gone. They take the workspace license, which main's cargo-deny check needs.cerceve.jsonand itscerceve*keys are nowframes.jsonandframe*.matrix.pyport race. The children bind port 0, and the script reads the port back from their logs.Nits
inconclusiveresult gets its own rejection class and metric.BoundTaskLimitscan be set in config or on the command line (bound_tasks_running,bound_tasks_budget_secs,bound_tasks_held; defaults unchanged).BlockSender::for_testsis behindtest-support.send_bound_raw_blocktimes out after 10 s.containlogs when it ends a client's bound path.Missing tests added
dedup_capacity = 0Defaults picked (each is a const or config value and easy to change)
strict_bodies_per_blockMAX_BODIES_PER_KEY_PER_BLOCKANCHOR_HEADER_WINDOWACTIVE_SETNODE_ASSEMBLY_TTLBOUND_SEND_TIMEOUTSHARED_EGRESS_BURSTanchor_ttl(90 s)Behaviour changes to review
Merged with main after #148
LoopbackRelaytest helper setsraw_wire: RawWire::Legacy, so Land #143 (Zakura native block source) with cargo-vet coverage #148'sforwarded_atdedup keeps working. On the opt-inDualandBoundwires, the Zakura native source would refetch announced blocks.compact_fast_path_outcome, so bound tasks log them too.Still open
known_gap_a_held_legacy_zebra_call_still_stops_this_taskpins this.matrix.pymatrix was not run here.Refs #134, #130.
Checklist
cargo fmt --allpassescargo clippy --workspace --all-targets -- -D warningspassescargo test --workspacepasses (115 suites, 2374 passed, 0 failed, 10 ignored, after merging main at 127a30a)cargo vet --lockedandcargo denypass (no new third-party crate versions)🤖 Generated with Claude Code