Skip to content

Descriptors a zone hands over close without waiting on them: SO_LINGER off, queues taken - #608

Draft
DevomB wants to merge 3 commits into
proof-compositor-stackfrom
broker-release-received
Draft

DevomB wants to merge 3 commits into
proof-compositor-stackfrom
broker-release-received

Conversation

@DevomB

@DevomB DevomB commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Based on the compositor stack's proof (#585), which reworked the daemon's request reading; retargeted to main once that lands. From a non-author review of the broker with a hostile zone (LOW, and one INFO).

The LOW. A zone could hold its own launcher in close(2).

  • It opens a TCP socket, fills its send queue against a peer that never reads, sets SO_LINGER {1, INT_MAX}, sends version with the socket attached, and closes its own copy.
  • The broker's drop of the last copy then waits on the linger, on the launcher's supervision loop. That loop then relays and reaps nothing, and the zone's cleanup (grants revoked, the volume closed and its key wiped) waits until something signals the launcher.

What changes (compartments/kryptikd/src/serve.rs, shared by the broker and the daemon):

  • recv_with_fds turns SO_LINGER off on every descriptor as it arrives.
    • Its control buffer has room for every descriptor one message can carry (the kernel's 253), because the kernel closes what does not fit, on the receiving thread.
    • Past twelve, a request is still refused, and all of its descriptors are released.
  • Received, the descriptors a peer handed over, releases them when dropped. Every refusal path in the broker and in the daemon's request reading now ends that way.
  • release() shuts a Unix socket handed over and takes its queue before closing it, and releases each queued descriptor the same way (a worklist, so nesting ends). Otherwise the kernel would close those with the socket, on this thread.
  • The broker reads a request's later bytes with recvmsg, not recv(), so descriptors sent with them are released and not dropped by the kernel. It releases each connection it is done with.
  • The daemon shuts each connection for reading once its request is read, and releases whatever is queued behind it. No verb reads a connection after its request.

The INFO. time-offset found the floor (floor_of_this_system(), which can check the release under the update chain's lock) before refusing a zone without the network. The floor is now a closure, looked for only after that refusal.

Tests (kryptikd unit tests):

  • a_lingering_socket_is_released_at_once: a loopback TCP socket with its send queue full and SO_LINGER {1, 30} is released in under 5 s.
  • a_socket_queued_on_a_connection_is_released_with_it: the same socket queued on a Unix connection goes with release() of the connection. With release_queued(), the connection read to its end, the socket goes too, and the peer can send nothing more.
  • every_descriptor_is_taken_and_past_twelve_refused: the control buffer is CMSG_SPACE of 253 descriptors, thirteen are refused, and two are taken.
  • a_datagram_queue_is_taken_past_its_empty_datagrams: an empty datagram ahead of the lingering socket on a datagram pair does not stop the drain.
  • only_nic_zone_reports_time: the floor is not looked for before a refusal.

Left as it is:

  • Seqpacket bound: a seqpacket socket's 0 can be an empty message or its end, so its drain keeps the bound of 1024 messages. Datagram queues are drained to EAGAIN, past empty datagrams, and streams end at their 0, as they carry no empty message.
  • Descriptor limit: a wide enough nest of Unix sockets can still reach the launcher's descriptor limit mid-drain. Only Unix sockets are held while their queues are taken; everything else closes at once, defused. The kernel then closes what does not fit, so the launcher can still be held until something signals it, as before.

…R off, queues taken

A zone could hold its launcher in close(2). It hands the broker a TCP socket whose send queue a
peer never reads, with SO_LINGER {1, INT_MAX}, and closes its own copy. The launcher's close of
the last copy then waits on the linger, relaying and reaping nothing, and the zone's cleanup
(grants revoked, the volume closed and its key wiped) waits with it.

Every descriptor recv_with_fds takes now has SO_LINGER turned off as it arrives. The kernel
closes descriptors in three more places on the receiving thread, and each is closed off:
- what does not fit the control buffer: it now has room for every descriptor a message can
  carry (253), and past twelve a request is refused with all of them released;
- descriptors sent with later bytes, which the broker read with recv(): it reads with recvmsg;
- descriptors still queued when a connection goes: the broker shuts and drains each connection
  it is done with, and the daemon each one whose request it has read.
release() also shuts and drains a Unix socket handed over before closing it, and releases
whatever was queued on it the same way.

time-offset also refuses a zone without the network before finding the floor, which can check
the release under the update chain's lock.
…zone's launcher up to a check's 60 s

Each piece is answered under the 5 s deadline, but the piece that completes the list and its signature, and the hourly newest pointer, also wait on a check of the release, which CHECK_DEADLINE bounds at 60 s.
…sockets are held while it is

An empty datagram queue answers EAGAIN even when shut, so a 0 ends the drain only on a stream,
which carries no empty message: a datagram socket's empty datagrams no longer stop the drain
ahead of the descriptors behind them. A seqpacket's 0 may be an empty message or its end, so it
keeps the bound of 1024 messages. Descriptors taken from a queue that are not Unix sockets close
at once, already defused, so only the sockets whose own queues are still to take are held.
DevomB added a commit that referenced this pull request Oct 16, 2026
# Conflicts:
#	compartments/kryptikd/src/serve/tests.rs

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant