Skip to content

Add --cpus and --memory resource limits - #25

Merged
anoek merged 5 commits into
mainfrom
resource-limits
Oct 31, 2026
Merged

anoek merged 5 commits into
mainfrom
resource-limits

Conversation

@anoek

@anoek anoek commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

Summary

Adds --cpus and --memory to limit how much CPU and memory a sandbox can use, enforced with cgroup v2.

  • --cpus=2, --cpus=1.5: limit total CPU time (cpu.max), not pinning to specific cores, so nproc inside still reports every CPU.
  • --cpus=-2: negative values mean "all but N", so on a 32 CPU machine this allows 30. "Available" comes from std::thread::available_parallelism(), which equals nproc unless already under a CPU quota.
  • --memory=512M, --memory=4G: sets memory.max and sets memory.swap.max to 0 (when swap accounting exists) so the limit means what it says. Going over the limit OOM-kills a process in the sandbox.
  • unlimited removes a limit set by a less specific source, e.g. --cpus=unlimited to run once without the cpus = -2 from your global config.
  • Also configurable via SANDBOX_CPUS / SANDBOX_MEMORY and cpus = -2 / memory = "8G" in config files, with the usual precedence. sandbox config cpus memory shows the resolved values.

How it works

  • Sandboxes with limits get a cgroup at /sys/fs/cgroup/sandbox/<uid>-<name>. The parent sandbox cgroup is created on first use, and cpu / memory are enabled in cgroup.subtree_control as needed.
  • PID 1 is started with CLONE_INTO_CGROUP (Linux 5.7+), which also makes that cgroup the root of the sandbox's existing cgroup namespace. Inside, /proc/self/cgroup is 0::/ and /sys/fs/cgroup/cpu.max etc. show the sandbox's own limits.
  • sandbox exec moves itself into the sandbox's cgroup before setns. Previously, joined processes stayed in the caller's cgroup (e.g. the terminal's systemd scope), since joining a cgroup namespace doesn't move a process between cgroups.
  • stop and delete find the cgroup from the sandbox's PID 1 (so delete, which runs as uid 0, finds it too), kill the sandbox as before, then remove the cgroup once the waiting sandbox commands have exited. They aren't killed with cgroup.kill, so they still report their child's exit status.
  • Limits are saved in settings.json (with serde(default), so settings from older running sandboxes still load). Running an existing sandbox with different limits errors and asks you to stop it first, the same as network settings.
  • Without limits, nothing changes: no cgroup is created.
  • Nested sandboxes can't manage cgroups (/sys/fs/cgroup is read-only inside a sandbox). Instead of failing, a nested sandbox warns and runs without its own cgroup, sharing the outer sandbox's limits. The warning names them, e.g. (cpus=2 memory=unlimited).
  • Outside a sandbox, if cgroups can't be managed (cgroup v1, a read-only /sys/fs/cgroup, or inside a container), starting a sandbox with limits fails with an explanation, before anything is mounted.

Testing

  • make test on a host with writable cgroup v2, with limits set in the user config (cpus=-2, memory="128G"): 321 passed, 0 failed (3 ignored, which the suite skips by design). That run was on the stack's top branch (Add tests for logger levels and uncovered error paths #26), before a final test-only change; the ai_test_resource_limits file (19 tests) passes again after that change.
  • tests/ai_test_resource_limits.rs covers:
    • config from CLI, environment and config files, plus unlimited and invalid values
    • limits visible inside the sandbox, and processes that join a running sandbox landing in its cgroup
    • negative CPU counts, real OOM enforcement, and memory-only limits
    • cgroup removal on stop and delete, replacing a stale cgroup, a busy cgroup, and a name collision between two storage directories
    • changed limits being rejected, nested sandboxes, and setup failures cleaning up after themselves
  • Tests that check defaults skip config files, so the results don't depend on the user's config.
  • Unit tests for CPU and memory parsing in src/config/limits.rs.
  • Every line added by this PR is covered.
  • clippy has no new warnings, and cspell is clean.

Notes

  • CLONE_INTO_CGROUP isn't in the pinned libc crate yet, so it's defined in src/util/clone3.rs.
  • cpuset pinning (which would change what nproc reports inside) isn't included.

@anoek
anoek added this pull request to stack #27 October 31, 2026 12:37
@anoek
anoek removed this pull request from stack #27 October 31, 2026 13:30
@anoek
anoek changed the base branch from main to fix-ci October 31, 2026 13:31
@anoek
anoek added this pull request to stack #30 October 31, 2026 13:31
Base automatically changed from fix-ci to main October 31, 2026 14:14
anoek added 5 commits October 31, 2026 08:14
Sandboxes can now be limited to a number of CPUs and an amount of memory
using cgroup v2. --cpus accepts fractional values, and negative values
mean all but that many CPUs, so -2 on a 32 CPU machine allows 30.
--memory accepts sizes like 512M or 4G and disables swap for the sandbox.
Both can also be set with SANDBOX_CPUS / SANDBOX_MEMORY or in config files.

Sandboxes with limits get a cgroup at /sys/fs/cgroup/sandbox/<uid>-<name>.
PID 1 is started in it with CLONE_INTO_CGROUP, which also makes it the
root of the sandbox's cgroup namespace, and processes that join the
sandbox move into it before entering its namespaces. Stopping the sandbox
removes the cgroup. Limits are recorded in the sandbox settings, and
changing them requires stopping the sandbox, like network settings.
Tests that check default, environment, or changed limits now skip config
files, so limits set in ~/.config/sandbox/config.toml don't change the
results.
A sandbox started inside another sandbox can't manage cgroups, so with
limits in a config file nested sandboxes refused to start. They now warn
and run without a cgroup of their own, which leaves them subject to the
outer sandbox's limits. Outside a sandbox, failing to set up limits is
still an error.

--cpus and --memory (and SANDBOX_CPUS, SANDBOX_MEMORY, and config files)
now accept "unlimited", which removes a limit set by a less specific
source.

Also read memory.events in the same command as the allocation in the
memory enforcement test, so the result doesn't depend on which process
the OOM killer picks.
delete stops sandboxes as uid 0, so looking for the cgroup by the name
we'd create (<uid>-<name>) missed it and the cgroup was never removed.
Find the cgroup from the sandbox's PID 1 instead, accepting any uid.

Also stop writing cgroup.kill when removing a cgroup. By that point the
only processes left in it are the `sandbox` commands waiting on the
killed sandbox processes. Killing those cut off their exit status
reporting (and truncated coverage profiles mid-write). They exit by
themselves once their child is gone, which the removal already waits for.
The swap accounting test now skips config files, so a cpus limit in
~/.config/sandbox/config.toml can't turn it into a cpus+memory test and
leave the memory-only path untested.
@anoek
anoek merged commit bfad2f8 into main Oct 31, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant