Skip to content

Let pull-request runs read the default branch's manifest and write their own - #186

Open
mdarocha wants to merge 2 commits into
Mic92:mainfrom
mdarocha:feat/pr-manifest-union
Open

mdarocha wants to merge 2 commits into
Mic92:mainfrom
mdarocha:feat/pr-manifest-union

Conversation

@mdarocha

@mdarocha mdarocha commented Nov 3, 2026 •

Copy link
Copy Markdown

Generated by Claude Opus 5.5

Requires #163. Without it, gc deletes superseded manifests by key across all refs, so one pull request's m3pr# chain would delete another's head; #163 makes that cleanup per ref.

A pull request's cache lookup returns its own m3# manifest before the default branch's (GitHub searches the current ref first), so once a PR has committed a manifest it no longer sees anything the default branch builds afterwards. This makes PR runs write their own family, m3pr#N, and load m3# (which only the default branch writes) together with it, merged with Manifest::merge.

  • PR mode is detected from GITHUB_REF (refs/pull/*), or set with --pr-manifest / the pr-manifest action input.
  • gc cleans both families and protects packs referenced by PR manifests.
  • Tests and docs (README, action README, docs/cli.md) are updated.

mdarocha added a commit to mdarocha/nix-magic-setup that referenced this pull request Nov 3, 2026
Switches the hestia step from Mic92/hestia v3.1.0 to the mdarocha/hestia
fork, release v3.1.0-mdarocha.1. The fork carries the gc fix from
Mic92/hestia#163 and the pull-request manifest union from draft
Mic92/hestia#186, so pull requests can write to the cache again: the
default-branch-only write gate (`scripts/plan-hestia-writes.sh` and its
step) is removed. The README now only notes that the action runs the
fork.

Tested on throwaway branches in this repo and in pondinfra
(mdarocha/pondinfra#322): the action downloaded the fork's release,
verified its attestation, and a pull request substituted both its own
and the default branch's paths.
@mdarocha

mdarocha commented Nov 3, 2026 •

Copy link
Copy Markdown
Author

Generated by Claude Opus 5.5

Results from a private NixOS configuration repository (two hosts, GitHub-hosted 2-core runners, one job per host that builds the host's install and VM tests), running this branch through a composite action.

The case this targets is repeated runs of the same pull request. Before, pull requests ran with read-only: true, because a PR that wrote m3# hid everything the default branch cached afterwards and gc could delete the default branch's chain (#162). So every push to a PR rebuilt whatever the PR changed. With this branch, the PR writes m3pr# and reads it together with m3#.

Same PR, same check derivations in every run (the later read-only push only changed the workflow), both host jobs in minutes:

Run Host A Host B Built in host B job
read-only, first push 15.6 19.8 everything the PR changed
read-only, next push 11.5 16.8 the same again
this branch, first run 11.1 17.6 109 derivations
this branch, re-run 2.2 3.3 3 derivations (4 substituted from hestia)

The re-run logged hestia serve: manifest m3pr#4 unioned with m3#: 1504 paths; the first run logged m3pr#0 unioned with m3#: 1314 paths, i.e. it saw the default branch's 1314 paths before writing anything.

Costs observed:

  • Cache space: each PR's own chain was about 30 MB, since most of what a PR needs is already on the default branch.
  • Load time: loading both manifests added no visible delay; the drain's load stage took about 6 s, as before.
  • Sharing between different pull requests still doesn't happen, because GitHub scopes PR caches to refs/pull/N/merge. Two PRs with the same change both build it.

In this repository it's worth it. A second push to a PR went from about 28 job-minutes to about 5.5. A gc run on the default branch while PR chains existed kept each ref's newest manifest; that run used this branch together with #163, which it requires.

@mdarocha
mdarocha marked this pull request as ready for review November 3, 2026 15:30
GitHub resolves a cache key in the run's own scope before the default
branch. A PR that committed to the default manifest chain (m3#) shadowed
every later default-branch manifest, so it never saw what main cached
after the PR's first write, and GC's cleanup mixed the chains.

Jobs on refs/pull/* (or with --pr-manifest / the pr-manifest input) now
commit to a separate chain (m3pr#) and read the union of it and the
default chain. GC cleans m3pr# per ref and treats the packs of every PR
manifest as live.

Builds on Mic92#163 (per-ref manifest cleanup by cache ID).
@mdarocha
mdarocha force-pushed the feat/pr-manifest-union branch from 81fb5c1 to 5d44e9e Compare November 3, 2026 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant