Skip to content

fix: share reproducible Ferrocene repositories - #52

Merged
4og merged 2 commits into
mainfrom
fix/ferrocene-reproducible-repositories
Nov 18, 2026
Merged

4og merged 2 commits into
mainfrom
fix/ferrocene-reproducible-repositories

Conversation

@4og

@4og 4og commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Problem

The Ferrocene repository rule downloads and extracts the compiler archive, optional coverage tools and optional Miri sysroot, but does not declare the resulting repository reproducible. Bazel therefore creates separate extracted copies in each output base instead of sharing them through repo_contents_cache, even when the repository inputs and checksummed archives are identical. The download cache alone does not prevent this duplication.

In two existing Baselibs worktrees, each QNX x86_64 copy occupied about 916 MB and each Linux x86_64 copy about 925 MB. Both pairs had 222 matching payload paths but no shared cross-worktree file identities. This is the same class of issue addressed by eclipse-score/bazel_cpp_toolchains#183.

Fix

Return ctx.repo_metadata(reproducible = True) only after all archive extractions and generated file writes succeed.

The declaration is guarded by both API availability and a nonempty primary sha256. Older Bazel versions and unchecksummed custom primary archives retain their existing behavior without a reproducibility declaration. Existing checksum requirements for optional coverage and Miri archives remain unchanged.

Add focused repository-context regression checks and refresh the two affected extension digests in the lockfile. Archive packaging and cross-architecture host-tool duplication are unchanged.

Verification

Verified commit: 4597cfa66f31f573b106f93c8fb258c217d0f746.

Local checks passed with Bazel 8.6.0:

buildifier -mode=check -lint=warn extensions/BUILD.bazel extensions/ferrocene_toolchain_ext.bzl
bazel mod deps --lockfile_mode=error
bazel test --lockfile_mode=error --config=smoke //examples/basic/... //extensions:ferrocene_toolchain_ext_test
bazel coverage --lockfile_mode=error --config=smoke //examples/basic:basic_test

The five regression checks cover declaration after generation, optional checked archives, an empty primary checksum, a context without the metadata API, missing optional checksums, and extraction/generation failures. Removing the declaration in memory caused the expected regression failures. Example tests passed, and coverage data contained executed lines from examples/basic/src/lib.rs. Compatibility with an older API was mock-tested, not tested with an older Bazel executable.

For direct cache-sharing verification, both existing Baselibs worktrees used Bazel 8.8.2 with shared repository, repository-contents and disk caches. Each temporarily pinned this exact local commit through a local Git override. After lock refresh, A fetched only the QNX and Linux x86_64 Ferrocene repositories first, then B fetched the same variants with --lockfile_mode=error. Profiles were captured for both phases.

Direct evidence Before After
Output-base entries, both variants Independent extracted directories Both worktrees symlink to the same shared directory per variant
Corresponding payload paths with matching device/inode identities 0 per variant 222 per variant
QNX unique-tree allocation across both worktrees 1,831,538,688 bytes 915,771,392 bytes
Linux unique-tree allocation across both worktrees 1,850,781,696 bytes 925,396,992 bytes

The resolved directories and representative cargo and rustc device/inode identities matched between A and B. Both existing comparison helpers returned SHARED and exit 0. B left all warmed payload identities and allocations unchanged, added no cache keys or Ferrocene trees, and added zero extracted Ferrocene payload bytes. Combined unique-tree allocation fell by 1,841,152,000 bytes, excluding small output-entry and marker overhead.

Profiles distinguish extraction from cache lookup: A executed two Ferrocene repository implementations and six archive extractions. B executed zero Ferrocene implementations, download/extract calls or extraction events; its repository-fetch spans were cache lookup/materialization only.

A's small QNX integration build also passed:

bazel build --lockfile_mode=error --config=score-qnx-x86_64-sdp_8.0.5 //score/result:error

Build success and action-cache hits were not used as proof of repository sharing. No caches were cleared, no fetch was forced and no SDK/output directory was manually deleted. After recording the evidence, both Baselibs module/lockfiles were restored byte-for-byte to their pre-test state, preserving the existing C++ override, branches, unrelated edits and cache configuration. The restored 0.10.0 pin does not retain the fix for future rematerialization until this change is adopted.

4og and others added 2 commits November 16, 2026 14:30
Declare checked Ferrocene repositories reproducible after successful generation, with compatibility guards for legacy Bazel and unchecksummed custom archives. Add focused repository-context regression checks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Include the Ferrocene repository-context regression target in the existing example smoke-test job.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@4og
4og force-pushed the fix/ferrocene-reproducible-repositories branch from d3d086a to e17d833 Compare November 16, 2026 13:31
@4og
4og requested a balanced review from Copilot November 16, 2026 13:53

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the change enabling reproducible Ferrocene repository caching.

No open issues 🎉

What changed in this PR

This PR enables Bazel to share reproducible Ferrocene repositories through repo_contents_cache, reducing duplicate compiler extractions across output bases.

Changes:

  • Declare repositories reproducible after successful extraction and file generation, only when the primary checksum and metadata API are available.
  • Add regression checks for reproducibility, compatibility, and failure handling, and run them in CI.
  • Refresh the two affected extension digests in the lockfile.
File Description
MODULE.bazel.lock Refreshes Ferrocene extension digests.
extensions/​ferrocene_toolchain_ext.bzl Adds the guarded reproducibility declaration.
extensions/​ferrocene_toolchain_ext_test.py Tests metadata timing, checksum guards, compatibility, and failures.
extensions/​BUILD.bazel Registers the extension regression test target.
.github/​workflows/​examples.yml Runs extension tests alongside example tests.

Review effort: Balanced

@4og
4og merged commit 181f87f into main Nov 18, 2026
9 checks passed
@4og
4og deleted the fix/ferrocene-reproducible-repositories branch November 18, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants