Repository navigation
ci: close the release allowlist bypass, harden adoption-harness, require mcp-extra (#1033) - #1186
Open
pengfei-threemoonslab wants to merge 11 commits into
Open
pengfei-threemoonslab wants to merge 11 commits into
pengfei-threemoonslab wants to merge 11 commits into
Conversation
…ire mcp-extra (#1033) The release-toolchain allowlist skipped every indented line as a hash continuation, but pip joins a line onto the previous one only after a trailing backslash: an indented `evilpkg==1.0` below a finished requirement passed the guard ("OK: every locked distribution is on the allowlist.") while pip's own parser installed it. The guard also split lines with `read`, where pip splits at CR, FF, NEL and U+2028 too and honours a PEP 263 `coding:` declaration. It now refuses those bytes and declarations, tracks continuation state, and accepts an indented line only as one `--hash` token after ` \`. Tests run the step on hostile lockfiles (each confirmed against pip's parser), on valid hash continuations, and on every committed constraints/release-*.txt, and the hollow allowlist comparison now reads the `allowed=` list instead of every word in the step. adoption-harness.yml passes dispatch inputs through quoted environment variables, scopes ANTHROPIC_API_KEY to the Run harness step, pins actions to ci.yml's SHAs, adds a timeout to smoke, keeps no checkout credentials, and installs hash-locked closures: constraints/dev.txt, build-backend.txt and a new constraints/harness.txt, compiled from harness/requirements.txt with the first two as constraints (LockTarget.constraints) and checked as co-installed. tests/test_workflow_hardening.py holds every workflow to the same rules. ci.yml: coverage, a required check, now needs mcp-extra and fails unless both legs succeeded, so a failing, cancelled or skipped mcp-extra job blocks a merge without a ruleset change.
…flow lint (#1033) Review follow-up on #1186. The release-toolchain guard's version pattern accepted a "version" ending in an archive suffix, which pip reads as a link to a local file rather than an index pin, while the guard recorded the distribution name. The pattern is now a normalized PEP 440 public version; every committed pin in constraints/ still matches. The pip oracle in the tests now uses install_req_from_parsed_requirement and reports a link as a link, so a file-versus-index difference fails the comparison too. The workflow lint reads `secrets` in any spelling, checks job container and service environments, and derives the quoted-input variables from every dispatch input a harness step's env reads. The CHANGELOG entry describes the line-boundary and encoding cases in general terms.
…he test-build audit is unchanged (#1033)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
Three CI and release-integrity gaps from the 2026-10-12 engineering review.
1. The release-toolchain allowlist did not read the lockfile the way pip does.
.github/actions/install-release-toolchain/action.ymlis the reviewed gate on whatstage,publish(id-token: write) andfinalizeinstall fromconstraints/release-publish.txt. Its loop skipped every line that started with whitespace, as a "hash continuation". pip only joins a line onto the previous one when that line ends with a backslash. So this lockfile:made main's guard print
OK: every locked distribution is on the allowlist.(exit 0), while pip 26.0.1's ownparse_requirementsreads['uv==0.12.9', 'evilpkg==1.0'].The same review found that the guard and pip could also disagree on where a line ends and on how the file's bytes are decoded. Both are closed here: the guard refuses any lockfile whose line boundaries or text encoding pip could read differently. The tests list the cases.
A third disagreement predates this PR. The guard recorded
uvfor a "version" ending in an archive suffix, such asuv==0.13.6.tar.gz, but pip reads that as a link to a local file, not an index pin. Review of this PR found it, and it is fixed here (see Review follow-up).The test meant to catch this was hollow.
test_every_publication_side_job_constrains_what_it_installsbuilt its "allowlist" from every lowercase word before agrep -oEthat the script no longer contains. On main that came to 226 words, comments included, so a lock that addedbuildorhookspassed it. No test fed the guard an indented line.2.
adoption-harness.ymlinterpolated dispatch inputs into shell, in a job holding an API key.--matrix=${{ inputs.matrix_file }} --budget-usd=${{ inputs.budget_usd }}was expanded into therun:text.ANTHROPIC_API_KEYwas in therunjob's environment for every step. It was also the only workflow that pinned actions by tag, installed unhashed from the index (including an unpinnedpip install --upgrade pip), and had a job (smoke) with no timeout.3.
mcp-extrablocked nothing. The "Protect main" ruleset requirescoveragebut neithermcp extra (floor)normcp extra (newest), and no required jobneeds:them. A PR that broke the[mcp]extra's server stayed mergeable.The fix
1. Release-toolchain guard (
.github/actions/install-release-toolchain/action.yml, still one self-contained bash step that the suite executes).Same bytes, same lines. It refuses any lockfile whose line boundaries or text encoding pip could read differently; the step's comment lists the cases. Non-ASCII text, such as the em dashes in the committed headers, is accepted in comments only, because every other line must match an ASCII pattern.
Continuation state, as in pip. A requirement starts at the left margin as an exact
name==versionpin with an optional; marker. The version must be a normalized PEP 440 public version: no epoch, no local label, no archive suffix. An indented line is accepted only as one--hash=sha256|384|512:<hex>token, and only on a line that follows one ending in\. Blank lines and comments never continue, and they end a continued requirement, as in pip'sjoin_lines. Everything else is refused with its line number:-(pip starts reading options at the first word beginning with-),#,\,$or;;===, extras, inline--hash.The allowlist and its error text are unchanged.
Why a parser and not
pip install --dry-run --report. The report needs the index at guard time, so the suite cannot run it offline against crafted lockfiles, and the step's comment says that ability is the point. More importantly, to learn an sdist's metadata pip must build it, which runs the candidate's build hooks in the token-bearing job before the allowlist has decided anything. The parser runs no project or package code. It closes the differential by accepting only the one shape uv writes. A differential test then holds it to pip's own parser on every committed lock and every hostile case.2.
adoption-harness.yml.env:variables (MATRIX_FILE,SHIPGATE_HARNESS_BUDGET_USD,AGENT_FILTER). They are expanded inside quotes into a bash array, so each one is a single argument, never shell syntax.SHIPGATE_HARNESS_BUDGET_USDfeeds--budget-usd, which is the spellingdocs/adoption-harness-automated.mddocuments.ANTHROPIC_API_KEYis set on theRun harnessstep only.ci.ymluses: checkout3d3c42e5…(v7.0.1, as before), setup-python6d4fd22b…(v7.1.0, as before), upload-artifact043fb46d…(v7.0.1; the floating@v7is now v7.0.2).smokegetstimeout-minutes: 20;runkeeps 90. Both checkouts setpersist-credentials: false, because live agents run commands in that job.constraints/dev.txt,constraints/build-backend.txtand a newconstraints/harness.txtwith--require-hashes, then-e . --no-deps --no-build-isolationandpip check, the same sequence as CI.python -m pip install --upgrade pipis gone.claude-agent-sdk, so there is a new one. It is compiled byscripts/update_locks.pyfromharness/requirements.txtand registered inLOCK_TARGETS. Compiled alone, it resolvedtyper==0.29.1againstdev.txt's0.27.3, so the third install would have moved the closure under the first.LockTargettherefore gainsconstraints.compile_commandpasses--constraint constraints/dev.txt --constraint constraints/build-backend.txtfor this lock, andCO_INSTALLEDgains a second group holding all three locks, the adoption harness's environment; the first group (dev.txt,build-backend.txt) stays exactly what CI installs, so the dependency audit'stest-buildsubject from Audit exact pins from reviewed dependency locks #1216 is unchanged. A full regeneration compiles it last, and it is reproducible: recompiling gives the same bytes. It was compiled with uv 0.13.6, the versionrelease-publish.txtpins.claude-agent-sdk==0.2.183, 41 distributions.3.
ci.yml.coverage(required) now hasneeds: [suite, mcp-extra]and a step, Require both mcp-extra jobs to have passed, that fails unlessneeds.mcp-extra.result == 'success'. This follows the existing suite-shard pattern.coveragealready runsif: ${{ !cancelled() }}, which looks at the workflow run, not at its dependencies. So a failed, timed-out (cancelled) or skipped mcp-extra leg still startscoverage, andcoveragefails by name before checkout. No ruleset change is needed. A whole-run cancellation skipscoverageas before, and the cancelledsuite (n)required checks still block.mcp-extrausually finishes well inside the suite shards' time, so this should add no wall-clock time.Docs. CHANGELOG (
### Packaging, three bullets),CONTRIBUTING.md§ Dependabot (recompileharness.txttogether withdev.txt), and the lock table indocs/release-runbook.md.Tests
tests/test_release_pipeline.pytest_the_allowlist_refuses_a_requirement_it_would_not_have_seen, eight cases: two indentation shapes (one is the reported case), five line-boundary shapes and one encoding shape.test_pip_installs_what_each_refused_lockfile_hides: pip's own parser reads{"uv", "evilpkg"}from each of those eight, so none of the refusals is a strawman. It skips if pip's internal parser is not importable.test_the_allowlist_accepts_valid_hash_continuations: multiple hashes,sha512, indented# viablocks, a blank line, a marker, a non-ASCII comment, mixed-case names, no final newline.test_the_guard_reads_every_committed_release_lock[3 locks]: what the guard records equals the lock's pins.release-publish.txtpasses outright.release-seal.txtandrelease-build.txt(build,hatchling) get as far as the allowlist and are refused there, and only there; see Judgement calls.test_the_guard_and_pip_read_the_same_requirements[3 locks]: the guard's names equal pip's for every committed release lock.allowed="…"assignments (_allowlisted_names).test_the_allowlist_parse_is_the_guards_ownchecks that the parse cannot become "every word" again.test_a_lock_installed_after_others_is_compiled_against_themcovers the new lock's constraints, its compile order and its co-install grouping.coverageneedsassertion is updated.tests/test_workflow_hardening.py(new). Every workflow in.github/workflows/, plus the composite action, is checked for:${{inside anyrun:;uses:(step or reusable-workflow job) pinned to a 40-hex SHA;secrets(any spelling) orgithub.tokenin workflow, job, job-container or serviceenv;timeout-minuteson every job that runs steps.The 11 other workflows and the composite action already pass all four rules, so the generalisation cost nothing. Harness-specific tests check:
Run harness;pip installis--require-hashes --requirement constraints/*.txtor the--no-deps --no-build-isolationeditable install;ci.yml's.The mcp gate test checks
coverage'sneeds,ifand step order. It also executes the gate step withsuccess/failure/cancelled/skipped/empty, and onlysuccessexits 0. The shard weight is appended totests/shard_seconds.json.Proof the new tests fail on main. These ran on a
git archive origin/maintree with only the test files copied in.test_release_pipeline.py -k "allowlist or guard or hidden or pip_installs or publication_side": 20 failed. All 8 hidden-requirement cases failed, because main's guard printedOK:. The other 12 were grammar refusals.test_workflow_hardening.py: 13 failed. These were the 4 generic rules (only foradoption-harness.yml), the 3 harness tests and the 6 mcp-gate tests. Every other workflow passed.Harness lock, end to end. In a fresh Python 3.13 venv, the job's exact install sequence ran with
--require-hashes, thenpip check(clean), thenpython -m harness.adoption smoke, which exited 0 and importedclaude_agent_sdk0.2.183.scripts/verify_dependency_lock.pyreportsOKfor all five locks.Gate (local, at
d6c3e309).ruff check .: exit 0.compileall -q src tests: exit 0.scripts/verify_dependency_lock.py: exit 0.pytest -n 4 -m "not perf" --ignore=tests/test_adapter_static_only.py: 23800 passed, 7 failed, 10 skipped, 120 xfailed. The machine was at load 100–330, so the run took 2 h 9 min. The 7 failures:test_release_suite_gate.py::…[ci.yml-coverage], which assertedneeds == "suite". It now pins["suite", "mcp-extra"]for CI and stays strict for the release gates. The file re-runs green..venv:test_schema_roundtrip(3) andtest_published_schema_freeze(1). These pass in the lock-exact venv (exit 0), and so doesscripts/generate_schemas.py --check(exit 0).test_check_unmodelled_host_config_keys[local_settings_enabled_plugins-worktree|git_range]. The fixture'sgit commitexits 1 under this machine's global git-ignore. These fail identically on agit archive origin/maintree.pytest tests/test_adapter_static_only.py: exit 0../shipgate check --base origin/main --head HEAD:require_review, two mediumPROTECTED-SURFACE-UNCLASSIFIEDitems foradoption-harness.ymlandci.yml, and one low non-widening row for the action-reference change. Nothing isblocked, so the advisoryverify/verify-selfjobs should not fail on it.Compatibility
.githubis registered as CI only).LockTargetgains an optional field whose default is empty.Protected paths
This PR touches protected paths and needs owner review before merge:
.github/actions/install-release-toolchain/action.yml: the allowlist guard that runs instage,publishandfinalize..github/workflows/adoption-harness.yml.github/workflows/ci.yml:coverageneedsmcp-extra.constraints/harness.txt(new lock),scripts/verify_dependency_lock.py,scripts/update_locks.py: the lock gate and its generator.CONTRIBUTING.mdNot touched:
.github/CODEOWNERS,SECURITY.md,MAINTAINERS.md,action.yml,scripts/install_action_engine.py,.github/workflows/release-engine-smoke.yml.Judgement calls
--hash,--hash sha256:…with a space,===, extras, epochs, local version labels, markers with-, and a trailing continuation. uv never writes any of these. A future uv format change would fail closed, with a line-numbered message.release-*.txtpasses" is met by design forrelease-publish.txtonly. The seal and build locks are installed elsewhere. Making them pass here would require allowlistingbuildandhatchlingin the jobs that hold the token. Whether that reading of the criterion is acceptable is for the owner to decide. The tests prove that the guard parses both of those locks completely, agrees with pip on them, and refuses them only at the allowlist.Review follow-up (
19374a09)One new commit, no force-push. It addresses the independent review, which found the head clean with nits.
Checks at this head:
ruff check .: exit 0.compileall -q tests: exit 0.pytest -n 2over the touched and adjacent files (test_release_pipeline.py,test_workflow_hardening.py,test_release_suite_gate.py,test_distribution_surface_parity.py): exit 0.The full suite was not re-run for this commit.
Anchored
version_re. The guard's version pattern was^[A-Za-z0-9][A-Za-z0-9.!+_-]*$, which accepteduv==0.13.6.tar.gz(also.zipand.whl) and recordeduv. pip 26 reads that as a link to a local file, not an index pin. This predates the PR, but this PR's guard claims to accept only the shape uv writes.^[0-9]+(\.[0-9]+)*((a|b|rc)[0-9]+)?(\.post[0-9]+)?(\.dev[0-9]+)?$, a normalized PEP 440 public version. All 215 committed pins, across everyconstraints/*.txt, match it..tar.gz,.zip,.whl, an epoch and a local label.test_pip_reads_an_archive_suffixed_version_as_a_file_not_a_pin[.tar.gz|.zip]._pip_requirement_names) now usesinstall_req_from_parsed_requirement(...), and it reports a requirement pip reads as a link as<link: …>rather than its name. A comparison therefore fails when pip installs the right name from the wrong place.d6c3e309passeduv==0.13.6.tar.gz(OK: …, recordeduv); this head refuses it.Disclosure wording. What was wrong, The fix, Tests, Judgement calls and the CHANGELOG now describe the line-boundary and encoding cases in general terms. The tests are unchanged.
Wider workflow lint (
tests/test_workflow_hardening.py).\bsecrets\b(catchingsecrets[...]andtoJSON(secrets)) orgithub.token.containerandservicesenvironments are checked too.env:variable filled from${{ inputs.* }}(orgithub.event.inputs.*) in any harness step, and asserts those cover every declared dispatch input. It no longer names three variables.Merge order with #1163
This PR conflicts textually with #1163 in
scripts/verify_dependency_lock.py(theLockTarget.constraintsdocstring, the tail ofLOCK_TARGETS,CO_INSTALLED) and inCONTRIBUTING.md. The two designs agree, andscripts/update_locks.pyis byte-identical in both.Whichever PR merges second keeps:
CO_INSTALLEDentries;It also generalises the
LockTarget.constraintsdocstring along #1163's wording, since #1163 uses the field for a lock installed in a separate environment, not only one installed after the others.Not done / follow-ups
scripts/verify_dependency_lock.py'sparse_lockhas the same blind spot. It treats any indented line as a continuation, so an indented requirement indev.txtorrelease-seal.txtis invisible to the lock gate while pip installs it. That gate checks consistency, not choice, andrelease-seal.txthas no allowlist at all. Worth its own issue; left out to keep this PR's scope.mcp extra (floor)/(newest)to "Protect main" is still possible as belt-and-braces (Enforce the repository and publication controls the release runbook already requires #573). Nothing here depends on it.claude-agent-sdkbumps. It readsconstraints/harness.txtas a requirements file, and the package is excluded from the weekly group, so expect one individual PR per weekly run while new releases appear. If that is noise, anignoreentry in.github/dependabot.ymlwould stop it. I did not make that change.runjob. Only the mocksmokepath was exercised, locally.Closes #1033
Refs #573
Type
Authorship
Verification
CI is authoritative for
python -m ruff check .,python -m compileall -q src tests, andpython -m pytest.Additional local checks run:
scripts/generate_schemas.py --checkin the lock-exact venv.git archive origin/maintree: 20 + 13 failures, as listed above.pip check, andpython -m harness.adoption smoke, in a fresh Python 3.13 venv.scripts/update_locks.py constraints/harness.txtrun twice: identical bytes.Release-readiness notes
docs/checks.md(none)STABILITY.md(none)🤖 Generated with Claude Code