Skip to content

ci: close the release allowlist bypass, harden adoption-harness, require mcp-extra (#1033) - #1186

Open
pengfei-threemoonslab wants to merge 11 commits into
mainfrom
claude/issue-1033-ci-hardening
Open

pengfei-threemoonslab wants to merge 11 commits into
mainfrom
claude/issue-1033-ci-hardening

Conversation

@pengfei-threemoonslab

@pengfei-threemoonslab pengfei-threemoonslab commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What was wrong

Three CI and release-integrity gaps from the 2026-10-12 engineering review.

1. The release-toolchain allowlist did not read the lockfile the way pip does. .github/actions/install-release-toolchain/action.yml is the reviewed gate on what stage, publish (id-token: write) and finalize install from constraints/release-publish.txt. Its loop skipped every line that started with whitespace, as a "hash continuation". pip only joins a line onto the previous one when that line ends with a backslash. So this lockfile:

uv==0.12.9 \
    --hash=sha256:aaaa
    evilpkg==1.0 --hash=sha256:bbbb

made main's guard print OK: every locked distribution is on the allowlist. (exit 0), while pip 26.0.1's own parse_requirements reads ['uv==0.12.9', 'evilpkg==1.0'].

The same review found that the guard and pip could also disagree on where a line ends and on how the file's bytes are decoded. Both are closed here: the guard refuses any lockfile whose line boundaries or text encoding pip could read differently. The tests list the cases.

A third disagreement predates this PR. The guard recorded uv for a "version" ending in an archive suffix, such as uv==0.13.6.tar.gz, but pip reads that as a link to a local file, not an index pin. Review of this PR found it, and it is fixed here (see Review follow-up).

The test meant to catch this was hollow. test_every_publication_side_job_constrains_what_it_installs built its "allowlist" from every lowercase word before a grep -oE that the script no longer contains. On main that came to 226 words, comments included, so a lock that added build or hooks passed it. No test fed the guard an indented line.

2. adoption-harness.yml interpolated dispatch inputs into shell, in a job holding an API key. --matrix=${{ inputs.matrix_file }} --budget-usd=${{ inputs.budget_usd }} was expanded into the run: text. ANTHROPIC_API_KEY was in the run job's environment for every step. It was also the only workflow that pinned actions by tag, installed unhashed from the index (including an unpinned pip install --upgrade pip), and had a job (smoke) with no timeout.

3. mcp-extra blocked nothing. The "Protect main" ruleset requires coverage but neither mcp extra (floor) nor mcp extra (newest), and no required job needs: them. A PR that broke the [mcp] extra's server stayed mergeable.

The fix

1. Release-toolchain guard (.github/actions/install-release-toolchain/action.yml, still one self-contained bash step that the suite executes).

  • Same bytes, same lines. It refuses any lockfile whose line boundaries or text encoding pip could read differently; the step's comment lists the cases. Non-ASCII text, such as the em dashes in the committed headers, is accepted in comments only, because every other line must match an ASCII pattern.

  • Continuation state, as in pip. A requirement starts at the left margin as an exact name==version pin with an optional ; marker. The version must be a normalized PEP 440 public version: no epoch, no local label, no archive suffix. An indented line is accepted only as one --hash=sha256|384|512:<hex> token, and only on a line that follows one ending in \. Blank lines and comments never continue, and they end a continued requirement, as in pip's join_lines. Everything else is refused with its line number:

    • an indented line after a finished requirement;
    • a non-hash continuation;
    • a backslash with no space before it;
    • a file that ends inside a continuation;
    • a marker containing - (pip starts reading options at the first word beginning with -), #, \, $ or ;;
    • ===, extras, inline --hash.

    The allowlist and its error text are unchanged.

  • Why a parser and not pip install --dry-run --report. The report needs the index at guard time, so the suite cannot run it offline against crafted lockfiles, and the step's comment says that ability is the point. More importantly, to learn an sdist's metadata pip must build it, which runs the candidate's build hooks in the token-bearing job before the allowlist has decided anything. The parser runs no project or package code. It closes the differential by accepting only the one shape uv writes. A differential test then holds it to pip's own parser on every committed lock and every hostile case.

2. adoption-harness.yml.

  • Inputs reach the script only as env: variables (MATRIX_FILE, SHIPGATE_HARNESS_BUDGET_USD, AGENT_FILTER). They are expanded inside quotes into a bash array, so each one is a single argument, never shell syntax. SHIPGATE_HARNESS_BUDGET_USD feeds --budget-usd, which is the spelling docs/adoption-harness-automated.md documents.
  • ANTHROPIC_API_KEY is set on the Run harness step only.
  • Actions are pinned to the same SHAs ci.yml uses: checkout 3d3c42e5… (v7.0.1, as before), setup-python 6d4fd22b… (v7.1.0, as before), upload-artifact 043fb46d… (v7.0.1; the floating @v7 is now v7.0.2).
  • smoke gets timeout-minutes: 20; run keeps 90. Both checkouts set persist-credentials: false, because live agents run commands in that job.
  • Both jobs install constraints/dev.txt, constraints/build-backend.txt and a new constraints/harness.txt with --require-hashes, then -e . --no-deps --no-build-isolation and pip check, the same sequence as CI. python -m pip install --upgrade pip is gone.
  • The new lock. No existing lock carries claude-agent-sdk, so there is a new one. It is compiled by scripts/update_locks.py from harness/requirements.txt and registered in LOCK_TARGETS. Compiled alone, it resolved typer==0.29.1 against dev.txt's 0.27.3, so the third install would have moved the closure under the first. LockTarget therefore gains constraints. compile_command passes --constraint constraints/dev.txt --constraint constraints/build-backend.txt for this lock, and CO_INSTALLED gains a second group holding all three locks, the adoption harness's environment; the first group (dev.txt, build-backend.txt) stays exactly what CI installs, so the dependency audit's test-build subject from Audit exact pins from reviewed dependency locks #1216 is unchanged. A full regeneration compiles it last, and it is reproducible: recompiling gives the same bytes. It was compiled with uv 0.13.6, the version release-publish.txt pins. claude-agent-sdk==0.2.183, 41 distributions.

3. ci.yml. coverage (required) now has needs: [suite, mcp-extra] and a step, Require both mcp-extra jobs to have passed, that fails unless needs.mcp-extra.result == 'success'. This follows the existing suite-shard pattern. coverage already runs if: ${{ !cancelled() }}, which looks at the workflow run, not at its dependencies. So a failed, timed-out (cancelled) or skipped mcp-extra leg still starts coverage, and coverage fails by name before checkout. No ruleset change is needed. A whole-run cancellation skips coverage as before, and the cancelled suite (n) required checks still block. mcp-extra usually finishes well inside the suite shards' time, so this should add no wall-clock time.

Docs. CHANGELOG (### Packaging, three bullets), CONTRIBUTING.md § Dependabot (recompile harness.txt together with dev.txt), and the lock table in docs/release-runbook.md.

Tests

tests/test_release_pipeline.py

  • test_the_allowlist_refuses_a_requirement_it_would_not_have_seen, eight cases: two indentation shapes (one is the reported case), five line-boundary shapes and one encoding shape.
  • test_pip_installs_what_each_refused_lockfile_hides: pip's own parser reads {"uv", "evilpkg"} from each of those eight, so none of the refusals is a strawman. It skips if pip's internal parser is not importable.
  • test_the_allowlist_accepts_valid_hash_continuations: multiple hashes, sha512, indented # via blocks, a blank line, a marker, a non-ASCII comment, mixed-case names, no final newline.
  • test_the_guard_reads_every_committed_release_lock[3 locks]: what the guard records equals the lock's pins. release-publish.txt passes outright. release-seal.txt and release-build.txt (build, hatchling) get as far as the allowlist and are refused there, and only there; see Judgement calls.
  • test_the_guard_and_pip_read_the_same_requirements[3 locks]: the guard's names equal pip's for every committed release lock.
  • The refusal table gains 12 grammar cases, plus 5 version cases from the review follow-up.
  • The hollow test now reads the allowed="…" assignments (_allowlisted_names). test_the_allowlist_parse_is_the_guards_own checks that the parse cannot become "every word" again.
  • test_a_lock_installed_after_others_is_compiled_against_them covers the new lock's constraints, its compile order and its co-install grouping.
  • The coverage needs assertion is updated.

tests/test_workflow_hardening.py (new). Every workflow in .github/workflows/, plus the composite action, is checked for:

  • no ${{ inside any run:;
  • every remote uses: (step or reusable-workflow job) pinned to a 40-hex SHA;
  • no secrets (any spelling) or github.token in workflow, job, job-container or service env;
  • a positive timeout-minutes on every job that runs steps.

The 11 other workflows and the composite action already pass all four rules, so the generalisation cost nothing. Harness-specific tests check:

  • the key is held by exactly Run harness;
  • every variable a step fills from a dispatch input is expanded only inside quotes, and those variables cover every declared input;
  • every pip install is --require-hashes --requirement constraints/*.txt or the --no-deps --no-build-isolation editable install;
  • its pins equal ci.yml's.

The mcp gate test checks coverage's needs, if and step order. It also executes the gate step with success/failure/cancelled/skipped/empty, and only success exits 0. The shard weight is appended to tests/shard_seconds.json.

Proof the new tests fail on main. These ran on a git archive origin/main tree with only the test files copied in.

  • test_release_pipeline.py -k "allowlist or guard or hidden or pip_installs or publication_side": 20 failed. All 8 hidden-requirement cases failed, because main's guard printed OK:. The other 12 were grammar refusals.
  • test_workflow_hardening.py: 13 failed. These were the 4 generic rules (only for adoption-harness.yml), the 3 harness tests and the 6 mcp-gate tests. Every other workflow passed.

Harness lock, end to end. In a fresh Python 3.13 venv, the job's exact install sequence ran with --require-hashes, then pip check (clean), then python -m harness.adoption smoke, which exited 0 and imported claude_agent_sdk 0.2.183. scripts/verify_dependency_lock.py reports OK for all five locks.

Gate (local, at d6c3e309).

  • ruff check .: exit 0.
  • compileall -q src tests: exit 0.
  • scripts/verify_dependency_lock.py: exit 0.
  • pytest -n 4 -m "not perf" --ignore=tests/test_adapter_static_only.py: 23800 passed, 7 failed, 10 skipped, 120 xfailed. The machine was at load 100–330, so the run took 2 h 9 min. The 7 failures:
    • Mine, fixed and re-run: test_release_suite_gate.py::…[ci.yml-coverage], which asserted needs == "suite". It now pins ["suite", "mcp-extra"] for CI and stays strict for the release gates. The file re-runs green.
    • Stale pydantic in the main .venv: test_schema_roundtrip (3) and test_published_schema_freeze (1). These pass in the lock-exact venv (exit 0), and so does scripts/generate_schemas.py --check (exit 0).
    • Environmental: test_check_unmodelled_host_config_keys[local_settings_enabled_plugins-worktree|git_range]. The fixture's git commit exits 1 under this machine's global git-ignore. These fail identically on a git archive origin/main tree.
  • pytest tests/test_adapter_static_only.py: exit 0.
  • ./shipgate check --base origin/main --head HEAD: require_review, two medium PROTECTED-SURFACE-UNCLASSIFIED items for adoption-harness.yml and ci.yml, and one low non-widening row for the action-reference change. Nothing is blocked, so the advisory verify/verify-self jobs should not fail on it.

Compatibility

  • No CLI, schema, report, contract or check-ID change, and no distribution surface (.github is registered as CI only).
  • LockTarget gains an optional field whose default is empty.
  • The guard now refuses lockfiles it used to accept. The committed one still passes, byte for byte as compiled.

Protected paths

This PR touches protected paths and needs owner review before merge:

  • .github/actions/install-release-toolchain/action.yml: the allowlist guard that runs in stage, publish and finalize.
  • .github/workflows/adoption-harness.yml
  • .github/workflows/ci.yml: coverage needs mcp-extra.
  • constraints/harness.txt (new lock), scripts/verify_dependency_lock.py, scripts/update_locks.py: the lock gate and its generator.
  • CONTRIBUTING.md

Not touched: .github/CODEOWNERS, SECURITY.md, MAINTAINERS.md, action.yml, scripts/install_action_engine.py, .github/workflows/release-engine-smoke.yml.

Judgement calls

  • The parser is stricter than pip. It refuses tabs, inline --hash, --hash sha256:… with a space, ===, extras, epochs, local version labels, markers with -, and a trailing continuation. uv never writes any of these. A future uv format change would fail closed, with a line-numbered message.
  • Acceptance for "every committed release-*.txt passes" is met by design for release-publish.txt only. The seal and build locks are installed elsewhere. Making them pass here would require allowlisting build and hatchling in the jobs that hold the token. Whether that reading of the criterion is acceptable is for the owner to decide. The tests prove that the guard parses both of those locks completely, agrees with pip on them, and refuses them only at the allowlist.

Review follow-up (19374a09)

One new commit, no force-push. It addresses the independent review, which found the head clean with nits.

Checks at this head:

  • ruff check .: exit 0.

  • compileall -q tests: exit 0.

  • pytest -n 2 over the touched and adjacent files (test_release_pipeline.py, test_workflow_hardening.py, test_release_suite_gate.py, test_distribution_surface_parity.py): exit 0.

  • The full suite was not re-run for this commit.

  • Anchored version_re. The guard's version pattern was ^[A-Za-z0-9][A-Za-z0-9.!+_-]*$, which accepted uv==0.13.6.tar.gz (also .zip and .whl) and recorded uv. pip 26 reads that as a link to a local file, not an index pin. This predates the PR, but this PR's guard claims to accept only the shape uv writes.

    • The pattern is now ^[0-9]+(\.[0-9]+)*((a|b|rc)[0-9]+)?(\.post[0-9]+)?(\.dev[0-9]+)?$, a normalized PEP 440 public version. All 215 committed pins, across every constraints/*.txt, match it.
    • The refusal table gains .tar.gz, .zip, .whl, an epoch and a local label.
    • New test test_pip_reads_an_archive_suffixed_version_as_a_file_not_a_pin[.tar.gz|.zip].
    • The pip oracle (_pip_requirement_names) now uses install_req_from_parsed_requirement(...), and it reports a requirement pip reads as a link as <link: …> rather than its name. A comparison therefore fails when pip installs the right name from the wrong place.
    • The guard at d6c3e309 passed uv==0.13.6.tar.gz (OK: …, recorded uv); this head refuses it.
    • The reviewer's two fuzzers, run against this head's guard, found 0 bypasses and 0 guard/pip mismatches: 3000 generated cases (seed 7) and 3000 mutations (seed 11, 403 accepted).
  • Disclosure wording. What was wrong, The fix, Tests, Judgement calls and the CHANGELOG now describe the line-boundary and encoding cases in general terms. The tests are unchanged.

  • Wider workflow lint (tests/test_workflow_hardening.py).

    • The credential pattern is now \bsecrets\b (catching secrets[...] and toJSON(secrets)) or github.token.
    • Job container and services environments are checked too.
    • The quoted-input rule reads every env: variable filled from ${{ inputs.* }} (or github.event.inputs.*) in any harness step, and asserts those cover every declared dispatch input. It no longer names three variables.
    • No existing workflow fails the widened rules.

Merge order with #1163

This PR conflicts textually with #1163 in scripts/verify_dependency_lock.py (the LockTarget.constraints docstring, the tail of LOCK_TARGETS, CO_INSTALLED) and in CONTRIBUTING.md. The two designs agree, and scripts/update_locks.py is byte-identical in both.

Whichever PR merges second keeps:

  • both lock targets;
  • both CO_INSTALLED entries;
  • both CONTRIBUTING sentences.

It also generalises the LockTarget.constraints docstring along #1163's wording, since #1163 uses the field for a lock installed in a separate environment, not only one installed after the others.

Not done / follow-ups

  • scripts/verify_dependency_lock.py's parse_lock has the same blind spot. It treats any indented line as a continuation, so an indented requirement in dev.txt or release-seal.txt is invisible to the lock gate while pip installs it. That gate checks consistency, not choice, and release-seal.txt has no allowlist at all. Worth its own issue; left out to keep this PR's scope.
  • Ruleset. Adding mcp extra (floor)/(newest) to "Protect main" is still possible as belt-and-braces (Enforce the repository and publication controls the release runbook already requires #573). Nothing here depends on it.
  • Dependabot will now propose claude-agent-sdk bumps. It reads constraints/harness.txt as a requirements file, and the package is excluded from the weekly group, so expect one individual PR per weekly run while new releases appear. If that is noise, an ignore entry in .github/dependabot.yml would stop it. I did not make that change.
  • I did not dispatch the paid run job. Only the mock smoke path was exercised, locally.

Closes #1033

Refs #573

Type

  • Check or risk-model change
  • Input adapter change
  • CLI or GitHub Action behavior (CI and release workflows only; no change to the published Action)
  • Report, schema, or SARIF output
  • Documentation only

Authorship

  • A coding agent wrote some or all of this change. Agent: Claude Code, Opus 5.5
  • A person read the whole diff before opening this pull request.

Verification

CI is authoritative for python -m ruff check ., python -m compileall -q src tests, and python -m pytest.

Additional local checks run:

  • The gate listed under Tests, including scripts/generate_schemas.py --check in the lock-exact venv.
  • The new tests on a git archive origin/main tree: 20 + 13 failures, as listed above.
  • The harness install sequence plus pip check, and python -m harness.adoption smoke, in a fresh Python 3.13 venv.
  • scripts/update_locks.py constraints/harness.txt run twice: identical bytes.

Release-readiness notes

  • No user-code import added to default scan paths
  • No network access added to default scan paths
  • New or changed check IDs are documented in docs/checks.md (none)
  • Report/schema changes are additive or documented in STABILITY.md (none)

🤖 Generated with Claude Code

…ire mcp-extra (#1033)

The release-toolchain allowlist skipped every indented line as a hash
continuation, but pip joins a line onto the previous one only after a
trailing backslash: an indented `evilpkg==1.0` below a finished
requirement passed the guard ("OK: every locked distribution is on the
allowlist.") while pip's own parser installed it. The guard also split
lines with `read`, where pip splits at CR, FF, NEL and U+2028 too and
honours a PEP 263 `coding:` declaration. It now refuses those bytes and
declarations, tracks continuation state, and accepts an indented line
only as one `--hash` token after ` \`. Tests run the step on hostile
lockfiles (each confirmed against pip's parser), on valid hash
continuations, and on every committed constraints/release-*.txt, and the
hollow allowlist comparison now reads the `allowed=` list instead of
every word in the step.

adoption-harness.yml passes dispatch inputs through quoted environment
variables, scopes ANTHROPIC_API_KEY to the Run harness step, pins actions
to ci.yml's SHAs, adds a timeout to smoke, keeps no checkout credentials,
and installs hash-locked closures: constraints/dev.txt, build-backend.txt
and a new constraints/harness.txt, compiled from harness/requirements.txt
with the first two as constraints (LockTarget.constraints) and checked as
co-installed. tests/test_workflow_hardening.py holds every workflow to
the same rules.

ci.yml: coverage, a required check, now needs mcp-extra and fails unless
both legs succeeded, so a failing, cancelled or skipped mcp-extra job
blocks a merge without a ruleset change.
…flow lint (#1033)

Review follow-up on #1186.

The release-toolchain guard's version pattern accepted a "version" ending
in an archive suffix, which pip reads as a link to a local file rather
than an index pin, while the guard recorded the distribution name. The
pattern is now a normalized PEP 440 public version; every committed pin
in constraints/ still matches. The pip oracle in the tests now uses
install_req_from_parsed_requirement and reports a link as a link, so a
file-versus-index difference fails the comparison too.

The workflow lint reads `secrets` in any spelling, checks job container
and service environments, and derives the quoted-input variables from
every dispatch input a harness step's env reads. The CHANGELOG entry
describes the line-boundary and encoding cases in general terms.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: close the release allowlist bypass, harden adoption-harness, require mcp-extra

1 participant