From 296fcf09420fff9754b292fff9d6d06cc819f2c1 Mon Sep 17 00:00:00 2001 From: Roman PASSLER Date: Sat, 7 Nov 2026 21:12:41 +0100 Subject: [PATCH] feat(team): a workflow that moves the mounted pins by pull request Dependabot's submodule updates need an organisation-wide grant for private submodules and stall every submodule of a group on one 403. A team brain now gets bump-mounted-pins instead: on weekday mornings it moves each mount's gitlink to the newest commit of its tracked branch whose CI on that branch did not fail (runs of side branches and tags do not count; a commit without runs inherits the state of the nearest older commit with runs; never backwards), commits it on chore/bump-mounted-pins, force-pushes with a lease and opens or updates the pull request, with short shas only, then dispatches ci on it. A mount without a branch line follows its default branch. A mount that cannot be resolved stays at main's pin and fails the run; nothing is carried over from the bump branch, so a pin pushed there by hand is never committed under the workflow's name. While a pull request is open and its commit moves a mount that failed (against the commit's parent, not against main now), the run pushes nothing, so a passing failure does not force-push that move away; a mount that fails for good and that the branch does not move holds nothing back. The bump branch counts as current only while its head is one commit on main that changes gitlinks and nothing else; anything pushed onto it is regenerated away. Three jobs: mounts reads .gitmodules without any token, so a brain without mounts stops there and no environment is created; resolve holds a read-only GitHub App token for the mounted repositories of the brain's owner, behind the environment mounted-pins, and installs nothing beyond git; publish alone can write. Until the app's secrets are set, resolve warns and stops instead of failing every morning. runs-on is ubuntu-latest unless the variable MOUNTED_PINS_RUNNER names a runner. The script is a template copied into the brain (.github/scripts/bump-mounted-pins.py), the one piece of km code a brain carries: the selection needs the app token, and the job holding it must not install the km package. km init --team writes both files always (a fresh brain has no mounts yet; the workflow is inert then), and km doctor notes a missing environment, or one open to every branch, where the brain has mounts and the workflow. km upgrade adds both files when missing, and replaces a copy whose hash, taken with its action pins masked (the brain's Dependabot moves those), is one of the versions km shipped (SHIPPED in upgrade.py, which keeps every earlier hash; a test checks the current templates are listed), so later fixes reach the brain, keeping an action pin whose version the old copy's Dependabot moved past the template's; a copy the brain edited is left with a note. Adding the workflow to a brain that already has mounts says to restrict the environment before the first run, since GitHub would otherwise create it open to every branch. Brains carry no km code otherwise; CLAUDE.md names this template as the exception and why. --- .github/dependabot.yml | 4 +- .github/workflows/test.yml | 1 + CLAUDE.md | 1 + README.md | 6 +- skills/km/reference/brains.md | 4 +- src/km/doctor.py | 36 +- src/km/init.py | 6 +- src/km/sync.py | 2 +- src/km/templates/team/bump-mounted-pins.py | 347 ++++++++++++++++++++ src/km/templates/team/bump-mounted-pins.yml | 257 +++++++++++++++ src/km/upgrade.py | 63 +++- tests/doctor_smoke.sh | 14 + tests/mounted_pins_check.py | 308 +++++++++++++++++ tests/mounted_pins_smoke.sh | 109 ++++++ tests/team_smoke.sh | 55 +++- 15 files changed, 1192 insertions(+), 21 deletions(-) create mode 100644 src/km/templates/team/bump-mounted-pins.py create mode 100644 src/km/templates/team/bump-mounted-pins.yml create mode 100644 tests/mounted_pins_check.py create mode 100644 tests/mounted_pins_smoke.sh diff --git a/.github/dependabot.yml b/.github/dependabot.yml index fd0ece4..ee791c1 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -1,6 +1,8 @@ # Keeps the commit-pinned actions in this repository's workflows and in action.yml current. The # templates brains get are not watched: a test fails until they pin the same commits, so a -# Dependabot update here has to be carried into src/km/templates/team/*.yml by hand. +# Dependabot update here has to be carried into src/km/templates/team/*.yml by hand. An action only a +# template uses (actions/create-github-app-token, TEMPLATE_ONLY in tests/team_smoke.sh) is watched by +# no one here: each brain's Dependabot moves it there; bump it in the template by hand. version: 2 updates: - package-ecosystem: github-actions diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 6af7989..438d333 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -32,6 +32,7 @@ jobs: - run: bash tests/list_smoke.sh - run: bash tests/skill_smoke.sh - run: bash tests/hook_smoke.sh + - run: bash tests/mounted_pins_smoke.sh - name: team_smoke.sh, with a Latin-1 locale for its file-name encoding test run: sudo localedef -i en_US -f ISO-8859-1 en_US.ISO-8859-1 && bash tests/team_smoke.sh diff --git a/CLAUDE.md b/CLAUDE.md index fe9dba5..6027812 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,6 +14,7 @@ brains. See README.md for the layout. - Brains pin a km version and never carry km code, but their `schema.local.yaml`, docs and templates from earlier versions stay: keep new versions working on them, and keep `km upgrade` able to move an older brain forward. +- The one exception is the team template `bump-mounted-pins.py` (with its workflow), copied into a team brain because the job holding the app token must not install third-party code, the km package included. `km upgrade` replaces a copy whose hash is in `SHIPPED` (src/km/upgrade.py): a change to either template adds its new hash there and keeps the old ones (a test checks the current one is listed). - A release bumps `src/km/__init__.py`, `.claude-plugin/plugin.json` and the README examples together (a test checks it). Merging it to main is the release: CI tags `vX.Y.Z` once main is green. The templates pin the tag of the km that wrote them. diff --git a/README.md b/README.md index 01b760f..c709a64 100644 --- a/README.md +++ b/README.md @@ -96,7 +96,7 @@ without installing: `uvx --from git+https://github.com/pasrom/km@ km val ## What a brain gets -Brains carry no km code. `km init` writes `CONVENTIONS.md`, `CLAUDE.md`, `schema.local.yaml`, +Brains carry no km code (the one exception, the script of the mounted-pins workflow, is below). `km init` writes `CONVENTIONS.md`, `CLAUDE.md`, `schema.local.yaml`, `inbox/`, a `.gitattributes` that keeps text files LF, and a pre-commit hook pinned to a km version. `km init --team` adds domain folders with generated indexes, a README, and CI that installs km through this repository's GitHub Action: @@ -108,6 +108,8 @@ generated indexes, a README, and CI that installs km through this repository's G The team brain's ci workflow also runs on `workflow_dispatch`: a pull request that a workflow opens with `GITHUB_TOKEN` starts no `pull_request` run, so such a workflow (a pin bump, say) dispatches ci on its branch and the pull request gets its check. `km upgrade` adds the trigger to a km ci workflow that lacks it. +A team brain also gets `bump-mounted-pins`, a workflow that moves the pins of the mounted brains (submodules) by pull request, in place of Dependabot's submodule updates, which need an organisation-wide grant for private submodules and stop every submodule of a group on one refusal. On weekday mornings it walks each mount's tracked branch (its `branch =`, else the repository's default branch) back from the tip to the newest commit whose CI on that branch did not fail (runs of a side branch or a tag do not count; a commit without a run of its own, such as a bot's, inherits the state of the nearest older commit that has runs; a pin never moves backwards), sets the gitlinks without cloning, force-pushes `chore/bump-mounted-pins` with a lease and opens or updates the pull request, which lists short shas only, and dispatches ci on it. A brain without mounts stops after reading `.gitmodules`. It reads the mounts with a GitHub App (Contents and Actions: read) whose secrets `CI_APP_CLIENT_ID` and `CI_APP_PRIVATE_KEY` live in an environment `mounted-pins` restricted to `main`; the workflow's header says how to set it up, and `km doctor` notes a missing or unrestricted environment. Its script, `.github/scripts/bump-mounted-pins.py`, is the one piece of km code a brain carries: the job that holds the app token installs nothing, so it cannot run the km package. `km upgrade` adds both files when missing and replaces a copy that is still a version km shipped, unchanged but for the action pins its Dependabot moves (a pin moved past the template's version is kept), so later fixes reach the brain; a copy the brain edited is its own, left alone with a note. + The pre-commit hook works the same way (`repo: https://github.com/pasrom/km`, `rev: # frozen: v1.9.0`, `id: km-validate`). Every pin names a release by its commit, so a tag moved later cannot change what a brain runs; the version rides along as a comment, and @@ -146,7 +148,7 @@ tests/ smoke tests ## Development The smoke tests need Python 3.11+, PyYAML and git: `bash tests/gate_smoke.sh`, `index_smoke.sh`, -`team_smoke.sh`, `doctor_smoke.sh`, `sync_smoke.sh`, `list_smoke.sh`, `skill_smoke.sh` (the skill's core +`team_smoke.sh`, `doctor_smoke.sh`, `sync_smoke.sh`, `list_smoke.sh`, `mounted_pins_smoke.sh` (the mounted-pins workflow and script a team brain gets, against a stand-in GitHub API), `skill_smoke.sh` (the skill's core keeps its size limits, its reference paths resolve, every `/km` verb has a dispatch row) and `hook_smoke.sh` (the plugin's edit guard; the pre-push hook too where pre-commit is installed). CI runs them plus the GitHub Action on every pull request, and `encoding_smoke.sh`, `doctor_smoke.sh`, `sync_smoke.sh` and `hook_smoke.sh` on Windows (team_smoke runs the encoding test on macOS under non-UTF-8 locales). A release bumps diff --git a/skills/km/reference/brains.md b/skills/km/reference/brains.md index ead25ad..943cefb 100644 --- a/skills/km/reference/brains.md +++ b/skills/km/reference/brains.md @@ -24,7 +24,7 @@ checks every mount out at its pin and never moves one ahead of it. Pass its notes on: a mount with changes of its own or ahead of its pin is left as it is (`brain reset ` puts it back), never a silent skip. Newer content of a mount comes only with a moved pin, through a pull - request: the host's Dependabot PR, or the pin-bump PR below when the user asks for it + request: a team brain's bump-mounted-pins PR, or the pin-bump PR below when the user asks for it ### Shared resources (self-describing repos) @@ -48,7 +48,7 @@ mount at a newer commit (or km sync reports one ahead of its pin that they want step may move the checkout past the pin: it is the bump), `git add brains/`, commit `chore(brains): bump to `, push, `gh pr create`; then `git switch main` and `km sync`, which puts the mount back at main's pin until the PR merges. A team brain gets the -same bump from Dependabot's submodule PRs. +same bump from its bump-mounted-pins workflow (`chore/bump-mounted-pins`). ## Brain management diff --git a/src/km/doctor.py b/src/km/doctor.py index 76a431e..b8e33fa 100644 --- a/src/km/doctor.py +++ b/src/km/doctor.py @@ -371,9 +371,10 @@ def check_line_endings(root: Path) -> None: "{ nodes { nameWithOwner parent { nameWithOwner } } } } }") -def check_github(origin: str, offline: bool) -> None: +def check_github(origin: str, offline: bool) -> str | None: """On a GitHub brain, `/km pr`, a contribution and the search of open pull requests run through - gh. Without push access to origin, a contribution goes through a fork.""" + gh. Without push access to origin, a contribution goes through a fork. Returns the repository's + owner/name when gh can see it.""" repo = github_repo(origin) if not repo: return @@ -400,7 +401,7 @@ def check_github(origin: str, offline: bool) -> None: push, _, full = out.partition("\t") # full_name: the current name, after a rename or transfer if push == "true": report("ok", f"you can push branches to {full}") - return + return full _, forks = gh("api", "--hostname", "github.com", "graphql", "-f", f"query={FORKS}", "--jq", f'.data.viewer.repositories.nodes[] | select((.parent.nameWithOwner // "" | ascii_downcase) == ' f'"{full.lower()}") | .nameWithOwner') @@ -409,6 +410,32 @@ def check_github(origin: str, offline: bool) -> None: else: report("note", f"you can read {full} but not push to it, and have no fork of it yet", "a contribution creates one (`gh repo fork --remote=false`); nothing to do before then") + return full + + +def check_mounted_pins(root: Path, repo: str) -> None: + """A brain with mounts and the bump-mounted-pins workflow needs the environment that holds the + workflow's app key, restricted to main (the workflow's header says how to set it up). One call; + a note, since only a maintainer can fix it.""" + if not ((root / ".gitmodules").is_file() and (root / ".github/workflows/bump-mounted-pins.yml").is_file()): + return + env = f"repos/{repo}/environments/mounted-pins" + code, policy = gh("api", "--hostname", "github.com", env, "--jq", + '.deployment_branch_policy | if . == null then "open" elif .custom_branch_policies then "custom" else "protected" end') + fix = "Settings, Environments, mounted-pins: deployment branches, selected branches, `main` only" + if code != 0: + report("note", f"{repo} has no environment mounted-pins (or gh cannot see it): the mounted pins do not move", + "a maintainer sets it up as the top of .github/workflows/bump-mounted-pins.yml describes") + return + names = "" + if policy == "custom": # a second call only for a list of chosen branches + _, names = gh("api", "--hostname", "github.com", f"{env}/deployment-branch-policies", "--jq", "[.branch_policies[].name] | join(\" \")") + if names == "main": + report("ok", f"{repo} has the environment mounted-pins, restricted to main") + else: + what = {"open": "every branch", "protected": "every protected branch"}.get(policy, names or "no branch") + report("note", f"the environment mounted-pins of {repo} is open to {what}, not to main alone: " + f"a workflow pushed on such a branch can read the app key", fix) def main() -> int: @@ -428,7 +455,8 @@ def main() -> int: check_submodules(root, origin, args.offline) check_hook(root) check_line_endings(root) - check_github(origin, args.offline) + if repo := check_github(origin, args.offline): + check_mounted_pins(root, repo) check_km_version(root) check_plugins(root) check_uv() diff --git a/src/km/init.py b/src/km/init.py index 56bd1d3..22cf64c 100644 --- a/src/km/init.py +++ b/src/km/init.py @@ -8,8 +8,8 @@ km init DIR --team --name NAME --desc TEXT --initials XX --folder DIR=PURPOSE [--folder ...] [--rule TEXT ...] [--profile SLUG] - A team brain: also the root and folder indexes, README, .gitignore, the two workflows and a - Dependabot config. Refuses to run if any file it would write already exists. + A team brain: also the root and folder indexes, README, .gitignore, its workflows (ci, + staleness, main-guard, bump-mounted-pins with its script) and a Dependabot config. Refuses to run if any file it would write already exists. Both render and check everything before writing anything, and refuse a release whose commit cannot be looked up (not released yet, or no network). Files only: git add, the first @@ -47,6 +47,8 @@ "team/staleness.yml": ".github/workflows/staleness.yml", "team/main-guard.yml": ".github/workflows/main-guard.yml", "team/dependabot.yml": ".github/dependabot.yml", + "team/bump-mounted-pins.yml": ".github/workflows/bump-mounted-pins.yml", + "team/bump-mounted-pins.py": ".github/scripts/bump-mounted-pins.py", "pre-commit-config.template.yaml": ".pre-commit-config.yaml", } PLACEHOLDERS = ("", "", "", "", "", "", diff --git a/src/km/sync.py b/src/km/sync.py index 3a36e09..348a3c6 100644 --- a/src/km/sync.py +++ b/src/km/sync.py @@ -15,7 +15,7 @@ - makes the mounted brains match what the brain pins: `git submodule update --init` for a mount that is missing or behind its pin; one with changes of its own, a branch checked out, or ahead of or diverged from its pin is left alone and reported. A pin never moves here; that is the - host's automation (Dependabot) and a reviewed pull request. + host's automation (the bump-mounted-pins workflow) and a reviewed pull request. A fork's parent that gh found is added once as the remote `upstream` (as `gh repo fork` does), so the next run needs no gh and the fetched branch has a ref of its own. diff --git a/src/km/templates/team/bump-mounted-pins.py b/src/km/templates/team/bump-mounted-pins.py new file mode 100644 index 0000000..b7abae5 --- /dev/null +++ b/src/km/templates/team/bump-mounted-pins.py @@ -0,0 +1,347 @@ +#!/usr/bin/env python3 +"""Move the gitlinks of the mounted repositories to the newest commit of their branch. + +Written by `km init --team` for the bump-mounted-pins workflow. `km upgrade` replaces it while it is +a version km shipped, unchanged; once this repo edits it, it is this repo's and upgrade leaves it. Standard library only, so the job holding the app token installs nothing. + + python3 .github/scripts/bump-mounted-pins.py # show what would change + python3 .github/scripts/bump-mounted-pins.py --write # move the gitlinks in the index + python3 .github/scripts/bump-mounted-pins.py --list-repos OWNER # mounted repositories of OWNER + python3 .github/scripts/bump-mounted-pins.py --branch-prs BRANCH SHA # pull requests of the branch + BODY=... python3 .github/scripts/bump-mounted-pins.py --open-pr BRANCH + + --emit-message FILE write a commit message for the change (only when something moves) + --emit-body FILE write a pull-request body for it (only when something moves) + --hold-if-moved REF write nothing when a submodule that fails is one REF's commit moves (the + open pull request's branch: its move would be lost over a passing failure) + +Every submodule in .gitmodules is read at run time: its path, its repository (a GitHub URL, or a +URL relative to this repository's own, `../name.git` or `../../owner/name.git`) and its tracked +branch (`branch =`, `.` for main, else the mounted repository's default branch). Nothing is cloned; without a .gitmodules there is nothing to do. The new commit is set in the index with +`git update-index --add --cacheinfo 160000,,`, so the working tree is not touched. + +Which commit: walking back from the branch tip, the newest commit whose CI did not fail. Per +workflow the newest push run of that commit counts; a failed or still running one refuses the +commit. A commit with no push run at all (a workflow that commits with GITHUB_TOKEN starts no run) +inherits the state of the nearest older commit that has runs, or of the current pin, and is taken +only if that is green and no newer commit was refused. A pin never moves to a commit that is not +ahead of it. + +Only the submodule name and the old and new short sha are reported: not every reader of this +repository reads the mounted ones, so no commit message, workflow name or compare link goes into +the log, the commit message or the pull request. + +Access: resolving reads the mounted repositories with GH_PIN_TOKEN (read-only), or with the gh +CLI's token when it is unset. --branch-prs reads and --open-pr writes this repository with GH_TOKEN. +""" + +from __future__ import annotations + +import argparse +import functools +import json +import os +import pathlib +import re +import http.client +import subprocess +import sys +import urllib.error +import urllib.parse +import urllib.request + +# One page of the commit list. Long enough that a pin left behind by a pull request open for weeks +# is still in it, so a commit without runs can inherit from it. +CANDIDATES = 100 +SUBJECT = "chore(brains): move the mounted pins to the newest commits" + + +class ApiError(Exception): + pass + + +def git(*args: str) -> str: + return subprocess.run(["git", *args], check=True, capture_output=True, text=True).stdout.strip() + + +def api(path: str, token: str, method: str = "GET", payload: dict | None = None): + """Call the GitHub REST API and return the decoded JSON (None for an empty answer).""" + request = urllib.request.Request( + f"https://api.github.com/{path}", method=method, + data=json.dumps(payload).encode() if payload else None, + headers={ + "Authorization": f"Bearer {token}", + "Accept": "application/vnd.github+json", + "X-GitHub-Api-Version": "2022-11-28", + "Content-Type": "application/json", + "User-Agent": "bump-mounted-pins", + }) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return json.loads(response.read() or b"null") + except urllib.error.HTTPError as error: + # 401, 403 and 404 are an expired token, a missing grant and a renamed repository: say + # which, rather than reporting every failure the same way. + raise ApiError(f"HTTP {error.code}", error.read().decode(errors="replace")) from None + except (OSError, ValueError, http.client.HTTPException) as error: + # no connection, a timeout, a dropped read, an answer that is not JSON: this mount fails + raise ApiError(type(error).__name__) from None + + +def submodules() -> list[dict]: + """name, path, slug and branch of every submodule in .gitmodules, in file order.""" + if not pathlib.Path(".gitmodules").is_file(): + return [] + found: dict[str, dict] = {} + owner = os.environ.get("GITHUB_REPOSITORY", "").partition("/")[0] + # exit 1: no such key, as in a .gitmodules left empty by removing the last mount + listed = subprocess.run(["git", "config", "-f", ".gitmodules", "--get-regexp", r"^submodule\..*\.(path|url|branch)$"], + capture_output=True, text=True) + if listed.returncode not in (0, 1): + raise subprocess.CalledProcessError(listed.returncode, listed.args, listed.stdout, listed.stderr) + for line in listed.stdout.splitlines(): + key, value = line.split(" ", 1) + name, field = key[len("submodule."):].rsplit(".", 1) + found.setdefault(name, {"name": name, "branch": None})[field] = value + for entry in found.values(): + if entry["branch"] == ".": # "the superproject's branch": main here + entry["branch"] = "main" + url = entry.get("url", "") + m = re.search(r"github\.com[/:]([^/]+/[^/]+?)(?:\.git)?/?$", url) + # ../name: a sibling of this repository, under the same owner; ../../owner/name: another owner + sibling = re.fullmatch(r"\.\./([^/]+?)(?:\.git)?/?", url) + other = re.fullmatch(r"\.\./\.\./([^/]+/[^/]+?)(?:\.git)?/?", url) + entry["slug"] = (m.group(1) if m else other.group(1) if other + else f"{owner}/{sibling.group(1)}" if sibling and owner else None) + return list(found.values()) + + +def q(value: str) -> str: + """A branch name as one query value.""" + return urllib.parse.quote(value, safe="") + + +def ci_state(slug: str, sha: str, branch: str, token: str) -> str: + """"green", "red" (the newest push run of a workflow failed or is still running) or "none". + Only runs of a push to `branch` count: a run on a side branch or a tag is not CI of this branch.""" + runs = api(f"repos/{slug}/actions/runs?head_sha={sha}&branch={q(branch)}&event=push&per_page=50", token)["workflow_runs"] + newest: dict[int, dict] = {} + for run in runs: # newest first, so the first run per workflow counts + newest.setdefault(run["workflow_id"], run) + if not newest: + return "none" + return "red" if any(r["conclusion"] not in ("success", "skipped", "neutral") for r in newest.values()) else "green" + + +def newest_commit(slug: str, branch: str | None, current: str | None, token: str) -> tuple[str | None, str]: + """(sha, why) of the newest pinnable commit on `branch` (None: the default branch), sha None + when the pin is up to date; raise ApiError when there is none.""" + commits = api(f"repos/{slug}/commits?{f'sha={q(branch)}&' if branch else ''}per_page={CANDIDATES}", token) + # The default branch's name, for the runs, only when a run is looked up. + tracked = (lambda: branch) if branch else functools.cache(lambda: api(f"repos/{slug}", token)["default_branch"]) + label = branch or "the default branch" + if not commits: + raise ApiError(f"no commits on {label}") + tip = commits[0]["sha"] + shas = [c["sha"] for c in commits] + pin_in_window = current in shas + if pin_in_window: + shas = shas[:shas.index(current)] + if not shas: + return None, "up to date" + # Below the window: the current pin was accepted when it was set; else looked up when needed. + below = "green" if pin_in_window else None + state = functools.cache(lambda sha: ci_state(slug, sha, tracked(), token)) + no_workflows = functools.cache( + lambda: api(f"repos/{slug}/actions/workflows?per_page=1", token)["total_count"] == 0) + + def inherited(i: int) -> str: + """A commit without a run of its own inherits the state of the nearest older commit that has + one: a bot commit on top of a failed push sits on a tree CI never passed.""" + found = next((s for s in map(state, shas[i + 1:]) if s != "none"), None) + if found or below: + return found or below + # The window ends before the pin. Without an older run to inherit from, only a repository + # with no workflow at all counts as green. + return "green" if no_workflows() else "none" + + # Nor is it taken below a refused commit: there sit the inner commits of the same push, which CI + # never built on their own. + refused = False + for i, sha in enumerate(shas): + own = state(sha) + if own == "red" or own == "none" and (refused or inherited(i) != "green"): + refused = True + continue + # The list is in date order, not ancestry order: a commit from a merged side branch can be + # older than the pin. Skip it rather than move back. + if current and api(f"repos/{slug}/compare/{current}...{sha}", token)["status"] != "ahead": + continue + return sha, "tip" if sha == tip else f"behind the tip {tip[:8]}, which was refused" + if pin_in_window: + return None, "no newer commit can be pinned" + raise ApiError(f"no usable commit among the last {len(commits)} on {label}") + + +def gitlink(ref: str, path: str) -> str | None: + """The commit `ref` pins at `path`, None when it pins none there.""" + entry = git("ls-tree", ref, "--", path).split() + return entry[2] if entry and entry[0] == "160000" else None + + +def moves(ref: str, path: str) -> bool: + """Does `ref`'s commit change the gitlink at `path`? Compared with its parent, the main it was + built on, not with main now, which may have moved the mount since. True when that is unknown.""" + try: + return gitlink(ref, path) != gitlink(f"{ref}^", path) + except subprocess.CalledProcessError: + return True + + +def resolve(args, token: str) -> int: + report: list[str] = [] + moved: list[str] = [] + problems: list[str] = [] + held: list[str] = [] # failed mounts the open pull request moves + for sub in submodules(): + name, path = sub["name"], sub.get("path") + current = None + try: + if not path or not sub["slug"]: + raise ApiError("no path or no GitHub url in .gitmodules") + current = gitlink("HEAD", path) + sha, why = newest_commit(sub["slug"], sub["branch"], current, token) + except (ApiError, KeyError, TypeError, subprocess.CalledProcessError) as error: + # One mount that cannot be resolved, whatever the reason, does not hold back the others. + why = error.args[0] if isinstance(error, ApiError) else f"{type(error).__name__}: {error}" + problems.append(f"{name}: {why}") + report.append(f"{name}: PROBLEM, {why}") + # It stays at main's pin. Nothing is carried over from the pull request branch: anyone + # who can push there could plant a pin the workflow would then commit as its own. + if path and args.hold_if_moved and moves(args.hold_if_moved, path): + held.append(name) + continue + was = current[:8] if current else "unpinned" + if sha is None: + report.append(f"{name}: {was} {why}") + continue + report.append(f"{name}: {was} -> {sha[:8]} ({why})") + moved.append(f"| {name} | `{was}` | `{sha[:8]}` |") + if args.write: + git("update-index", "--add", "--cacheinfo", f"160000,{sha},{path}") # --add: a mount not pinned yet + print("\n".join(report)) + if held: + # Pushing now would drop their proposed move over what may be a passing failure, and the + # next run would put it back: nothing is written, the pull request stays as it is. + print(f"The open pull request moves {', '.join(held)}, which could not be resolved: it is left as it is.") + moved = [] + + if moved and args.emit_message: + pathlib.Path(args.emit_message).write_text( + SUBJECT + "\n\n" + "\n".join(report) + "\n", + encoding="utf-8") + if moved and args.emit_body: + pathlib.Path(args.emit_body).write_text( + "Moves the mounted repositories to the newest commit of their tracked branch whose CI did not fail. Generated by the bump-mounted-pins workflow; the branch is regenerated on each run.\n\n" + "| Submodule | From | To |\n|---|---|---|\n" + "\n".join(moved) + "\n\n" + + ("Not moved:\n\n" + "\n".join(f"- {p}" for p in problems) + "\n\n" if problems else "") + + "Only the short shas are listed on purpose: not every reader of this repository reads the mounted ones.\n", + encoding="utf-8") + if problems: + print(f"{len(problems)} submodule(s) could not be resolved.", file=sys.stderr) + return 1 if problems else 0 + + +def list_repos(owner: str) -> int: + """Print the names of the mounted repositories of `owner`, comma-separated.""" + print(",".join(dict.fromkeys(sub["slug"].split("/")[1] for sub in submodules() + if sub["slug"] and sub["slug"].split("/")[0].lower() == owner.lower()))) + return 0 + + +def branch_prs(repo: str, branch: str, token: str, state: str) -> list[dict]: + return api(f"repos/{repo}/pulls?head={q(repo.split('/')[0] + ':' + branch)}&state={state}&per_page=100", token) + + +def report_branch_prs(branch: str, sha: str, token: str) -> int: + """Print two words, true or false: is a pull request of the branch open, and has one, open or + closed, `sha` as its head while `sha` is what this workflow pushes: one commit on main that + changes the submodule gitlinks and nothing else. A branch someone pushed more onto is not taken + as current, so the next push regenerates it.""" + repo = os.environ["GITHUB_REPOSITORY"] + prs = branch_prs(repo, branch, token, "all") + is_open = any(p["state"] == "open" for p in prs) + seen = any(p["head"]["sha"] == sha for p in prs) + if seen: + diff = api(f"repos/{repo}/compare/main...{sha}", token) + paths = {sub.get("path") for sub in submodules()} + files = diff.get("files") or [] # none listed: not known to be gitlinks only + seen = diff["ahead_by"] == 1 and bool(files) and all(f["filename"] in paths for f in files) + print(str(is_open).lower(), str(seen).lower()) + return 0 + + +def open_pr(args, token: str) -> int: + """Open or update the pull request, dispatch ci.yml.""" + repo = os.environ["GITHUB_REPOSITORY"] + body = os.environ["BODY"] + open_prs = branch_prs(repo, args.open_pr, token, "open") + if open_prs: + # The branch moved, so the description moves with it; otherwise it keeps describing the + # first day's pins. + number = open_prs[0]["number"] + api(f"repos/{repo}/pulls/{number}", token, "PATCH", {"body": body}) + print(f"Pull request #{number} updated.") + else: + created = api(f"repos/{repo}/pulls", token, "POST", { + "title": SUBJECT, + "head": args.open_pr, "base": "main", "body": body}) + print("opened", created["html_url"]) + + # workflow_dispatch is the one trigger GITHUB_TOKEN may start, and a dispatched run on the branch + # shows up on the pull request. km's ci.yml has the trigger; a ci.yml without it gets a 422, and + # the pull request then shows no check. + try: + api(f"repos/{repo}/actions/workflows/ci.yml/dispatches", token, "POST", {"ref": args.open_pr}) + print("ci.yml dispatched on", args.open_pr) + except ApiError as error: + if "workflow_dispatch" not in (error.args[1:] or [""])[0]: + raise + print("::notice::ci.yml has no workflow_dispatch trigger, so it does not run on the pull request") + return 0 + + +def main() -> int: + ap = argparse.ArgumentParser(description="Move the mounted repositories' gitlinks forward.") + ap.add_argument("--write", action="store_true", help="move the gitlinks in the index") + ap.add_argument("--emit-message", metavar="FILE", help="write a commit message here") + ap.add_argument("--emit-body", metavar="FILE", help="write a pull-request body here") + ap.add_argument("--hold-if-moved", metavar="REF", + help="write nothing when a submodule that could not be resolved is one REF's head commit moves") + ap.add_argument("--open-pr", metavar="BRANCH", + help="open or update the pull request for BRANCH, with the body in $BODY") + ap.add_argument("--list-repos", metavar="OWNER", help="print OWNER's mounted repositories") + ap.add_argument("--branch-prs", nargs=2, metavar=("BRANCH", "SHA"), + help="print whether BRANCH has an open pull request and one with head SHA") + args = ap.parse_args() + try: + if args.open_pr: + return open_pr(args, os.environ["GH_TOKEN"]) + if args.list_repos: + return list_repos(args.list_repos) + if args.branch_prs: + return report_branch_prs(*args.branch_prs, os.environ["GH_TOKEN"]) + token = os.environ.get("GH_PIN_TOKEN") or git_token() + return resolve(args, token) + except ApiError as error: + print(f"GitHub API: {' '.join(error.args)}", file=sys.stderr) + return 1 + + +def git_token() -> str: + """The gh CLI's token, for a run on a workstation.""" + return subprocess.run(["gh", "auth", "token"], check=True, capture_output=True, text=True).stdout.strip() + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/km/templates/team/bump-mounted-pins.yml b/src/km/templates/team/bump-mounted-pins.yml new file mode 100644 index 0000000..cfb520e --- /dev/null +++ b/src/km/templates/team/bump-mounted-pins.yml @@ -0,0 +1,257 @@ +# Written by `km init --team`. `km upgrade` adds it when missing and replaces it while it is a +# version km shipped, unchanged but for its action pins; once this repo edits it, it is this repo's +# and upgrade leaves it. Actions and the image are pinned by commit and digest; Dependabot moves the +# actions, the image digest is moved by hand. +name: bump-mounted-pins + +# The repositories mounted as submodules (.gitmodules) are pinned to a commit, so a change in one of +# them reaches this brain only when the pin moves. That keeps what the brain reads reproducible and +# every move a reviewed change. This workflow connects the two events: it looks for newer commits +# and opens a pull request, so the move stays reviewed instead of becoming automatic. +# +# Three jobs: mounts reads .gitmodules (a brain without mounts stops there), resolve holds the read +# token for the mounted repositories, publish alone can write here. +# +# Set up once, before the first mount (until then the workflow does nothing): +# 1. A GitHub App with the repository permissions Contents: read and Actions: read, installed on +# the owner of this brain for the mounted repositories (or all). Generate a private key. +# 2. Settings, Environments, New environment `mounted-pins`; Deployment branches: selected +# branches, `main` only. GitHub creates an environment a workflow names on its first run, +# without that restriction, so create it first. +# 3. In that environment, the secrets CI_APP_CLIENT_ID (the app's client ID) and +# CI_APP_PRIVATE_KEY (the private key). +# 4. Settings, Actions, General: "Allow GitHub Actions to create and approve pull requests". +# 5. In the ruleset or branch protection of main: "Require approval of the most recent reviewable +# push", so a commit someone pushes onto the bump branch needs a review by someone else. +# Mounted repositories of another owner are not in the token's scope: next to a mount of this +# brain's owner, a public one still resolves and a private one is reported and fails the run; a +# brain whose mounts all belong to other owners is not handled (mounts finds none and stops). +# Submodule paths must not contain spaces. The runner is ubuntu-latest unless the repository +# variable MOUNTED_PINS_RUNNER names another label. +on: + schedule: + # Weekday mornings, 06:25 UTC. Off the hour on purpose: scheduled runs cluster there and are + # queued longest. + - cron: '25 6 * * 1-5' + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: bump-mounted-pins + cancel-in-progress: false + +env: + BRANCH: chore/bump-mounted-pins + +# In a container job the default shell is sh; the steps use bash arrays and process substitution. +defaults: + run: + shell: bash + +jobs: + mounts: + runs-on: ${{ vars.MOUNTED_PINS_RUNNER || 'ubuntu-latest' }} + container: python:3.12-slim@sha256:4840d514c583a4d3d8cad3940e68dee6f0f6f3635c12fcd0a3d74c21a5b93a54 # 3.12-slim + outputs: + names: ${{ steps.mounted.outputs.names }} + steps: + - name: Install git + run: apt-get update && apt-get install -y --no-install-recommends git + + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + # actions/checkout registers the workspace as a safe directory in a HOME that exists only for + # its own step; later steps run with the real HOME and git then refuses the repository as + # dubiously owned. + - name: Trust the workspace + run: git config --global --add safe.directory "$PWD" + + # The token resolve gets reads only these repositories: the mounted ones of this brain's + # owner. None: no environment, no token, nothing to do. + - name: List the mounted repositories + id: mounted + env: + OWNER: ${{ github.repository_owner }} + run: | + names=$(python3 .github/scripts/bump-mounted-pins.py --list-repos "$OWNER") + [ -n "$names" ] || echo "No repository of $OWNER is mounted (.gitmodules): nothing to move." + echo "names=$names" >> "$GITHUB_OUTPUT" + + resolve: + needs: mounts + if: needs.mounts.outputs.names != '' + runs-on: ${{ vars.MOUNTED_PINS_RUNNER || 'ubuntu-latest' }} + # The app key lives in this environment, which only main may use. A workflow pushed on another + # branch therefore cannot reach the key. + environment: mounted-pins + container: python:3.12-slim@sha256:4840d514c583a4d3d8cad3940e68dee6f0f6f3635c12fcd0a3d74c21a5b93a54 # 3.12-slim + permissions: + contents: read + pull-requests: read # is a pull request of the bump branch open, or was one opened for its head + outputs: + changed: ${{ steps.bump.outputs.changed }} + proposed: ${{ steps.bump.outputs.proposed }} + pins: ${{ steps.bump.outputs.pins }} + message: ${{ steps.bump.outputs.message }} + body: ${{ steps.bump.outputs.body }} + steps: + - name: Install git + run: apt-get update && apt-get install -y --no-install-recommends git + + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Trust the workspace # as in mounts + run: git config --global --add safe.directory "$PWD" + + # Not set up yet (see the top): say so and stop, rather than fail every morning. + - name: Check the app's secrets + id: app + env: + CLIENT_ID: ${{ secrets.CI_APP_CLIENT_ID }} + KEY: ${{ secrets.CI_APP_PRIVATE_KEY }} + run: | + if [ -n "$CLIENT_ID" ] && [ -n "$KEY" ]; then echo "ready=true" >> "$GITHUB_OUTPUT" + else echo "::warning::the environment mounted-pins has no CI_APP_CLIENT_ID and CI_APP_PRIVATE_KEY: no pin moves (see the top of this workflow)" + fi + + # A short-lived installation token of the read-only app. Contents to resolve the commits, + # actions to check that their CI did not fail. The action revokes the token at the end of + # the job. + - name: Get a read token for the mounted repositories + id: app-token + if: steps.app.outputs.ready == 'true' + uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0 + with: + client-id: ${{ secrets.CI_APP_CLIENT_ID }} + private-key: ${{ secrets.CI_APP_PRIVATE_KEY }} + owner: ${{ github.repository_owner }} + repositories: ${{ needs.mounts.outputs.names }} + permission-contents: read + permission-actions: read + + - name: Look for newer commits + id: bump + if: steps.app.outputs.ready == 'true' + env: + # GITHUB_TOKEN is scoped to this repository alone and cannot see the mounted ones. + GH_PIN_TOKEN: ${{ steps.app-token.outputs.token }} + GH_TOKEN: ${{ github.token }} + run: | + # The token reaches git through the environment of these calls only, never in argv. + auth=$(printf 'x-access-token:%s' "$GH_TOKEN" | base64 -w0) + echo "::add-mask::$auth" + authgit() { + GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=http.extraheader \ + GIT_CONFIG_VALUE_0="AUTHORIZATION: basic $auth" git "$@" + } + # The pull request branch as it stands, if there is one. Exit 2 is "no such branch"; any + # other failure fails the job rather than passing for a missing branch. + rc=0 + authgit ls-remote --exit-code --heads origin "refs/heads/$BRANCH" > /dev/null || rc=$? + proposed="" open=false seen=false + if [ "$rc" = 0 ]; then + authgit fetch -q --depth 2 origin "refs/heads/$BRANCH" # with its parent, for --hold-if-moved + proposed=$(git rev-parse FETCH_HEAD) + git update-ref refs/bump/proposed "$proposed" + prs=$(python3 .github/scripts/bump-mounted-pins.py --branch-prs "$BRANCH" "$proposed") + read -r open seen <<< "$prs" + elif [ "$rc" != 2 ]; then + echo "::error::git ls-remote failed with exit $rc"; exit 1 + fi + # A submodule that cannot be resolved stays at main's pin and does not hold back the + # others; the body names it and the last step fails this job. Nothing is carried over + # from the branch, so no pin pushed there by hand is committed under this workflow's + # name. While a pull request is open and its commit moves such a submodule, the run + # writes nothing (--hold-if-moved): the move would be lost over a passing failure and + # come back the next day. One that fails for good (a private mount of another owner, + # say) and that the branch does not move holds nothing back. + hold=() + [ "$open" = true ] && hold=(--hold-if-moved refs/bump/proposed) + rc=0 + python3 .github/scripts/bump-mounted-pins.py --write "${hold[@]}" \ + --emit-message /tmp/pin-message --emit-body /tmp/pin-body || rc=$? + echo "problems=$rc" >> "$GITHUB_OUTPUT" + [ -s /tmp/pin-message ] || { echo "changed=false" >> "$GITHUB_OUTPUT"; exit 0; } + # Compared against the bump branch, not only against main: with a pull request open the + # pins differ from main every day, and committing them again would force-push each + # morning. A pull request closed without merging leaves its branch, so the same pins are + # not proposed again; newer commits are. Only when a pull request was opened for the + # branch head, though: if opening it failed, the next run tries again. And only while that + # head is what publish pushes, one commit on main changing gitlinks only (--branch-prs + # checks it): a branch someone pushed more onto is regenerated. The submodules are + # `ignore = all`, which would hide every gitlink change from a plain diff. + mapfile -t paths < <(git config -f .gitmodules --get-regexp '\.path$' | cut -d' ' -f2) + if [ "$seen" = true ] && + git diff --cached --quiet --ignore-submodules=none refs/bump/proposed -- "${paths[@]}"; then + echo "changed=false" >> "$GITHUB_OUTPUT" + echo "The pull request branch already proposes exactly these pins." + exit 0 + fi + eof="EOF_$(od -An -N8 -tx1 /dev/urandom | tr -d ' \n')" + { + echo "changed=true" + echo "proposed=$proposed" + echo "pins<<$eof"; git ls-files -s -- "${paths[@]}" | awk '{print $2, $4}'; echo "$eof" + echo "message<<$eof"; cat /tmp/pin-message; echo "$eof" + echo "body<<$eof"; cat /tmp/pin-body; echo "$eof" + } >> "$GITHUB_OUTPUT" + + - name: Fail on a submodule that could not be resolved + if: steps.app.outputs.ready == 'true' && steps.bump.outputs.problems != '0' + run: echo "::error::a submodule could not be resolved, see the step log" && exit 1 + + publish: + needs: resolve + if: ${{ !cancelled() && needs.resolve.outputs.changed == 'true' }} + runs-on: ${{ vars.MOUNTED_PINS_RUNNER || 'ubuntu-latest' }} + container: python:3.12-slim@sha256:4840d514c583a4d3d8cad3940e68dee6f0f6f3635c12fcd0a3d74c21a5b93a54 # 3.12-slim + permissions: + contents: write # push the bump branch + pull-requests: write # open or update the pull request + actions: write # dispatch ci.yml on the bump branch + steps: + - name: Install git + run: apt-get update && apt-get install -y --no-install-recommends git + + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Trust the workspace # as in mounts + run: git config --global --add safe.directory "$PWD" + + - name: Commit and push the bump branch + env: + PINS: ${{ needs.resolve.outputs.pins }} + MESSAGE: ${{ needs.resolve.outputs.message }} + PROPOSED: ${{ needs.resolve.outputs.proposed }} + GH_TOKEN: ${{ github.token }} + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git checkout -q -B "$BRANCH" + while read -r sha path; do git update-index --add --cacheinfo "160000,$sha,$path"; done <<< "$PINS" + git commit -q -m "$MESSAGE" + # Force: the branch is regenerated on every run rather than accumulating one commit per + # day, which also drops any commit pushed onto it by hand. The lease holds it to the head + # resolve saw (empty: the branch must not exist), so at least a commit pushed while this + # run is under way is not lost. The token reaches git for this push only. + auth=$(printf 'x-access-token:%s' "$GH_TOKEN" | base64 -w0) + echo "::add-mask::$auth" + authgit() { + GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=http.extraheader \ + GIT_CONFIG_VALUE_0="AUTHORIZATION: basic $auth" git "$@" + } + authgit push -q --force-with-lease="refs/heads/$BRANCH:$PROPOSED" origin "HEAD:refs/heads/$BRANCH" + + - name: Open or update the pull request + env: + GH_TOKEN: ${{ github.token }} + BODY: ${{ needs.resolve.outputs.body }} + run: python3 .github/scripts/bump-mounted-pins.py --open-pr "$BRANCH" diff --git a/src/km/upgrade.py b/src/km/upgrade.py index 79d3ded..152a51a 100644 --- a/src/km/upgrade.py +++ b/src/km/upgrade.py @@ -15,9 +15,12 @@ still runs stays until that workflow is switched to the km command; - a legacy schema.yaml becomes schema.local.yaml, the overlay it already acts as; - a team brain gets the Dependabot config, and the obsolete `team_brain:` line goes. -- A team brain gets the main-guard workflow and the pre-commit hooks that refuse a commit on main - and a push to main, whichever it lacks, and its km ci workflow gets the `workflow_dispatch` - trigger, so a workflow that opens a pull request with GITHUB_TOKEN can run the check on it. +- A team brain gets the main-guard workflow, the bump-mounted-pins workflow and its script, and the + pre-commit hooks that refuse a commit on main and a push to main, whichever it lacks, and its km + ci workflow gets the `workflow_dispatch` trigger, so a workflow that opens a pull request with + GITHUB_TOKEN can run the check on it. The bump-mounted-pins workflow and script are copies of km + code: one that is still a version km shipped (its action pins aside) is replaced by this km's, one + the brain changed is left with a note. - A brain without a .gitattributes gets km's, so text files are LF on every platform; one whose own file sets no line ending gets a note. @@ -36,7 +39,7 @@ import yaml from km import KM_REF -from km.init import ensure_lf, pin_values, template, with_team_hooks +from km.init import TEAM, ensure_lf, pin_values, template, with_team_hooks from km.paths import repo_root, write_text from km.pins import SCHEMA_LINK, hook_rev, movable_refs, read_pins, resolve, set_pins, set_schema_link, version_key, workflow_files @@ -57,6 +60,41 @@ ".github/workflows/staleness.yml": ("team/staleness.yml", {"e9768164f8a03461", "199bf98454e2f922", "fb3fcd7353e99cde"}), ".pre-commit-config.yaml": ("pre-commit-config.template.yaml", {"8ef7f1ea977e2308", "d6819c5f6e97baa1"}), } +# The km files a team brain carries as copies (its workflow's token job must not install km) -> +# shipped_hash of every version km shipped, the current one included (a test checks it): a copy that +# is still one of them is replaced by the current template, an edited one is left with a note. +SHIPPED = { + "team/bump-mounted-pins.yml": {"c72b295b1ba76e65"}, + "team/bump-mounted-pins.py": {"52f798d7ca2f7d32"}, +} +USES = re.compile(r"(?m)^([ \t]*(?:-[ \t]+)?uses:[ \t]*([^@\s]+)@)\S+.*$") # 1: up to the @, 2: the action + + +def shipped_hash(text: str) -> str: + """sha256 prefix of a shipped copy with its action refs masked: the brain's Dependabot moves + those, and a moved pin is not an edit of the brain's own.""" + return hashlib.sha256(USES.sub(r"\1", text).encode()).hexdigest()[:16] + + +def _pinned_version(pin: str) -> tuple[int, ...] | None: + """(7, 0, 1) for ` # v7.0.1`, the version comment Dependabot keeps; None without one.""" + m = re.search(r"#\s*v?(\d+(?:\.\d+)*)\b", pin) + return tuple(int(x) for x in m.group(1).split(".")) if m else None + + +def carry_pins(old: str, new: str) -> str: + """`new` with the pin `old` gives an action where its version comment is newer than km's: the + brain's Dependabot moved it past the template. Otherwise km's pin, which may be newer and which + the new template may need.""" + pins = {m.group(2): m.group(0)[len(m.group(1)):] for m in USES.finditer(old)} + + def pin(m: re.Match) -> str: + ours, theirs = m.group(0)[len(m.group(1)):], pins.get(m.group(2)) + newer = theirs and (a := _pinned_version(theirs)) and (b := _pinned_version(ours)) and a > b + return m.group(1) + (theirs if newer else ours) + return USES.sub(pin, new) + + MARKER = re.compile(r"(?m)^(# Marks a team brain:.*\n)?team_brain:.*\n") @@ -173,11 +211,22 @@ def main() -> int: write_text(local, new) done.append("dropped team_brain from schema.local.yaml (no longer used)") if team: - for tpl, rel in (("team/dependabot.yml", ".github/dependabot.yml"), ("team/main-guard.yml", ".github/workflows/main-guard.yml")): - if not (p := root / rel).exists(): + for tpl in ("team/dependabot.yml", "team/main-guard.yml", *SHIPPED): + new = template(tpl, {}) + if not (p := root / (rel := TEAM[tpl])).exists(): p.parent.mkdir(parents=True, exist_ok=True) - write_text(p, template(tpl, {})) + write_text(p, new) done.append(f"added {rel}") + if tpl == "team/bump-mounted-pins.yml" and (root / ".gitmodules").is_file(): + done.append("this brain has mounts: before the workflow's first run, create the environment mounted-pins " + "restricted to main, or GitHub creates it open to every branch (setup in the workflow's header)") + elif tpl in SHIPPED and (copy := p.read_text(encoding="utf-8")) != new and (h := shipped_hash(copy)) != shipped_hash(new): + if h in SHIPPED[tpl]: + write_text(p, carry_pins(copy, new)) + done.append(f"updated {rel} to this km's version (it was an earlier one, unchanged but for its action pins, keeping those newer than km's)") + else: + done.append(f"left {rel}: the brain changed it, so it gets no later fixes; " + f"compare it with km's src/km/templates/{tpl}") ci = str(Path(".github/workflows/ci.yml")) # as read_pins names it, with the OS's separator if ci in pins and (got := with_dispatch(text := (root / ci).read_text(encoding="utf-8"))) != text: # km's ci workflow if got is None: diff --git a/tests/doctor_smoke.sh b/tests/doctor_smoke.sh index 969af3b..a7f633c 100644 --- a/tests/doctor_smoke.sh +++ b/tests/doctor_smoke.sh @@ -201,6 +201,8 @@ case "$*" in "auth status --hostname github.com") [ -n "$GH_AUTH" ] ;; "api --hostname github.com repos/o/r --jq [.permissions.push, .full_name] | @tsv") printf '%s\tO/R\n' "$GH_PUSH" ;; "api --hostname github.com graphql -f query="*) [ "$GH_FORK" = me ] && echo me/r-1 ;; + "api --hostname github.com repos/O/R/environments/mounted-pins --jq .deployment_branch_policy"*) [ -n "$GH_ENV" ] && echo "$GH_ENV" || exit 1 ;; + "api --hostname github.com repos/O/R/environments/mounted-pins/deployment-branch-policies --jq"*) echo "$GH_BRANCHES" ;; *) exit 2 ;; esac SH @@ -211,8 +213,20 @@ for s, w, f in d.results: print(s, w, "|", f)' 2>&1)"; } GH_LOGIN= ghcheck; out_has "FAIL gh is not signed in" && ok "gh without a login fails with gh auth login" || no "gh login ($out)" GH_LOGIN= GH_AUTH=1 ghcheck; out_has "FAIL gh is signed in but github.com did not answer" && ok "a network error is not taken for a missing login" || no "gh network ($out)" GH_LOGIN=me GH_PUSH=true ghcheck; out_has "ok you can push branches to O/R" && ok "push access is ok" || no "push access ($out)" + out="$(GH_LOGIN=me GH_PUSH=false PATH="$G:$PATH" kmpy -c 'import km.doctor as d; print(d.check_github("https://github.com/o/r.git", False))' 2>&1)" + [ "$out" = O/R ] && ok "the GitHub check names the repository for the checks that follow, also without push access" || no "check_github returns the repo ($out)" GH_LOGIN=me GH_PUSH=false GH_FORK=me ghcheck; out_has "ok you contribute through your fork me/r-1" && ok "a fork is found, also under another name" || no "fork found ($out)" GH_LOGIN=me GH_PUSH=false GH_FORK=other ghcheck; out_has "note you can read O/R but not push to it" && ok "no fork yet is a note" || no "no fork ($out)" + E="$T/envbrain"; mkdir -p "$E/.github/workflows" && touch "$E/.gitmodules" "$E/.github/workflows/bump-mounted-pins.yml" + envcheck(){ out="$(PATH="$G:$PATH" kmpy -c 'import sys, pathlib, km.doctor as d +d.check_mounted_pins(pathlib.Path(sys.argv[1]), "O/R") +for s, w, f in d.results: print(s, w, "|", f)' "$1" 2>&1)"; } + GH_ENV= envcheck "$E"; out_has "note O/R has no environment mounted-pins" && ok "a brain with mounts and the pins workflow but no environment gets a note" || no "missing environment ($out)" + GH_ENV=open envcheck "$E"; out_has "note the environment mounted-pins of O/R is open to every branch" && ok "an environment open to every branch gets a note" || no "open environment ($out)" + GH_ENV=protected envcheck "$E"; out_has "open to every protected branch, not to main alone" && ok "an environment open to protected branches gets a note" || no "protected environment ($out)" + GH_ENV=custom GH_BRANCHES="main release" envcheck "$E"; out_has "open to main release, not to main alone" && ok "an environment open to main and another branch gets a note" || no "custom environment ($out)" + GH_ENV=custom GH_BRANCHES=main envcheck "$E"; out_has "ok O/R has the environment mounted-pins, restricted to main" && ok "an environment restricted to main is ok" || no "restricted environment ($out)" + rm "$E/.gitmodules"; GH_ENV= envcheck "$E"; ! out_has "mounted-pins" && ok "without mounts the environment is not asked for" || no "no mounts, no check ($out)" out="$(PATH="$G:$PATH" kmpy -c 'import km.doctor as d d.check_github("git@github.com:o/r.git", True); d.check_github("https://gitlab.com/o/r.git", False) print(d.results)' 2>&1)" diff --git a/tests/mounted_pins_check.py b/tests/mounted_pins_check.py new file mode 100644 index 0000000..7e9ec14 --- /dev/null +++ b/tests/mounted_pins_check.py @@ -0,0 +1,308 @@ +"""The commit selection and the resolve step of the bump-mounted-pins script a team brain gets +(src/km/templates/team/bump-mounted-pins.py), against a stand-in for the GitHub API. + + python3 tests/mounted_pins_check.py +""" +from __future__ import annotations + +import importlib.util +import os +import re +import subprocess +import sys +import tempfile +import urllib.parse +from pathlib import Path +from types import SimpleNamespace + +REPO = Path(__file__).resolve().parent.parent +spec = importlib.util.spec_from_file_location("bump", REPO / "src/km/templates/team/bump-mounted-pins.py") +bump = importlib.util.module_from_spec(spec) +spec.loader.exec_module(bump) +REAL_API = bump.api # the cases below replace bump.api with stand-ins + +passed = failed = 0 + + +def check(cond: bool, what: str, detail: object = "") -> None: + global passed, failed + if cond: + passed += 1 + print(f" ok: {what}") + else: + failed += 1 + print(f" FAIL: {what} ({detail})") + + +def sha(name: str) -> str: + """A 40-hex sha that shows its name: c1 -> c1c1c1...""" + return (name.encode().hex() * 40)[:40] + + +class Fake: + """One mounted repository: `commits` newest first, each `name` or `name:state`, where state is a + comma list of conclusions, one per workflow, newest run first per workflow written `wf=a/b` + (a the newest), each a push to `main` unless written `conclusion@ref`; no state: no run. + `behind`: commits that are not ahead of the pin. `default`: the repository's default branch.""" + + def __init__(self, commits: list[str], workflows: int = 1, behind: tuple[str, ...] = (), down: bool = False, + default: str = "main"): + self.order, self.runs = [], {} + for c in commits: + name, _, state = c.partition(":") + self.order.append(sha(name)) + runs = [] + for i, part in enumerate(filter(None, state.split(","))): + wf, _, conc = part.rpartition("=") + for x in conc.split("/"): + x, _, ref = x.partition("@") + runs.append({"workflow_id": wf or i, "conclusion": None if x == "running" else x, "head_branch": ref or "main"}) + self.runs[sha(name)] = runs + self.workflows, self.behind, self.down = workflows, {sha(b) for b in behind}, down + self.default = default + self.branches: list[str | None] = [] # the branch each commit listing asked for, as sent + + def api(self, path: str, token: str, method: str = "GET", payload: dict | None = None): + if self.down: + raise bump.ApiError("HTTP 401", "Bad credentials") + if path == "repos/o/r": + return {"default_branch": self.default} + if m := re.match(r"repos/o/r/commits\?(?:sha=([^&]+)&)?per_page=\d+$", path): + self.branches.append(m.group(1)) # None: the default branch + return [{"sha": s} for s in self.order] + if m := re.match(r"repos/o/r/actions/runs\?head_sha=(\w+)(?:&branch=([^&]+))?&event=push", path): + branch = m.group(2) and urllib.parse.unquote(m.group(2)) # no filter: runs of every ref + return {"workflow_runs": [r for r in self.runs.get(m.group(1), []) if branch in (None, r["head_branch"])]} + if path == "repos/o/r/actions/workflows?per_page=1": + return {"total_count": self.workflows} + if m := re.match(r"repos/o/r/compare/(\w+)\.\.\.(\w+)$", path): + return {"status": "behind" if m.group(2) in self.behind else "ahead"} + raise AssertionError(f"unexpected call {path}") + + +def pick(commits: list[str], pin: str | None, **kw) -> str: + """The name of the commit chosen, 'none' when the pin stays, or 'error: ...'.""" + fake = Fake(commits, **kw) + bump.api = fake.api + try: + got, _ = bump.newest_commit("o/r", "main", sha(pin) if pin else None, "t") + except bump.ApiError as e: + return f"error: {e.args[0]}" + names = {sha(c.partition(":")[0]): c.partition(":")[0] for c in commits} + return names.get(got, got) if got else "none" + + +CASES = [ # (what, commits newest first, pin, keyword args, expected) + ("a green tip is taken", ["c2:success", "c1:success", "p:success"], "p", {}, "c2"), + ("a bot tip on a green commit is taken", ["bot", "c1:success", "p:success"], "p", {}, "bot"), + ("a bot tip directly on the pin is taken", ["bot", "p"], "p", {}, "bot"), + ("a bot tip on a failed commit is refused, the green one below the failure is taken", + ["bot", "c2:failure", "c1:success", "p"], "p", {}, "c1"), + ("two bot commits on a failed commit are refused", ["bot2", "bot1", "c1:failure", "p"], "p", {}, "none"), + ("an inner commit below a refused head stays refused", ["c2:failure", "inner", "p"], "p", {}, "none"), + ("below a refused head and its inner commit, an older green commit is taken", + ["c3:failure", "inner", "c1:success", "p"], "p", {}, "c1"), + ("a candidate that is not ahead of the pin is skipped, not moved back", + ["c2:success", "side:success", "p"], "p", {"behind": ("c2",)}, "side"), + ("the newest run of a workflow counts: a success after a failure", ["c1:success/failure", "p"], "p", {}, "c1"), + ("the newest run of a workflow counts: a failure after a success", ["c1:failure/success", "p"], "p", {}, "none"), + ("a cancelled run refuses", ["c1:cancelled", "p"], "p", {}, "none"), + ("a run still in progress refuses", ["c1:running", "p"], "p", {}, "none"), + ("skipped and neutral runs count as green", ["c1:a=skipped,b=neutral", "p"], "p", {}, "c1"), + ("one failed workflow of two refuses", ["c1:a=success,b=failure", "p"], "p", {}, "none"), + ("a pin at the tip is up to date", ["p:success", "c0:success"], "p", {}, "none"), + ("a pin beyond the window, a repository without workflows: its commits are green", + ["c2", "c1"], "old", {"workflows": 0}, "c2"), + ("a pin beyond the window, a repository with workflows and no runs: an error, not a guess", + ["c2", "c1"], "old", {}, "error: no usable commit among the last 2 on main"), + ("a pin beyond the window: a commit without runs inherits the nearest older run", + ["bot", "c1:success"], "old", {}, "bot"), + ("unpinned: the green tip is taken", ["c1:success"], None, {}, "c1"), + ("unpinned, no run anywhere, workflows exist: an error", ["c1"], None, {}, "error: no usable commit among the last 1 on main"), + ("a branch without commits is an error", [], "p", {}, "error: no commits on main"), + ("a green run on a side branch is not CI of the tracked branch", + ["m:running", "f:success@feature", "p:success"], "p", {}, "none"), + ("a green run of a tag push does not hide a failed run of the branch", ["t:w=success@v1/failure", "p:success"], "p", {}, "none"), +] + + +def git(*args: str, cwd: Path) -> str: + return subprocess.run(["git", "-c", "user.name=t", "-c", "user.email=t@t", "-c", "commit.gpgsign=false", *args], + cwd=cwd, check=True, capture_output=True, text=True).stdout.strip() + + +def resolve_cases(tmp: Path) -> None: + """The resolve step in a brain with mounts: `good` moves, `new` (not pinned yet) gets its first pin, + `bad` fails (401), `odd` gets an answer without the expected key, `else` is not on GitHub.""" + old = os.getcwd() + os.environ["GITHUB_REPOSITORY"] = "Own/brain" + try: + empty = tmp / "empty" + empty.mkdir() + os.chdir(empty) + git("init", "-q", cwd=empty) + check(bump.submodules() == [], "without a .gitmodules there is nothing to move", bump.submodules()) + (empty / ".gitmodules").write_text("") + check(bump.submodules() == [], "an empty .gitmodules (the last mount removed) is nothing to move, no crash", bump.submodules()) + + b = tmp / "brain" + b.mkdir() + os.chdir(b) + git("init", "-q", "-b", "main", cwd=b) + (b / ".gitmodules").write_text( + '[submodule "good"]\n\tpath = brains/good\n\turl = ../good.git\n\tignore = all\n' + '[submodule "bad"]\n\tpath = brains/bad\n\turl = git@github.com:own/bad.git\n\tbranch = .\n' + '[submodule "else"]\n\tpath = brains/else\n\turl = https://gitlab.com/x/else.git\n' + '[submodule "new"]\n\tpath = brains/new\n\turl = ../new.git\n' + '[submodule "new-again"]\n\tpath = brains/new-again\n\turl = ../new.git\n' + '[submodule "odd"]\n\tpath = brains/odd\n\turl = ../odd.git\n\tbranch = main\n' + '[submodule "other"]\n\tpath = brains/other\n\turl = ../../them/other.git\n') + for path, pin in (("brains/good", "g0"), ("brains/bad", "b0"), ("brains/odd", "o0")): + git("update-index", "--add", "--cacheinfo", f"160000,{sha(pin)},{path}", cwd=b) + git("add", ".gitmodules", cwd=b) + git("commit", "-q", "-m", "brain", cwd=b) + subs = {s["name"]: s for s in bump.submodules()} + check([subs["good"]["slug"], subs["bad"]["slug"], subs["bad"]["branch"], subs["good"]["branch"], subs["else"]["slug"], subs["other"]["slug"]] + == ["Own/good", "own/bad", "main", None, None, "them/other"], + "slugs: ../x is a sibling under the brain's owner, ../../o/x another owner's, ssh URLs parse, " + "`branch = .` is main, no branch is the default branch, another host has none", subs) + out = subprocess.run([sys.executable, str(REPO / "src/km/templates/team/bump-mounted-pins.py"), "--list-repos", "own"], + cwd=b, capture_output=True, text=True) + check(out.stdout.strip() == "good,bad,new,odd", "--list-repos names the mounts of the owner, in any case", out.stdout + out.stderr) + + (b / ".gitmodules").write_text((b / ".gitmodules").read_text().split('[submodule "other"]')[0]) + git("add", ".gitmodules", cwd=b) + git("commit", "-q", "-m", "drop other", cwd=b) + fakes = {"Own/good": Fake(["g1:w=success@trunk/failure@main", "g0:success@trunk"], default="trunk"), "Own/new": Fake(["n1:success"])} + + def api(path, token, method="GET", payload=None): # bad is refused, odd answers without workflow_runs + slug = "/".join(path.split("/")[1:3]) + if slug == "Own/odd": + return [{"sha": sha("o1")}, {"sha": sha("o0")}] if "/commits?" in path else {} + if slug not in fakes: + raise bump.ApiError("HTTP 401", "Bad credentials") + return fakes[slug].api(path.replace(slug, "o/r", 1), token) + bump.api = api + msg, body = tmp / "msg", tmp / "body" + rc = bump.resolve(SimpleNamespace(write=True, emit_message=str(msg), emit_body=str(body), hold_if_moved=None), "t") + staged = dict(reversed(ln.split()[1:4:2]) for ln in git("ls-files", "-s", "brains", cwd=b).splitlines()) + text = body.read_text(encoding="utf-8") + check(rc == 1, "a mount that cannot be resolved fails the run", rc) + check(staged.get("brains/good") == sha("g1"), "the other mounts still move", staged) + check(staged.get("brains/new") == sha("n1"), "a mount not pinned yet gets its first pin", staged) + check(staged.get("brains/odd") == sha("o0") and "odd: KeyError" in text, "an unexpected answer fails that mount only", text) + check(staged.get("brains/bad") == sha("b0"), "the failed mount stays at main's pin", staged) + check(not re.search(r"[0-9a-f]{40}", text + msg.read_text(encoding="utf-8")) and f"`{sha('g1')[:8]}`" in text, + "the body and the message carry short shas only", text) + check("bad: HTTP 401" in text and "else: no path or no GitHub url" in text, "the body names what did not move", text) + check("--keep-from" not in REPO.joinpath("src/km/templates/team/bump-mounted-pins.py").read_text(encoding="utf-8"), + "nothing is carried over from the pull request branch", "") + failed_diff = subprocess.run(["git", "diff", "--cached", "--quiet", "HEAD", "--", "brains/good"], cwd=b).returncode + none_diff = subprocess.run(["git", "diff", "--cached", "--quiet", "--ignore-submodules=none", "HEAD", "--", "brains/good"], cwd=b).returncode + check(failed_diff == 0 and none_diff == 1, "with `ignore = all` only --ignore-submodules=none sees the moved gitlink", + (failed_diff, none_diff)) + def held(change: str) -> bool: + """Build a bump commit on main changing `change` (path=pin), then main moves bad by hand; + resolve with --hold-if-moved on that commit: True when it wrote nothing.""" + git("reset", "-q", "--hard", "HEAD", cwd=b) + base = git("rev-parse", "HEAD", cwd=b) + path, pin = change.split("=") + git("update-index", "--add", "--cacheinfo", f"160000,{sha(pin)},{path}", cwd=b) + git("commit", "-q", "-m", "bump", cwd=b) + git("update-ref", "refs/bump/proposed", "HEAD", cwd=b) + git("reset", "-q", "--hard", base, cwd=b) + git("update-index", "--cacheinfo", f"160000,{sha('b5')},brains/bad", cwd=b) # by hand on main + git("commit", "-q", "-m", "by hand", cwd=b) + msg.unlink(missing_ok=True) + args = SimpleNamespace(write=True, emit_message=str(msg), emit_body=str(body), hold_if_moved="refs/bump/proposed") + rc = bump.resolve(args, "t") + git("reset", "-q", "--hard", base, cwd=b) + return rc == 1 and not msg.exists() + check(held("brains/bad=b1"), "a failing mount the open pull request moves: nothing is written", "") + check(not held("brains/good=g1"), "a failing mount the pull request does not move (main moved it since) holds nothing back", "") + check(set(fakes["Own/good"].branches) == {None}, "a mount without a branch line lists its default branch, and its runs count", + fakes["Own/good"].branches) + finally: + os.chdir(old) + + +def api_errors() -> None: + """An answer that is not JSON, or a connection dropped mid-read, fails the call as ApiError (one + mount), not with a traceback that stops every mount.""" + import http.client + import io + real = bump.urllib.request.urlopen + for what, answer in (("an answer that is not JSON", lambda *a, **k: io.BytesIO(b"")), + ("a connection dropped mid-read", lambda *a, **k: (_ for _ in ()).throw(http.client.RemoteDisconnected("x")))): + bump.urllib.request.urlopen = answer + try: + REAL_API("repos/o/r", "t") + got = "no error" + except bump.ApiError as e: + got = "ApiError" + except Exception as e: # noqa: BLE001 + got = type(e).__name__ + finally: + bump.urllib.request.urlopen = real + check(got == "ApiError", f"{what} is an ApiError", got) + + +def branch_head_cases(tmp: Path) -> None: + """--branch-prs takes a head a pull request was opened for as current only while it is one + commit on main that changes gitlinks and nothing else; branch names are sent quoted.""" + import contextlib + import io + old, old_repo, old_api = os.getcwd(), os.environ.get("GITHUB_REPOSITORY"), bump.api + os.environ["GITHUB_REPOSITORY"] = "own/brain" + os.chdir(tmp) + (tmp / ".gitmodules").write_text('[submodule "m"]\n\tpath = brains/m\n\turl = ../m.git\n') + try: + for what, ahead, files, want in ( + ("a head that only moves gitlinks is current", 1, ["brains/m"], "true true"), + ("a head that also changes another file is regenerated", 1, ["brains/m", "README.md"], "true false"), + ("a branch with a commit pushed below the head is regenerated", 2, ["brains/m"], "true false"), + ("a compare answer without its files is not taken as gitlinks only", 1, [], "true false")): + calls = [] + + def api(path, token, method="GET", payload=None): + calls.append(path) + if path.startswith("repos/own/brain/pulls?"): + return [{"state": "open", "head": {"sha": "h"}}] + if path == "repos/own/brain/compare/main...h": + return {"ahead_by": ahead, "files": [{"filename": f} for f in files]} + raise AssertionError(path) + bump.api = api + out = io.StringIO() + with contextlib.redirect_stdout(out): + bump.report_branch_prs("chore/bump-mounted-pins", "h", "t") + check(out.getvalue().split() == want.split(), what, out.getvalue()) + check("head=own%3Achore%2Fbump-mounted-pins&" in calls[0], "the branch is sent quoted", calls[0]) + fake = Fake(["c1:success@rel/1.x", "p:success@rel/1.x"]) + bump.api = fake.api + got, _ = bump.newest_commit("o/r", "rel/1.x", sha("p"), "t") + check(got == sha("c1") and fake.branches == ["rel%2F1.x"], "a branch name with a slash is sent quoted and its runs count", + (got, fake.branches)) + finally: + os.chdir(old) + bump.api = old_api + if old_repo is None: + os.environ.pop("GITHUB_REPOSITORY", None) + else: + os.environ["GITHUB_REPOSITORY"] = old_repo + + +def main() -> int: + for what, commits, pin, kw, want in CASES: + got = pick(commits, pin, **kw) + check(got == want, what, f"got {got}, want {want}") + api_errors() + with tempfile.TemporaryDirectory() as tmp: + resolve_cases(Path(tmp)) + with tempfile.TemporaryDirectory() as tmp: + branch_head_cases(Path(tmp)) + print(f"mounted_pins: {passed} passed, {failed} failed") + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/mounted_pins_smoke.sh b/tests/mounted_pins_smoke.sh new file mode 100644 index 0000000..2eb1fd0 --- /dev/null +++ b/tests/mounted_pins_smoke.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash +# Smoke test for the bump-mounted-pins workflow and script a team brain gets: the commit selection +# against a stand-in GitHub API (tests/mounted_pins_check.py), the workflow's shape (which job holds +# which token), and its resolve step run against a local origin with a stand-in script. +# Run: bash tests/mounted_pins_smoke.sh +set -u +source "$(dirname "$0")/lib.sh" +WF="$REPO/src/km/templates/team/bump-mounted-pins.yml" +T="$(mktemp -d)"; trap 'rm -rf "$T"' EXIT + +out="$(kmpy "$REPO/tests/mounted_pins_check.py" 2>&1)"; rc=$? +echo "$out" | grep -E "FAIL|Traceback|Error" ; echo "$out" | tail -1 +[ "$rc" = 0 ] && ok "commit selection and resolve against a stand-in API" || no "mounted_pins_check.py" + +# --- the workflow's shape --- +kmpy - "$WF" <<'PY' && ok "the app token lives in resolve alone, behind the main-only environment; publish alone writes" || no "workflow shape" +import sys, yaml +wf = yaml.safe_load(open(sys.argv[1], encoding="utf-8")) +jobs, text = wf["jobs"], open(sys.argv[1], encoding="utf-8").read() +problems = [] +def need(cond, what): + cond or problems.append(what) +need(wf["defaults"]["run"]["shell"] == "bash", "defaults.run.shell is bash (a container job runs sh otherwise)") +need(wf["permissions"] == {"contents": "read"}, "read-only by default") +for name, job in jobs.items(): + need(job["runs-on"] == "${{ vars.MOUNTED_PINS_RUNNER || 'ubuntu-latest' }}", f"{name}: ubuntu-latest unless the brain names a runner") + need("@sha256:" in job.get("container", ""), f"{name}: the image is pinned by digest") + uses = [s["uses"].split("@")[0] for s in job["steps"] if "uses" in s] + allowed = ["actions/checkout", "actions/create-github-app-token"] if name == "resolve" else ["actions/checkout"] + need(set(uses) <= set(allowed), f"{name}: runs no other action ({uses})") + need(all(s.get("with", {}).get("persist-credentials") is False for s in job["steps"] if s.get("uses", "").startswith("actions/checkout")), + f"{name}: checkout keeps no credentials") + need(not any("pip " in s.get("run", "") for s in job["steps"]), f"{name}: installs no Python package") + secrets = "secrets." in yaml.safe_dump(job) + need(secrets == (name == "resolve"), f"{name}: {'reads' if name == 'resolve' else 'never reads'} the app's secrets") + need(("environment" in job) == (name == "resolve"), f"{name}: environment only where the key is") +need(jobs["resolve"]["environment"] == "mounted-pins", "the key's environment is mounted-pins") +need(jobs["resolve"]["if"] == "needs.mounts.outputs.names != ''", "no mounts: resolve does not run, no environment is created") +token = next(s for s in jobs["resolve"]["steps"] if s.get("uses", "").startswith("actions/create-github-app-token")) +need(token["with"]["owner"] == "${{ github.repository_owner }}", "the token's owner is the brain's owner") +need(token["with"]["permission-contents"] == "read" and token["with"]["permission-actions"] == "read", "a read-only token") +need(jobs["publish"]["permissions"] == {"contents": "write", "pull-requests": "write", "actions": "write"}, "publish permissions") +need(jobs["resolve"]["permissions"] == {"contents": "read", "pull-requests": "read"}, "resolve permissions") +need("--force-with-lease=" in text and "--ignore-submodules=none" in text, "lease push, gitlink diff that sees `ignore = all` mounts") +need("://x-access-token" not in text and "GIT_CONFIG_VALUE_0=" in text, "the token reaches git through the environment, not a URL") +for p in problems: + print(f" {p}") +sys.exit(1 if problems else 0) +PY + +# --- the resolve step against a local origin --- +# A stand-in for the script: --branch-prs prints $PRS ("open seen"), --write sets the gitlink of +# brains/m to $NEW; $FAILED (paths) could not be resolved, and with --hold-if-moved REF it writes +# nothing when REF's commit moves one of them, as the script does. +if [ "${BASH_VERSINFO[0]}" -lt 4 ]; then + echo " skip: the resolve step needs bash 4 or newer (mapfile), this is ${BASH_VERSION}" +else + kmpy -c 'import sys, yaml +step = next(s for s in yaml.safe_load(open(sys.argv[1]))["jobs"]["resolve"]["steps"] if s.get("id") == "bump") +print(step["run"].replace("/tmp/pin-", sys.argv[2] + "/pin-"))' "$WF" "$T" > "$T/step.sh" # files of its own, as in a fresh container + mkdir -p "$T/bin" && printf '#!/bin/sh\n[ "$1" = -w0 ] && shift\nexec %s "$@"\n' "$(command -v base64)" > "$T/bin/base64" && chmod +x "$T/bin/base64" + O="$T/origin.git"; W="$T/work" + git init -q --bare -b main "$O" && git init -q -b main "$W" + mkdir -p "$W/.github/scripts" + printf '[submodule "m"]\n\tpath = brains/m\n\turl = ../m.git\n\tignore = all\n' > "$W/.gitmodules" + cat > "$W/.github/scripts/bump-mounted-pins.py" <<'PY' +import os, subprocess, sys +args = sys.argv[1:] +if args[0] == "--branch-prs": + print(os.environ["PRS"]) + sys.exit(0) +failed = os.environ.get("FAILED", "").split() +ls = lambda ref, p: subprocess.run(["git", "ls-tree", ref, "--", p], capture_output=True, text=True, check=True).stdout +hold = "--hold-if-moved" in args and any(ls(args[args.index("--hold-if-moved") + 1], p) != ls(args[args.index("--hold-if-moved") + 1] + "^", p) + for p in failed) +if os.environ["NEW"] != os.environ["MAIN_PIN"] and not hold: + subprocess.run(["git", "update-index", "--cacheinfo", f"160000,{os.environ['NEW']},brains/m"], check=True) + open(args[args.index("--emit-message") + 1], "w").write("move\n") + open(args[args.index("--emit-body") + 1], "w").write("body\n") +sys.exit(1 if failed else 0) +PY + A="$(printf 'a%.0s' {1..40})"; B="$(printf 'b%.0s' {1..40})"; C="$(printf 'c%.0s' {1..40})" + ( cd "$W" && git update-index --add --cacheinfo "160000,$A,brains/m" && git add .gitmodules .github && git "${G[@]}" commit -q -m main \ + && git remote add origin "$O" && git push -q origin main ) && git clone -q "$O" "$T/side" + branch_with(){ # the bump branch on origin, proposing pin $1; none: no branch. Made in a clone of its + # own, since the step's fetch makes the work tree's repository shallow. + git -C "$O" update-ref -d refs/heads/chore/bump-mounted-pins + [ "$1" = none ] && return + ( cd "$T/side" && git checkout -q -B tmp origin/main && git update-index --cacheinfo "160000,$1,brains/m" && git "${G[@]}" commit -q -m p \ + && git push -q -f origin HEAD:refs/heads/chore/bump-mounted-pins ) + } + step(){ # PRS NEW [FAILED]: run the step, print changed= and problems= + ( cd "$W" && git reset -q --hard main && : > "$T/out" && rm -f "$T"/pin-* \ + && PATH="$T/bin:$PATH" PRS="$1" NEW="$2" FAILED="${3:-}" MAIN_PIN="$A" GITHUB_OUTPUT="$T/out" BRANCH=chore/bump-mounted-pins \ + GH_TOKEN=t GH_PIN_TOKEN=t bash --noprofile --norc -eo pipefail "$T/step.sh" >/dev/null 2>&1 ) || echo "step failed" + grep -o "^changed=.*" "$T/out"; grep -o "^problems=.*" "$T/out" + } + r(){ echo $(step "$@"); } + branch_with none; [ "$(r "false false" "$B")" = "changed=true problems=0" ] && ok "no bump branch: the move is proposed" || no "no branch ($(r "false false" "$B"))" + branch_with "$B"; [ "$(r "true true" "$B")" = "changed=false problems=0" ] && ok "an open pull request already proposing these pins: nothing is pushed" || no "open PR, same pins ($(r "true true" "$B"))" + branch_with "$B"; [ "$(r "true true" "$C")" = "changed=true problems=0" ] && ok "an open pull request with older pins is moved on" || no "open PR, newer pins ($(r "true true" "$C"))" + branch_with "$B"; [ "$(r "true true" "$C" brains/m)" = "changed=false problems=1" ] && ok "a mount that fails while the open pull request moves it: nothing is pushed, the run fails" || no "open PR moving a failing mount ($(r "true true" "$C" brains/m))" + branch_with "$B"; [ "$(r "true true" "$C" brains/x)" = "changed=true problems=1" ] && ok "a mount that fails and the open pull request does not move: the others are still proposed, the run fails" || no "open PR, a failing mount it does not move ($(r "true true" "$C" brains/x))" + branch_with none; [ "$(r "false false" "$B" brains/m)" = "changed=true problems=1" ] && ok "a mount that fails with no pull request open: the others are still proposed, the run fails" || no "no PR, a failing mount ($(r "false false" "$B" brains/m))" + branch_with "$B"; [ "$(r "false false" "$B")" = "changed=true problems=0" ] && ok "a branch whose pull request never opened is pushed again, so the pull request opens" || no "M1 retry ($(r "false false" "$B"))" + branch_with "$B"; [ "$(r "false true" "$B")" = "changed=false problems=0" ] && ok "a pull request closed on these pins is not proposed again" || no "closed PR ($(r "false true" "$B"))" + branch_with none; [ "$(r "false false" "$A")" = "changed=false problems=0" ] && ok "nothing to move: no change" || no "nothing to move ($(r "false false" "$A"))" +fi +summary mounted_pins_smoke diff --git a/tests/team_smoke.sh b/tests/team_smoke.sh index f22dc8c..9fae3bd 100644 --- a/tests/team_smoke.sh +++ b/tests/team_smoke.sh @@ -148,7 +148,7 @@ python3 -c "import json,sys; c=json.load(open('$m'))['counts']; sys.exit(0 if c= G checkout -q -- schema.local.yaml process/cust.md # --- workflows --- -grep -q "scripts/" "$T"/.github/workflows/*.yml && no "workflows call km, not copied scripts" || ok "workflows call km, not copied scripts" +grep -h "scripts/" "$T"/.github/workflows/*.yml | grep -qv "\.github/scripts/bump-mounted-pins\.py" && no "workflows call km, not copied scripts" || ok "workflows call km, not copied scripts" python3 -c "import yaml,glob; [yaml.safe_load(open(f)) for f in glob.glob('$T/.github/workflows/*.yml')]" 2>/dev/null \ && ok "workflows are valid YAML" || no "workflows are valid YAML" python3 - "$T/.github/workflows/staleness.yml" <<'PY' && ok "staleness validates after demoting, even after a failed step" || no "staleness validates after demoting" @@ -383,7 +383,9 @@ def pins(paths): return out own = pins([f"{repo}/action.yml", *glob.glob(f"{repo}/.github/workflows/*.yml")]) tpl = pins(glob.glob(f"{repo}/src/km/templates/team/*.yml")) -sys.exit(0 if all(name in own and shas == own[name] for name, shas in tpl.items()) else 1) +# Actions only a template uses: no Dependabot here sees them; each brain's own Dependabot moves them. +TEMPLATE_ONLY = {"actions/create-github-app-token"} +sys.exit(0 if all(shas == own[name] if name in own else name in TEMPLATE_ONLY for name, shas in tpl.items()) else 1) PY # --- non-ASCII file names on a system whose encoding is not UTF-8 (Windows, cp1252) --- @@ -458,6 +460,55 @@ printf 'on:\n pull_request:\njobs:\n v:\n steps:\n - run: make\n' > "$ out="$(km upgrade --root "$D" 2>&1)" [ "$(on_keys "$D/.github/workflows/ci.yml")" = "pull_request" ] && ok "a ci.yml that does not install km gets no trigger" || no "own ci.yml gets no trigger ($out)" +# --- the workflow that moves the mounted pins, and its script --- +PW=.github/workflows/bump-mounted-pins.yml; PS=.github/scripts/bump-mounted-pins.py +{ cmp -s "$T/$PW" "$REPO/src/km/templates/team/bump-mounted-pins.yml" && cmp -s "$T/$PS" "$REPO/src/km/templates/team/bump-mounted-pins.py"; } \ + && ok "a team brain gets the mounted-pins workflow and its script" || no "team brain mounted-pins files" +{ [ -f "$O/$PW" ] && [ -f "$O/$PS" ]; } && ok "upgrade adds them to a team brain without them" || no "upgrade adds the mounted-pins files" +[ ! -e "$P/$PW" ] && [ ! -e "$P/$PS" ] && ok "a personal brain does not get them" || no "personal brain has no mounted-pins files" +kmpy - <<'PY' && ok "SHIPPED lists the current mounted-pins templates" || no "a mounted-pins template changed: add its hash to SHIPPED in upgrade.py" +import sys, km.upgrade as u +from km.init import template +sys.exit(any(u.shipped_hash(template(t, {})) not in h for t, h in u.SHIPPED.items())) +PY +kmpy -c 'import km.upgrade as u +h = u.shipped_hash +assert h(" - uses: a/b@1 # v1\n") == h(" - uses: a/b@2 # v2\n") and h(" uses: \"a/b@1\"\n") == h(" uses: \"a/b@2\"\n") +assert h(" - uses: a/b@1\n") != h(" - uses: a/c@1\n")' && ok "an action pin and its comment do not count as an edit, the action does" || no "shipped_hash masking" +kmpy -c 'import km.upgrade as u +new = " - uses: a/b@k # v2.0.0\n - uses: a/c@k # v2.0.0\n" +got = u.carry_pins(" - uses: a/b@n # v2.1.0\n - uses: a/c@o # v1.9.9\n", new) +assert got == " - uses: a/b@n # v2.1.0\n - uses: a/c@k # v2.0.0\n", got' \ + && ok "an upgrade keeps an action pin the brain moved past the template, and takes the template pin where that is newer" || no "carry_pins direction" +# an earlier version km shipped is replaced, also after the brain's Dependabot moved its action +# pins: a stand-in for one (a line ahead of the template), registered as shipped for this run +MOVE='s/\(uses: actions\/checkout@\)[^ ]*.*/\1aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa # v99/' +for f in "$PW:bump-mounted-pins.yml" "$PS:bump-mounted-pins.py"; do + { echo "# an earlier km version"; sed "$MOVE" "$REPO/src/km/templates/team/${f#*:}"; } > "$O/${f%%:*}" +done +upgrade_older(){ out="$(cd "$O" && kmpy - 2>&1 <<'PY' +import sys, km.upgrade as u +from km.init import template +for t, h in u.SHIPPED.items(): + h.add(u.shipped_hash("# an earlier km version\n" + template(t, {}))) +sys.argv = ["km upgrade"] +sys.exit(u.main()) +PY +)"; } +upgrade_older +{ cmp -s "$O/$PW" <(sed "$MOVE" "$REPO/src/km/templates/team/bump-mounted-pins.yml") && cmp -s "$O/$PS" "$REPO/src/km/templates/team/bump-mounted-pins.py" \ + && out_has "updated $PW to this km's version" && out_has "updated $PS to this km's version"; } \ + && ok "upgrade replaces a copy that is an earlier version km shipped, keeping the action pins its Dependabot moved" || no "upgrade replaces an unchanged earlier copy ($out)" +sed "$MOVE" "$O/$PW" > "$F/moved.yml" && mv "$F/moved.yml" "$O/$PW"; cp "$O/$PW" "$F/moved.yml"; upgrade_older +{ cmp -s "$O/$PW" "$F/moved.yml" && ! out_has "mounted-pins"; } \ + && ok "the current version with pins the brain's Dependabot moved is left as it is" || no "current copy with moved pins ($out)" +echo "# the brain's edit" >> "$O/$PW"; echo "# the brain's edit" >> "$O/$PS"; out="$(km upgrade --root "$O" 2>&1)" +{ tail -1 "$O/$PW" | grep -q "the brain's edit" && tail -1 "$O/$PS" | grep -q "the brain's edit" \ + && out_has "left $PW: the brain changed it" && out_has "left $PS: the brain changed it"; } \ + && ok "upgrade leaves an edited copy, with a note" || no "upgrade keeps edited mounted-pins files ($out)" +rm "$O/$PW"; touch "$O/.gitmodules"; out="$(km upgrade --root "$O" 2>&1)"; rm "$O/.gitmodules" +out_has "create the environment mounted-pins restricted to main" && ok "upgrade adding the workflow to a brain with mounts says to restrict the environment first" || no "environment note on upgrade ($out)" + # --- main guard: the hook that refuses a commit on main, and the workflow that flags a direct push --- hooks_of(){ kmpy -c 'import sys, yaml print(" ".join(r["repo"].rsplit("/", 1)[-1] + ":" + h["id"] + ":" + ",".join(h.get("args", []))