diff --git a/.github/dependabot.yml b/.github/dependabot.yml index fd0ece4..ee791c1 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -1,6 +1,8 @@ # Keeps the commit-pinned actions in this repository's workflows and in action.yml current. The # templates brains get are not watched: a test fails until they pin the same commits, so a -# Dependabot update here has to be carried into src/km/templates/team/*.yml by hand. +# Dependabot update here has to be carried into src/km/templates/team/*.yml by hand. An action only a +# template uses (actions/create-github-app-token, TEMPLATE_ONLY in tests/team_smoke.sh) is watched by +# no one here: each brain's Dependabot moves it there; bump it in the template by hand. version: 2 updates: - package-ecosystem: github-actions diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 6af7989..438d333 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -32,6 +32,7 @@ jobs: - run: bash tests/list_smoke.sh - run: bash tests/skill_smoke.sh - run: bash tests/hook_smoke.sh + - run: bash tests/mounted_pins_smoke.sh - name: team_smoke.sh, with a Latin-1 locale for its file-name encoding test run: sudo localedef -i en_US -f ISO-8859-1 en_US.ISO-8859-1 && bash tests/team_smoke.sh diff --git a/CLAUDE.md b/CLAUDE.md index fe9dba5..6027812 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,6 +14,7 @@ brains. See README.md for the layout. - Brains pin a km version and never carry km code, but their `schema.local.yaml`, docs and templates from earlier versions stay: keep new versions working on them, and keep `km upgrade` able to move an older brain forward. +- The one exception is the team template `bump-mounted-pins.py` (with its workflow), copied into a team brain because the job holding the app token must not install third-party code, the km package included. `km upgrade` replaces a copy whose hash is in `SHIPPED` (src/km/upgrade.py): a change to either template adds its new hash there and keeps the old ones (a test checks the current one is listed). - A release bumps `src/km/__init__.py`, `.claude-plugin/plugin.json` and the README examples together (a test checks it). Merging it to main is the release: CI tags `vX.Y.Z` once main is green. The templates pin the tag of the km that wrote them. diff --git a/README.md b/README.md index 01b760f..c709a64 100644 --- a/README.md +++ b/README.md @@ -96,7 +96,7 @@ without installing: `uvx --from git+https://github.com/pasrom/km@ km val ## What a brain gets -Brains carry no km code. `km init` writes `CONVENTIONS.md`, `CLAUDE.md`, `schema.local.yaml`, +Brains carry no km code (the one exception, the script of the mounted-pins workflow, is below). `km init` writes `CONVENTIONS.md`, `CLAUDE.md`, `schema.local.yaml`, `inbox/`, a `.gitattributes` that keeps text files LF, and a pre-commit hook pinned to a km version. `km init --team` adds domain folders with generated indexes, a README, and CI that installs km through this repository's GitHub Action: @@ -108,6 +108,8 @@ generated indexes, a README, and CI that installs km through this repository's G The team brain's ci workflow also runs on `workflow_dispatch`: a pull request that a workflow opens with `GITHUB_TOKEN` starts no `pull_request` run, so such a workflow (a pin bump, say) dispatches ci on its branch and the pull request gets its check. `km upgrade` adds the trigger to a km ci workflow that lacks it. +A team brain also gets `bump-mounted-pins`, a workflow that moves the pins of the mounted brains (submodules) by pull request, in place of Dependabot's submodule updates, which need an organisation-wide grant for private submodules and stop every submodule of a group on one refusal. On weekday mornings it walks each mount's tracked branch (its `branch =`, else the repository's default branch) back from the tip to the newest commit whose CI on that branch did not fail (runs of a side branch or a tag do not count; a commit without a run of its own, such as a bot's, inherits the state of the nearest older commit that has runs; a pin never moves backwards), sets the gitlinks without cloning, force-pushes `chore/bump-mounted-pins` with a lease and opens or updates the pull request, which lists short shas only, and dispatches ci on it. A brain without mounts stops after reading `.gitmodules`. It reads the mounts with a GitHub App (Contents and Actions: read) whose secrets `CI_APP_CLIENT_ID` and `CI_APP_PRIVATE_KEY` live in an environment `mounted-pins` restricted to `main`; the workflow's header says how to set it up, and `km doctor` notes a missing or unrestricted environment. Its script, `.github/scripts/bump-mounted-pins.py`, is the one piece of km code a brain carries: the job that holds the app token installs nothing, so it cannot run the km package. `km upgrade` adds both files when missing and replaces a copy that is still a version km shipped, unchanged but for the action pins its Dependabot moves (a pin moved past the template's version is kept), so later fixes reach the brain; a copy the brain edited is its own, left alone with a note. + The pre-commit hook works the same way (`repo: https://github.com/pasrom/km`, `rev: # frozen: v1.9.0`, `id: km-validate`). Every pin names a release by its commit, so a tag moved later cannot change what a brain runs; the version rides along as a comment, and @@ -146,7 +148,7 @@ tests/ smoke tests ## Development The smoke tests need Python 3.11+, PyYAML and git: `bash tests/gate_smoke.sh`, `index_smoke.sh`, -`team_smoke.sh`, `doctor_smoke.sh`, `sync_smoke.sh`, `list_smoke.sh`, `skill_smoke.sh` (the skill's core +`team_smoke.sh`, `doctor_smoke.sh`, `sync_smoke.sh`, `list_smoke.sh`, `mounted_pins_smoke.sh` (the mounted-pins workflow and script a team brain gets, against a stand-in GitHub API), `skill_smoke.sh` (the skill's core keeps its size limits, its reference paths resolve, every `/km` verb has a dispatch row) and `hook_smoke.sh` (the plugin's edit guard; the pre-push hook too where pre-commit is installed). CI runs them plus the GitHub Action on every pull request, and `encoding_smoke.sh`, `doctor_smoke.sh`, `sync_smoke.sh` and `hook_smoke.sh` on Windows (team_smoke runs the encoding test on macOS under non-UTF-8 locales). A release bumps diff --git a/skills/km/reference/brains.md b/skills/km/reference/brains.md index ead25ad..943cefb 100644 --- a/skills/km/reference/brains.md +++ b/skills/km/reference/brains.md @@ -24,7 +24,7 @@ checks every mount out at its pin and never moves one ahead of it. Pass its notes on: a mount with changes of its own or ahead of its pin is left as it is (`brain reset ` puts it back), never a silent skip. Newer content of a mount comes only with a moved pin, through a pull - request: the host's Dependabot PR, or the pin-bump PR below when the user asks for it + request: a team brain's bump-mounted-pins PR, or the pin-bump PR below when the user asks for it ### Shared resources (self-describing repos) @@ -48,7 +48,7 @@ mount at a newer commit (or km sync reports one ahead of its pin that they want step may move the checkout past the pin: it is the bump), `git add brains/`, commit `chore(brains): bump to `, push, `gh pr create`; then `git switch main` and `km sync`, which puts the mount back at main's pin until the PR merges. A team brain gets the -same bump from Dependabot's submodule PRs. +same bump from its bump-mounted-pins workflow (`chore/bump-mounted-pins`). ## Brain management diff --git a/src/km/doctor.py b/src/km/doctor.py index 76a431e..b8e33fa 100644 --- a/src/km/doctor.py +++ b/src/km/doctor.py @@ -371,9 +371,10 @@ def check_line_endings(root: Path) -> None: "{ nodes { nameWithOwner parent { nameWithOwner } } } } }") -def check_github(origin: str, offline: bool) -> None: +def check_github(origin: str, offline: bool) -> str | None: """On a GitHub brain, `/km pr`, a contribution and the search of open pull requests run through - gh. Without push access to origin, a contribution goes through a fork.""" + gh. Without push access to origin, a contribution goes through a fork. Returns the repository's + owner/name when gh can see it.""" repo = github_repo(origin) if not repo: return @@ -400,7 +401,7 @@ def check_github(origin: str, offline: bool) -> None: push, _, full = out.partition("\t") # full_name: the current name, after a rename or transfer if push == "true": report("ok", f"you can push branches to {full}") - return + return full _, forks = gh("api", "--hostname", "github.com", "graphql", "-f", f"query={FORKS}", "--jq", f'.data.viewer.repositories.nodes[] | select((.parent.nameWithOwner // "" | ascii_downcase) == ' f'"{full.lower()}") | .nameWithOwner') @@ -409,6 +410,32 @@ def check_github(origin: str, offline: bool) -> None: else: report("note", f"you can read {full} but not push to it, and have no fork of it yet", "a contribution creates one (`gh repo fork --remote=false`); nothing to do before then") + return full + + +def check_mounted_pins(root: Path, repo: str) -> None: + """A brain with mounts and the bump-mounted-pins workflow needs the environment that holds the + workflow's app key, restricted to main (the workflow's header says how to set it up). One call; + a note, since only a maintainer can fix it.""" + if not ((root / ".gitmodules").is_file() and (root / ".github/workflows/bump-mounted-pins.yml").is_file()): + return + env = f"repos/{repo}/environments/mounted-pins" + code, policy = gh("api", "--hostname", "github.com", env, "--jq", + '.deployment_branch_policy | if . == null then "open" elif .custom_branch_policies then "custom" else "protected" end') + fix = "Settings, Environments, mounted-pins: deployment branches, selected branches, `main` only" + if code != 0: + report("note", f"{repo} has no environment mounted-pins (or gh cannot see it): the mounted pins do not move", + "a maintainer sets it up as the top of .github/workflows/bump-mounted-pins.yml describes") + return + names = "" + if policy == "custom": # a second call only for a list of chosen branches + _, names = gh("api", "--hostname", "github.com", f"{env}/deployment-branch-policies", "--jq", "[.branch_policies[].name] | join(\" \")") + if names == "main": + report("ok", f"{repo} has the environment mounted-pins, restricted to main") + else: + what = {"open": "every branch", "protected": "every protected branch"}.get(policy, names or "no branch") + report("note", f"the environment mounted-pins of {repo} is open to {what}, not to main alone: " + f"a workflow pushed on such a branch can read the app key", fix) def main() -> int: @@ -428,7 +455,8 @@ def main() -> int: check_submodules(root, origin, args.offline) check_hook(root) check_line_endings(root) - check_github(origin, args.offline) + if repo := check_github(origin, args.offline): + check_mounted_pins(root, repo) check_km_version(root) check_plugins(root) check_uv() diff --git a/src/km/init.py b/src/km/init.py index 56bd1d3..22cf64c 100644 --- a/src/km/init.py +++ b/src/km/init.py @@ -8,8 +8,8 @@ km init DIR --team --name NAME --desc TEXT --initials XX --folder DIR=PURPOSE [--folder ...] [--rule TEXT ...] [--profile SLUG] - A team brain: also the root and folder indexes, README, .gitignore, the two workflows and a - Dependabot config. Refuses to run if any file it would write already exists. + A team brain: also the root and folder indexes, README, .gitignore, its workflows (ci, + staleness, main-guard, bump-mounted-pins with its script) and a Dependabot config. Refuses to run if any file it would write already exists. Both render and check everything before writing anything, and refuse a release whose commit cannot be looked up (not released yet, or no network). Files only: git add, the first @@ -47,6 +47,8 @@ "team/staleness.yml": ".github/workflows/staleness.yml", "team/main-guard.yml": ".github/workflows/main-guard.yml", "team/dependabot.yml": ".github/dependabot.yml", + "team/bump-mounted-pins.yml": ".github/workflows/bump-mounted-pins.yml", + "team/bump-mounted-pins.py": ".github/scripts/bump-mounted-pins.py", "pre-commit-config.template.yaml": ".pre-commit-config.yaml", } PLACEHOLDERS = ("", "", "", "", "", "", diff --git a/src/km/sync.py b/src/km/sync.py index 3a36e09..348a3c6 100644 --- a/src/km/sync.py +++ b/src/km/sync.py @@ -15,7 +15,7 @@ - makes the mounted brains match what the brain pins: `git submodule update --init` for a mount that is missing or behind its pin; one with changes of its own, a branch checked out, or ahead of or diverged from its pin is left alone and reported. A pin never moves here; that is the - host's automation (Dependabot) and a reviewed pull request. + host's automation (the bump-mounted-pins workflow) and a reviewed pull request. A fork's parent that gh found is added once as the remote `upstream` (as `gh repo fork` does), so the next run needs no gh and the fetched branch has a ref of its own. diff --git a/src/km/templates/team/bump-mounted-pins.py b/src/km/templates/team/bump-mounted-pins.py new file mode 100644 index 0000000..b7abae5 --- /dev/null +++ b/src/km/templates/team/bump-mounted-pins.py @@ -0,0 +1,347 @@ +#!/usr/bin/env python3 +"""Move the gitlinks of the mounted repositories to the newest commit of their branch. + +Written by `km init --team` for the bump-mounted-pins workflow. `km upgrade` replaces it while it is +a version km shipped, unchanged; once this repo edits it, it is this repo's and upgrade leaves it. Standard library only, so the job holding the app token installs nothing. + + python3 .github/scripts/bump-mounted-pins.py # show what would change + python3 .github/scripts/bump-mounted-pins.py --write # move the gitlinks in the index + python3 .github/scripts/bump-mounted-pins.py --list-repos OWNER # mounted repositories of OWNER + python3 .github/scripts/bump-mounted-pins.py --branch-prs BRANCH SHA # pull requests of the branch + BODY=... python3 .github/scripts/bump-mounted-pins.py --open-pr BRANCH + + --emit-message FILE write a commit message for the change (only when something moves) + --emit-body FILE write a pull-request body for it (only when something moves) + --hold-if-moved REF write nothing when a submodule that fails is one REF's commit moves (the + open pull request's branch: its move would be lost over a passing failure) + +Every submodule in .gitmodules is read at run time: its path, its repository (a GitHub URL, or a +URL relative to this repository's own, `../name.git` or `../../owner/name.git`) and its tracked +branch (`branch =`, `.` for main, else the mounted repository's default branch). Nothing is cloned; without a .gitmodules there is nothing to do. The new commit is set in the index with +`git update-index --add --cacheinfo 160000,,`, so the working tree is not touched. + +Which commit: walking back from the branch tip, the newest commit whose CI did not fail. Per +workflow the newest push run of that commit counts; a failed or still running one refuses the +commit. A commit with no push run at all (a workflow that commits with GITHUB_TOKEN starts no run) +inherits the state of the nearest older commit that has runs, or of the current pin, and is taken +only if that is green and no newer commit was refused. A pin never moves to a commit that is not +ahead of it. + +Only the submodule name and the old and new short sha are reported: not every reader of this +repository reads the mounted ones, so no commit message, workflow name or compare link goes into +the log, the commit message or the pull request. + +Access: resolving reads the mounted repositories with GH_PIN_TOKEN (read-only), or with the gh +CLI's token when it is unset. --branch-prs reads and --open-pr writes this repository with GH_TOKEN. +""" + +from __future__ import annotations + +import argparse +import functools +import json +import os +import pathlib +import re +import http.client +import subprocess +import sys +import urllib.error +import urllib.parse +import urllib.request + +# One page of the commit list. Long enough that a pin left behind by a pull request open for weeks +# is still in it, so a commit without runs can inherit from it. +CANDIDATES = 100 +SUBJECT = "chore(brains): move the mounted pins to the newest commits" + + +class ApiError(Exception): + pass + + +def git(*args: str) -> str: + return subprocess.run(["git", *args], check=True, capture_output=True, text=True).stdout.strip() + + +def api(path: str, token: str, method: str = "GET", payload: dict | None = None): + """Call the GitHub REST API and return the decoded JSON (None for an empty answer).""" + request = urllib.request.Request( + f"https://api.github.com/{path}", method=method, + data=json.dumps(payload).encode() if payload else None, + headers={ + "Authorization": f"Bearer {token}", + "Accept": "application/vnd.github+json", + "X-GitHub-Api-Version": "2022-11-28", + "Content-Type": "application/json", + "User-Agent": "bump-mounted-pins", + }) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return json.loads(response.read() or b"null") + except urllib.error.HTTPError as error: + # 401, 403 and 404 are an expired token, a missing grant and a renamed repository: say + # which, rather than reporting every failure the same way. + raise ApiError(f"HTTP {error.code}", error.read().decode(errors="replace")) from None + except (OSError, ValueError, http.client.HTTPException) as error: + # no connection, a timeout, a dropped read, an answer that is not JSON: this mount fails + raise ApiError(type(error).__name__) from None + + +def submodules() -> list[dict]: + """name, path, slug and branch of every submodule in .gitmodules, in file order.""" + if not pathlib.Path(".gitmodules").is_file(): + return [] + found: dict[str, dict] = {} + owner = os.environ.get("GITHUB_REPOSITORY", "").partition("/")[0] + # exit 1: no such key, as in a .gitmodules left empty by removing the last mount + listed = subprocess.run(["git", "config", "-f", ".gitmodules", "--get-regexp", r"^submodule\..*\.(path|url|branch)$"], + capture_output=True, text=True) + if listed.returncode not in (0, 1): + raise subprocess.CalledProcessError(listed.returncode, listed.args, listed.stdout, listed.stderr) + for line in listed.stdout.splitlines(): + key, value = line.split(" ", 1) + name, field = key[len("submodule."):].rsplit(".", 1) + found.setdefault(name, {"name": name, "branch": None})[field] = value + for entry in found.values(): + if entry["branch"] == ".": # "the superproject's branch": main here + entry["branch"] = "main" + url = entry.get("url", "") + m = re.search(r"github\.com[/:]([^/]+/[^/]+?)(?:\.git)?/?$", url) + # ../name: a sibling of this repository, under the same owner; ../../owner/name: another owner + sibling = re.fullmatch(r"\.\./([^/]+?)(?:\.git)?/?", url) + other = re.fullmatch(r"\.\./\.\./([^/]+/[^/]+?)(?:\.git)?/?", url) + entry["slug"] = (m.group(1) if m else other.group(1) if other + else f"{owner}/{sibling.group(1)}" if sibling and owner else None) + return list(found.values()) + + +def q(value: str) -> str: + """A branch name as one query value.""" + return urllib.parse.quote(value, safe="") + + +def ci_state(slug: str, sha: str, branch: str, token: str) -> str: + """"green", "red" (the newest push run of a workflow failed or is still running) or "none". + Only runs of a push to `branch` count: a run on a side branch or a tag is not CI of this branch.""" + runs = api(f"repos/{slug}/actions/runs?head_sha={sha}&branch={q(branch)}&event=push&per_page=50", token)["workflow_runs"] + newest: dict[int, dict] = {} + for run in runs: # newest first, so the first run per workflow counts + newest.setdefault(run["workflow_id"], run) + if not newest: + return "none" + return "red" if any(r["conclusion"] not in ("success", "skipped", "neutral") for r in newest.values()) else "green" + + +def newest_commit(slug: str, branch: str | None, current: str | None, token: str) -> tuple[str | None, str]: + """(sha, why) of the newest pinnable commit on `branch` (None: the default branch), sha None + when the pin is up to date; raise ApiError when there is none.""" + commits = api(f"repos/{slug}/commits?{f'sha={q(branch)}&' if branch else ''}per_page={CANDIDATES}", token) + # The default branch's name, for the runs, only when a run is looked up. + tracked = (lambda: branch) if branch else functools.cache(lambda: api(f"repos/{slug}", token)["default_branch"]) + label = branch or "the default branch" + if not commits: + raise ApiError(f"no commits on {label}") + tip = commits[0]["sha"] + shas = [c["sha"] for c in commits] + pin_in_window = current in shas + if pin_in_window: + shas = shas[:shas.index(current)] + if not shas: + return None, "up to date" + # Below the window: the current pin was accepted when it was set; else looked up when needed. + below = "green" if pin_in_window else None + state = functools.cache(lambda sha: ci_state(slug, sha, tracked(), token)) + no_workflows = functools.cache( + lambda: api(f"repos/{slug}/actions/workflows?per_page=1", token)["total_count"] == 0) + + def inherited(i: int) -> str: + """A commit without a run of its own inherits the state of the nearest older commit that has + one: a bot commit on top of a failed push sits on a tree CI never passed.""" + found = next((s for s in map(state, shas[i + 1:]) if s != "none"), None) + if found or below: + return found or below + # The window ends before the pin. Without an older run to inherit from, only a repository + # with no workflow at all counts as green. + return "green" if no_workflows() else "none" + + # Nor is it taken below a refused commit: there sit the inner commits of the same push, which CI + # never built on their own. + refused = False + for i, sha in enumerate(shas): + own = state(sha) + if own == "red" or own == "none" and (refused or inherited(i) != "green"): + refused = True + continue + # The list is in date order, not ancestry order: a commit from a merged side branch can be + # older than the pin. Skip it rather than move back. + if current and api(f"repos/{slug}/compare/{current}...{sha}", token)["status"] != "ahead": + continue + return sha, "tip" if sha == tip else f"behind the tip {tip[:8]}, which was refused" + if pin_in_window: + return None, "no newer commit can be pinned" + raise ApiError(f"no usable commit among the last {len(commits)} on {label}") + + +def gitlink(ref: str, path: str) -> str | None: + """The commit `ref` pins at `path`, None when it pins none there.""" + entry = git("ls-tree", ref, "--", path).split() + return entry[2] if entry and entry[0] == "160000" else None + + +def moves(ref: str, path: str) -> bool: + """Does `ref`'s commit change the gitlink at `path`? Compared with its parent, the main it was + built on, not with main now, which may have moved the mount since. True when that is unknown.""" + try: + return gitlink(ref, path) != gitlink(f"{ref}^", path) + except subprocess.CalledProcessError: + return True + + +def resolve(args, token: str) -> int: + report: list[str] = [] + moved: list[str] = [] + problems: list[str] = [] + held: list[str] = [] # failed mounts the open pull request moves + for sub in submodules(): + name, path = sub["name"], sub.get("path") + current = None + try: + if not path or not sub["slug"]: + raise ApiError("no path or no GitHub url in .gitmodules") + current = gitlink("HEAD", path) + sha, why = newest_commit(sub["slug"], sub["branch"], current, token) + except (ApiError, KeyError, TypeError, subprocess.CalledProcessError) as error: + # One mount that cannot be resolved, whatever the reason, does not hold back the others. + why = error.args[0] if isinstance(error, ApiError) else f"{type(error).__name__}: {error}" + problems.append(f"{name}: {why}") + report.append(f"{name}: PROBLEM, {why}") + # It stays at main's pin. Nothing is carried over from the pull request branch: anyone + # who can push there could plant a pin the workflow would then commit as its own. + if path and args.hold_if_moved and moves(args.hold_if_moved, path): + held.append(name) + continue + was = current[:8] if current else "unpinned" + if sha is None: + report.append(f"{name}: {was} {why}") + continue + report.append(f"{name}: {was} -> {sha[:8]} ({why})") + moved.append(f"| {name} | `{was}` | `{sha[:8]}` |") + if args.write: + git("update-index", "--add", "--cacheinfo", f"160000,{sha},{path}") # --add: a mount not pinned yet + print("\n".join(report)) + if held: + # Pushing now would drop their proposed move over what may be a passing failure, and the + # next run would put it back: nothing is written, the pull request stays as it is. + print(f"The open pull request moves {', '.join(held)}, which could not be resolved: it is left as it is.") + moved = [] + + if moved and args.emit_message: + pathlib.Path(args.emit_message).write_text( + SUBJECT + "\n\n" + "\n".join(report) + "\n", + encoding="utf-8") + if moved and args.emit_body: + pathlib.Path(args.emit_body).write_text( + "Moves the mounted repositories to the newest commit of their tracked branch whose CI did not fail. Generated by the bump-mounted-pins workflow; the branch is regenerated on each run.\n\n" + "| Submodule | From | To |\n|---|---|---|\n" + "\n".join(moved) + "\n\n" + + ("Not moved:\n\n" + "\n".join(f"- {p}" for p in problems) + "\n\n" if problems else "") + + "Only the short shas are listed on purpose: not every reader of this repository reads the mounted ones.\n", + encoding="utf-8") + if problems: + print(f"{len(problems)} submodule(s) could not be resolved.", file=sys.stderr) + return 1 if problems else 0 + + +def list_repos(owner: str) -> int: + """Print the names of the mounted repositories of `owner`, comma-separated.""" + print(",".join(dict.fromkeys(sub["slug"].split("/")[1] for sub in submodules() + if sub["slug"] and sub["slug"].split("/")[0].lower() == owner.lower()))) + return 0 + + +def branch_prs(repo: str, branch: str, token: str, state: str) -> list[dict]: + return api(f"repos/{repo}/pulls?head={q(repo.split('/')[0] + ':' + branch)}&state={state}&per_page=100", token) + + +def report_branch_prs(branch: str, sha: str, token: str) -> int: + """Print two words, true or false: is a pull request of the branch open, and has one, open or + closed, `sha` as its head while `sha` is what this workflow pushes: one commit on main that + changes the submodule gitlinks and nothing else. A branch someone pushed more onto is not taken + as current, so the next push regenerates it.""" + repo = os.environ["GITHUB_REPOSITORY"] + prs = branch_prs(repo, branch, token, "all") + is_open = any(p["state"] == "open" for p in prs) + seen = any(p["head"]["sha"] == sha for p in prs) + if seen: + diff = api(f"repos/{repo}/compare/main...{sha}", token) + paths = {sub.get("path") for sub in submodules()} + files = diff.get("files") or [] # none listed: not known to be gitlinks only + seen = diff["ahead_by"] == 1 and bool(files) and all(f["filename"] in paths for f in files) + print(str(is_open).lower(), str(seen).lower()) + return 0 + + +def open_pr(args, token: str) -> int: + """Open or update the pull request, dispatch ci.yml.""" + repo = os.environ["GITHUB_REPOSITORY"] + body = os.environ["BODY"] + open_prs = branch_prs(repo, args.open_pr, token, "open") + if open_prs: + # The branch moved, so the description moves with it; otherwise it keeps describing the + # first day's pins. + number = open_prs[0]["number"] + api(f"repos/{repo}/pulls/{number}", token, "PATCH", {"body": body}) + print(f"Pull request #{number} updated.") + else: + created = api(f"repos/{repo}/pulls", token, "POST", { + "title": SUBJECT, + "head": args.open_pr, "base": "main", "body": body}) + print("opened", created["html_url"]) + + # workflow_dispatch is the one trigger GITHUB_TOKEN may start, and a dispatched run on the branch + # shows up on the pull request. km's ci.yml has the trigger; a ci.yml without it gets a 422, and + # the pull request then shows no check. + try: + api(f"repos/{repo}/actions/workflows/ci.yml/dispatches", token, "POST", {"ref": args.open_pr}) + print("ci.yml dispatched on", args.open_pr) + except ApiError as error: + if "workflow_dispatch" not in (error.args[1:] or [""])[0]: + raise + print("::notice::ci.yml has no workflow_dispatch trigger, so it does not run on the pull request") + return 0 + + +def main() -> int: + ap = argparse.ArgumentParser(description="Move the mounted repositories' gitlinks forward.") + ap.add_argument("--write", action="store_true", help="move the gitlinks in the index") + ap.add_argument("--emit-message", metavar="FILE", help="write a commit message here") + ap.add_argument("--emit-body", metavar="FILE", help="write a pull-request body here") + ap.add_argument("--hold-if-moved", metavar="REF", + help="write nothing when a submodule that could not be resolved is one REF's head commit moves") + ap.add_argument("--open-pr", metavar="BRANCH", + help="open or update the pull request for BRANCH, with the body in $BODY") + ap.add_argument("--list-repos", metavar="OWNER", help="print OWNER's mounted repositories") + ap.add_argument("--branch-prs", nargs=2, metavar=("BRANCH", "SHA"), + help="print whether BRANCH has an open pull request and one with head SHA") + args = ap.parse_args() + try: + if args.open_pr: + return open_pr(args, os.environ["GH_TOKEN"]) + if args.list_repos: + return list_repos(args.list_repos) + if args.branch_prs: + return report_branch_prs(*args.branch_prs, os.environ["GH_TOKEN"]) + token = os.environ.get("GH_PIN_TOKEN") or git_token() + return resolve(args, token) + except ApiError as error: + print(f"GitHub API: {' '.join(error.args)}", file=sys.stderr) + return 1 + + +def git_token() -> str: + """The gh CLI's token, for a run on a workstation.""" + return subprocess.run(["gh", "auth", "token"], check=True, capture_output=True, text=True).stdout.strip() + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/km/templates/team/bump-mounted-pins.yml b/src/km/templates/team/bump-mounted-pins.yml new file mode 100644 index 0000000..cfb520e --- /dev/null +++ b/src/km/templates/team/bump-mounted-pins.yml @@ -0,0 +1,257 @@ +# Written by `km init --team`. `km upgrade` adds it when missing and replaces it while it is a +# version km shipped, unchanged but for its action pins; once this repo edits it, it is this repo's +# and upgrade leaves it. Actions and the image are pinned by commit and digest; Dependabot moves the +# actions, the image digest is moved by hand. +name: bump-mounted-pins + +# The repositories mounted as submodules (.gitmodules) are pinned to a commit, so a change in one of +# them reaches this brain only when the pin moves. That keeps what the brain reads reproducible and +# every move a reviewed change. This workflow connects the two events: it looks for newer commits +# and opens a pull request, so the move stays reviewed instead of becoming automatic. +# +# Three jobs: mounts reads .gitmodules (a brain without mounts stops there), resolve holds the read +# token for the mounted repositories, publish alone can write here. +# +# Set up once, before the first mount (until then the workflow does nothing): +# 1. A GitHub App with the repository permissions Contents: read and Actions: read, installed on +# the owner of this brain for the mounted repositories (or all). Generate a private key. +# 2. Settings, Environments, New environment `mounted-pins`; Deployment branches: selected +# branches, `main` only. GitHub creates an environment a workflow names on its first run, +# without that restriction, so create it first. +# 3. In that environment, the secrets CI_APP_CLIENT_ID (the app's client ID) and +# CI_APP_PRIVATE_KEY (the private key). +# 4. Settings, Actions, General: "Allow GitHub Actions to create and approve pull requests". +# 5. In the ruleset or branch protection of main: "Require approval of the most recent reviewable +# push", so a commit someone pushes onto the bump branch needs a review by someone else. +# Mounted repositories of another owner are not in the token's scope: next to a mount of this +# brain's owner, a public one still resolves and a private one is reported and fails the run; a +# brain whose mounts all belong to other owners is not handled (mounts finds none and stops). +# Submodule paths must not contain spaces. The runner is ubuntu-latest unless the repository +# variable MOUNTED_PINS_RUNNER names another label. +on: + schedule: + # Weekday mornings, 06:25 UTC. Off the hour on purpose: scheduled runs cluster there and are + # queued longest. + - cron: '25 6 * * 1-5' + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: bump-mounted-pins + cancel-in-progress: false + +env: + BRANCH: chore/bump-mounted-pins + +# In a container job the default shell is sh; the steps use bash arrays and process substitution. +defaults: + run: + shell: bash + +jobs: + mounts: + runs-on: ${{ vars.MOUNTED_PINS_RUNNER || 'ubuntu-latest' }} + container: python:3.12-slim@sha256:4840d514c583a4d3d8cad3940e68dee6f0f6f3635c12fcd0a3d74c21a5b93a54 # 3.12-slim + outputs: + names: ${{ steps.mounted.outputs.names }} + steps: + - name: Install git + run: apt-get update && apt-get install -y --no-install-recommends git + + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + # actions/checkout registers the workspace as a safe directory in a HOME that exists only for + # its own step; later steps run with the real HOME and git then refuses the repository as + # dubiously owned. + - name: Trust the workspace + run: git config --global --add safe.directory "$PWD" + + # The token resolve gets reads only these repositories: the mounted ones of this brain's + # owner. None: no environment, no token, nothing to do. + - name: List the mounted repositories + id: mounted + env: + OWNER: ${{ github.repository_owner }} + run: | + names=$(python3 .github/scripts/bump-mounted-pins.py --list-repos "$OWNER") + [ -n "$names" ] || echo "No repository of $OWNER is mounted (.gitmodules): nothing to move." + echo "names=$names" >> "$GITHUB_OUTPUT" + + resolve: + needs: mounts + if: needs.mounts.outputs.names != '' + runs-on: ${{ vars.MOUNTED_PINS_RUNNER || 'ubuntu-latest' }} + # The app key lives in this environment, which only main may use. A workflow pushed on another + # branch therefore cannot reach the key. + environment: mounted-pins + container: python:3.12-slim@sha256:4840d514c583a4d3d8cad3940e68dee6f0f6f3635c12fcd0a3d74c21a5b93a54 # 3.12-slim + permissions: + contents: read + pull-requests: read # is a pull request of the bump branch open, or was one opened for its head + outputs: + changed: ${{ steps.bump.outputs.changed }} + proposed: ${{ steps.bump.outputs.proposed }} + pins: ${{ steps.bump.outputs.pins }} + message: ${{ steps.bump.outputs.message }} + body: ${{ steps.bump.outputs.body }} + steps: + - name: Install git + run: apt-get update && apt-get install -y --no-install-recommends git + + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Trust the workspace # as in mounts + run: git config --global --add safe.directory "$PWD" + + # Not set up yet (see the top): say so and stop, rather than fail every morning. + - name: Check the app's secrets + id: app + env: + CLIENT_ID: ${{ secrets.CI_APP_CLIENT_ID }} + KEY: ${{ secrets.CI_APP_PRIVATE_KEY }} + run: | + if [ -n "$CLIENT_ID" ] && [ -n "$KEY" ]; then echo "ready=true" >> "$GITHUB_OUTPUT" + else echo "::warning::the environment mounted-pins has no CI_APP_CLIENT_ID and CI_APP_PRIVATE_KEY: no pin moves (see the top of this workflow)" + fi + + # A short-lived installation token of the read-only app. Contents to resolve the commits, + # actions to check that their CI did not fail. The action revokes the token at the end of + # the job. + - name: Get a read token for the mounted repositories + id: app-token + if: steps.app.outputs.ready == 'true' + uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0 + with: + client-id: ${{ secrets.CI_APP_CLIENT_ID }} + private-key: ${{ secrets.CI_APP_PRIVATE_KEY }} + owner: ${{ github.repository_owner }} + repositories: ${{ needs.mounts.outputs.names }} + permission-contents: read + permission-actions: read + + - name: Look for newer commits + id: bump + if: steps.app.outputs.ready == 'true' + env: + # GITHUB_TOKEN is scoped to this repository alone and cannot see the mounted ones. + GH_PIN_TOKEN: ${{ steps.app-token.outputs.token }} + GH_TOKEN: ${{ github.token }} + run: | + # The token reaches git through the environment of these calls only, never in argv. + auth=$(printf 'x-access-token:%s' "$GH_TOKEN" | base64 -w0) + echo "::add-mask::$auth" + authgit() { + GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=http.extraheader \ + GIT_CONFIG_VALUE_0="AUTHORIZATION: basic $auth" git "$@" + } + # The pull request branch as it stands, if there is one. Exit 2 is "no such branch"; any + # other failure fails the job rather than passing for a missing branch. + rc=0 + authgit ls-remote --exit-code --heads origin "refs/heads/$BRANCH" > /dev/null || rc=$? + proposed="" open=false seen=false + if [ "$rc" = 0 ]; then + authgit fetch -q --depth 2 origin "refs/heads/$BRANCH" # with its parent, for --hold-if-moved + proposed=$(git rev-parse FETCH_HEAD) + git update-ref refs/bump/proposed "$proposed" + prs=$(python3 .github/scripts/bump-mounted-pins.py --branch-prs "$BRANCH" "$proposed") + read -r open seen <<< "$prs" + elif [ "$rc" != 2 ]; then + echo "::error::git ls-remote failed with exit $rc"; exit 1 + fi + # A submodule that cannot be resolved stays at main's pin and does not hold back the + # others; the body names it and the last step fails this job. Nothing is carried over + # from the branch, so no pin pushed there by hand is committed under this workflow's + # name. While a pull request is open and its commit moves such a submodule, the run + # writes nothing (--hold-if-moved): the move would be lost over a passing failure and + # come back the next day. One that fails for good (a private mount of another owner, + # say) and that the branch does not move holds nothing back. + hold=() + [ "$open" = true ] && hold=(--hold-if-moved refs/bump/proposed) + rc=0 + python3 .github/scripts/bump-mounted-pins.py --write "${hold[@]}" \ + --emit-message /tmp/pin-message --emit-body /tmp/pin-body || rc=$? + echo "problems=$rc" >> "$GITHUB_OUTPUT" + [ -s /tmp/pin-message ] || { echo "changed=false" >> "$GITHUB_OUTPUT"; exit 0; } + # Compared against the bump branch, not only against main: with a pull request open the + # pins differ from main every day, and committing them again would force-push each + # morning. A pull request closed without merging leaves its branch, so the same pins are + # not proposed again; newer commits are. Only when a pull request was opened for the + # branch head, though: if opening it failed, the next run tries again. And only while that + # head is what publish pushes, one commit on main changing gitlinks only (--branch-prs + # checks it): a branch someone pushed more onto is regenerated. The submodules are + # `ignore = all`, which would hide every gitlink change from a plain diff. + mapfile -t paths < <(git config -f .gitmodules --get-regexp '\.path$' | cut -d' ' -f2) + if [ "$seen" = true ] && + git diff --cached --quiet --ignore-submodules=none refs/bump/proposed -- "${paths[@]}"; then + echo "changed=false" >> "$GITHUB_OUTPUT" + echo "The pull request branch already proposes exactly these pins." + exit 0 + fi + eof="EOF_$(od -An -N8 -tx1 /dev/urandom | tr -d ' \n')" + { + echo "changed=true" + echo "proposed=$proposed" + echo "pins<<$eof"; git ls-files -s -- "${paths[@]}" | awk '{print $2, $4}'; echo "$eof" + echo "message<<$eof"; cat /tmp/pin-message; echo "$eof" + echo "body<<$eof"; cat /tmp/pin-body; echo "$eof" + } >> "$GITHUB_OUTPUT" + + - name: Fail on a submodule that could not be resolved + if: steps.app.outputs.ready == 'true' && steps.bump.outputs.problems != '0' + run: echo "::error::a submodule could not be resolved, see the step log" && exit 1 + + publish: + needs: resolve + if: ${{ !cancelled() && needs.resolve.outputs.changed == 'true' }} + runs-on: ${{ vars.MOUNTED_PINS_RUNNER || 'ubuntu-latest' }} + container: python:3.12-slim@sha256:4840d514c583a4d3d8cad3940e68dee6f0f6f3635c12fcd0a3d74c21a5b93a54 # 3.12-slim + permissions: + contents: write # push the bump branch + pull-requests: write # open or update the pull request + actions: write # dispatch ci.yml on the bump branch + steps: + - name: Install git + run: apt-get update && apt-get install -y --no-install-recommends git + + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Trust the workspace # as in mounts + run: git config --global --add safe.directory "$PWD" + + - name: Commit and push the bump branch + env: + PINS: ${{ needs.resolve.outputs.pins }} + MESSAGE: ${{ needs.resolve.outputs.message }} + PROPOSED: ${{ needs.resolve.outputs.proposed }} + GH_TOKEN: ${{ github.token }} + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git checkout -q -B "$BRANCH" + while read -r sha path; do git update-index --add --cacheinfo "160000,$sha,$path"; done <<< "$PINS" + git commit -q -m "$MESSAGE" + # Force: the branch is regenerated on every run rather than accumulating one commit per + # day, which also drops any commit pushed onto it by hand. The lease holds it to the head + # resolve saw (empty: the branch must not exist), so at least a commit pushed while this + # run is under way is not lost. The token reaches git for this push only. + auth=$(printf 'x-access-token:%s' "$GH_TOKEN" | base64 -w0) + echo "::add-mask::$auth" + authgit() { + GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=http.extraheader \ + GIT_CONFIG_VALUE_0="AUTHORIZATION: basic $auth" git "$@" + } + authgit push -q --force-with-lease="refs/heads/$BRANCH:$PROPOSED" origin "HEAD:refs/heads/$BRANCH" + + - name: Open or update the pull request + env: + GH_TOKEN: ${{ github.token }} + BODY: ${{ needs.resolve.outputs.body }} + run: python3 .github/scripts/bump-mounted-pins.py --open-pr "$BRANCH" diff --git a/src/km/upgrade.py b/src/km/upgrade.py index 79d3ded..152a51a 100644 --- a/src/km/upgrade.py +++ b/src/km/upgrade.py @@ -15,9 +15,12 @@ still runs stays until that workflow is switched to the km command; - a legacy schema.yaml becomes schema.local.yaml, the overlay it already acts as; - a team brain gets the Dependabot config, and the obsolete `team_brain:` line goes. -- A team brain gets the main-guard workflow and the pre-commit hooks that refuse a commit on main - and a push to main, whichever it lacks, and its km ci workflow gets the `workflow_dispatch` - trigger, so a workflow that opens a pull request with GITHUB_TOKEN can run the check on it. +- A team brain gets the main-guard workflow, the bump-mounted-pins workflow and its script, and the + pre-commit hooks that refuse a commit on main and a push to main, whichever it lacks, and its km + ci workflow gets the `workflow_dispatch` trigger, so a workflow that opens a pull request with + GITHUB_TOKEN can run the check on it. The bump-mounted-pins workflow and script are copies of km + code: one that is still a version km shipped (its action pins aside) is replaced by this km's, one + the brain changed is left with a note. - A brain without a .gitattributes gets km's, so text files are LF on every platform; one whose own file sets no line ending gets a note. @@ -36,7 +39,7 @@ import yaml from km import KM_REF -from km.init import ensure_lf, pin_values, template, with_team_hooks +from km.init import TEAM, ensure_lf, pin_values, template, with_team_hooks from km.paths import repo_root, write_text from km.pins import SCHEMA_LINK, hook_rev, movable_refs, read_pins, resolve, set_pins, set_schema_link, version_key, workflow_files @@ -57,6 +60,41 @@ ".github/workflows/staleness.yml": ("team/staleness.yml", {"e9768164f8a03461", "199bf98454e2f922", "fb3fcd7353e99cde"}), ".pre-commit-config.yaml": ("pre-commit-config.template.yaml", {"8ef7f1ea977e2308", "d6819c5f6e97baa1"}), } +# The km files a team brain carries as copies (its workflow's token job must not install km) -> +# shipped_hash of every version km shipped, the current one included (a test checks it): a copy that +# is still one of them is replaced by the current template, an edited one is left with a note. +SHIPPED = { + "team/bump-mounted-pins.yml": {"c72b295b1ba76e65"}, + "team/bump-mounted-pins.py": {"52f798d7ca2f7d32"}, +} +USES = re.compile(r"(?m)^([ \t]*(?:-[ \t]+)?uses:[ \t]*([^@\s]+)@)\S+.*$") # 1: up to the @, 2: the action + + +def shipped_hash(text: str) -> str: + """sha256 prefix of a shipped copy with its action refs masked: the brain's Dependabot moves + those, and a moved pin is not an edit of the brain's own.""" + return hashlib.sha256(USES.sub(r"\1", text).encode()).hexdigest()[:16] + + +def _pinned_version(pin: str) -> tuple[int, ...] | None: + """(7, 0, 1) for ` # v7.0.1`, the version comment Dependabot keeps; None without one.""" + m = re.search(r"#\s*v?(\d+(?:\.\d+)*)\b", pin) + return tuple(int(x) for x in m.group(1).split(".")) if m else None + + +def carry_pins(old: str, new: str) -> str: + """`new` with the pin `old` gives an action where its version comment is newer than km's: the + brain's Dependabot moved it past the template. Otherwise km's pin, which may be newer and which + the new template may need.""" + pins = {m.group(2): m.group(0)[len(m.group(1)):] for m in USES.finditer(old)} + + def pin(m: re.Match) -> str: + ours, theirs = m.group(0)[len(m.group(1)):], pins.get(m.group(2)) + newer = theirs and (a := _pinned_version(theirs)) and (b := _pinned_version(ours)) and a > b + return m.group(1) + (theirs if newer else ours) + return USES.sub(pin, new) + + MARKER = re.compile(r"(?m)^(# Marks a team brain:.*\n)?team_brain:.*\n") @@ -173,11 +211,22 @@ def main() -> int: write_text(local, new) done.append("dropped team_brain from schema.local.yaml (no longer used)") if team: - for tpl, rel in (("team/dependabot.yml", ".github/dependabot.yml"), ("team/main-guard.yml", ".github/workflows/main-guard.yml")): - if not (p := root / rel).exists(): + for tpl in ("team/dependabot.yml", "team/main-guard.yml", *SHIPPED): + new = template(tpl, {}) + if not (p := root / (rel := TEAM[tpl])).exists(): p.parent.mkdir(parents=True, exist_ok=True) - write_text(p, template(tpl, {})) + write_text(p, new) done.append(f"added {rel}") + if tpl == "team/bump-mounted-pins.yml" and (root / ".gitmodules").is_file(): + done.append("this brain has mounts: before the workflow's first run, create the environment mounted-pins " + "restricted to main, or GitHub creates it open to every branch (setup in the workflow's header)") + elif tpl in SHIPPED and (copy := p.read_text(encoding="utf-8")) != new and (h := shipped_hash(copy)) != shipped_hash(new): + if h in SHIPPED[tpl]: + write_text(p, carry_pins(copy, new)) + done.append(f"updated {rel} to this km's version (it was an earlier one, unchanged but for its action pins, keeping those newer than km's)") + else: + done.append(f"left {rel}: the brain changed it, so it gets no later fixes; " + f"compare it with km's src/km/templates/{tpl}") ci = str(Path(".github/workflows/ci.yml")) # as read_pins names it, with the OS's separator if ci in pins and (got := with_dispatch(text := (root / ci).read_text(encoding="utf-8"))) != text: # km's ci workflow if got is None: diff --git a/tests/doctor_smoke.sh b/tests/doctor_smoke.sh index 969af3b..a7f633c 100644 --- a/tests/doctor_smoke.sh +++ b/tests/doctor_smoke.sh @@ -201,6 +201,8 @@ case "$*" in "auth status --hostname github.com") [ -n "$GH_AUTH" ] ;; "api --hostname github.com repos/o/r --jq [.permissions.push, .full_name] | @tsv") printf '%s\tO/R\n' "$GH_PUSH" ;; "api --hostname github.com graphql -f query="*) [ "$GH_FORK" = me ] && echo me/r-1 ;; + "api --hostname github.com repos/O/R/environments/mounted-pins --jq .deployment_branch_policy"*) [ -n "$GH_ENV" ] && echo "$GH_ENV" || exit 1 ;; + "api --hostname github.com repos/O/R/environments/mounted-pins/deployment-branch-policies --jq"*) echo "$GH_BRANCHES" ;; *) exit 2 ;; esac SH @@ -211,8 +213,20 @@ for s, w, f in d.results: print(s, w, "|", f)' 2>&1)"; } GH_LOGIN= ghcheck; out_has "FAIL gh is not signed in" && ok "gh without a login fails with gh auth login" || no "gh login ($out)" GH_LOGIN= GH_AUTH=1 ghcheck; out_has "FAIL gh is signed in but github.com did not answer" && ok "a network error is not taken for a missing login" || no "gh network ($out)" GH_LOGIN=me GH_PUSH=true ghcheck; out_has "ok you can push branches to O/R" && ok "push access is ok" || no "push access ($out)" + out="$(GH_LOGIN=me GH_PUSH=false PATH="$G:$PATH" kmpy -c 'import km.doctor as d; print(d.check_github("https://github.com/o/r.git", False))' 2>&1)" + [ "$out" = O/R ] && ok "the GitHub check names the repository for the checks that follow, also without push access" || no "check_github returns the repo ($out)" GH_LOGIN=me GH_PUSH=false GH_FORK=me ghcheck; out_has "ok you contribute through your fork me/r-1" && ok "a fork is found, also under another name" || no "fork found ($out)" GH_LOGIN=me GH_PUSH=false GH_FORK=other ghcheck; out_has "note you can read O/R but not push to it" && ok "no fork yet is a note" || no "no fork ($out)" + E="$T/envbrain"; mkdir -p "$E/.github/workflows" && touch "$E/.gitmodules" "$E/.github/workflows/bump-mounted-pins.yml" + envcheck(){ out="$(PATH="$G:$PATH" kmpy -c 'import sys, pathlib, km.doctor as d +d.check_mounted_pins(pathlib.Path(sys.argv[1]), "O/R") +for s, w, f in d.results: print(s, w, "|", f)' "$1" 2>&1)"; } + GH_ENV= envcheck "$E"; out_has "note O/R has no environment mounted-pins" && ok "a brain with mounts and the pins workflow but no environment gets a note" || no "missing environment ($out)" + GH_ENV=open envcheck "$E"; out_has "note the environment mounted-pins of O/R is open to every branch" && ok "an environment open to every branch gets a note" || no "open environment ($out)" + GH_ENV=protected envcheck "$E"; out_has "open to every protected branch, not to main alone" && ok "an environment open to protected branches gets a note" || no "protected environment ($out)" + GH_ENV=custom GH_BRANCHES="main release" envcheck "$E"; out_has "open to main release, not to main alone" && ok "an environment open to main and another branch gets a note" || no "custom environment ($out)" + GH_ENV=custom GH_BRANCHES=main envcheck "$E"; out_has "ok O/R has the environment mounted-pins, restricted to main" && ok "an environment restricted to main is ok" || no "restricted environment ($out)" + rm "$E/.gitmodules"; GH_ENV= envcheck "$E"; ! out_has "mounted-pins" && ok "without mounts the environment is not asked for" || no "no mounts, no check ($out)" out="$(PATH="$G:$PATH" kmpy -c 'import km.doctor as d d.check_github("git@github.com:o/r.git", True); d.check_github("https://gitlab.com/o/r.git", False) print(d.results)' 2>&1)" diff --git a/tests/mounted_pins_check.py b/tests/mounted_pins_check.py new file mode 100644 index 0000000..7e9ec14 --- /dev/null +++ b/tests/mounted_pins_check.py @@ -0,0 +1,308 @@ +"""The commit selection and the resolve step of the bump-mounted-pins script a team brain gets +(src/km/templates/team/bump-mounted-pins.py), against a stand-in for the GitHub API. + + python3 tests/mounted_pins_check.py +""" +from __future__ import annotations + +import importlib.util +import os +import re +import subprocess +import sys +import tempfile +import urllib.parse +from pathlib import Path +from types import SimpleNamespace + +REPO = Path(__file__).resolve().parent.parent +spec = importlib.util.spec_from_file_location("bump", REPO / "src/km/templates/team/bump-mounted-pins.py") +bump = importlib.util.module_from_spec(spec) +spec.loader.exec_module(bump) +REAL_API = bump.api # the cases below replace bump.api with stand-ins + +passed = failed = 0 + + +def check(cond: bool, what: str, detail: object = "") -> None: + global passed, failed + if cond: + passed += 1 + print(f" ok: {what}") + else: + failed += 1 + print(f" FAIL: {what} ({detail})") + + +def sha(name: str) -> str: + """A 40-hex sha that shows its name: c1 -> c1c1c1...""" + return (name.encode().hex() * 40)[:40] + + +class Fake: + """One mounted repository: `commits` newest first, each `name` or `name:state`, where state is a + comma list of conclusions, one per workflow, newest run first per workflow written `wf=a/b` + (a the newest), each a push to `main` unless written `conclusion@ref`; no state: no run. + `behind`: commits that are not ahead of the pin. `default`: the repository's default branch.""" + + def __init__(self, commits: list[str], workflows: int = 1, behind: tuple[str, ...] = (), down: bool = False, + default: str = "main"): + self.order, self.runs = [], {} + for c in commits: + name, _, state = c.partition(":") + self.order.append(sha(name)) + runs = [] + for i, part in enumerate(filter(None, state.split(","))): + wf, _, conc = part.rpartition("=") + for x in conc.split("/"): + x, _, ref = x.partition("@") + runs.append({"workflow_id": wf or i, "conclusion": None if x == "running" else x, "head_branch": ref or "main"}) + self.runs[sha(name)] = runs + self.workflows, self.behind, self.down = workflows, {sha(b) for b in behind}, down + self.default = default + self.branches: list[str | None] = [] # the branch each commit listing asked for, as sent + + def api(self, path: str, token: str, method: str = "GET", payload: dict | None = None): + if self.down: + raise bump.ApiError("HTTP 401", "Bad credentials") + if path == "repos/o/r": + return {"default_branch": self.default} + if m := re.match(r"repos/o/r/commits\?(?:sha=([^&]+)&)?per_page=\d+$", path): + self.branches.append(m.group(1)) # None: the default branch + return [{"sha": s} for s in self.order] + if m := re.match(r"repos/o/r/actions/runs\?head_sha=(\w+)(?:&branch=([^&]+))?&event=push", path): + branch = m.group(2) and urllib.parse.unquote(m.group(2)) # no filter: runs of every ref + return {"workflow_runs": [r for r in self.runs.get(m.group(1), []) if branch in (None, r["head_branch"])]} + if path == "repos/o/r/actions/workflows?per_page=1": + return {"total_count": self.workflows} + if m := re.match(r"repos/o/r/compare/(\w+)\.\.\.(\w+)$", path): + return {"status": "behind" if m.group(2) in self.behind else "ahead"} + raise AssertionError(f"unexpected call {path}") + + +def pick(commits: list[str], pin: str | None, **kw) -> str: + """The name of the commit chosen, 'none' when the pin stays, or 'error: ...'.""" + fake = Fake(commits, **kw) + bump.api = fake.api + try: + got, _ = bump.newest_commit("o/r", "main", sha(pin) if pin else None, "t") + except bump.ApiError as e: + return f"error: {e.args[0]}" + names = {sha(c.partition(":")[0]): c.partition(":")[0] for c in commits} + return names.get(got, got) if got else "none" + + +CASES = [ # (what, commits newest first, pin, keyword args, expected) + ("a green tip is taken", ["c2:success", "c1:success", "p:success"], "p", {}, "c2"), + ("a bot tip on a green commit is taken", ["bot", "c1:success", "p:success"], "p", {}, "bot"), + ("a bot tip directly on the pin is taken", ["bot", "p"], "p", {}, "bot"), + ("a bot tip on a failed commit is refused, the green one below the failure is taken", + ["bot", "c2:failure", "c1:success", "p"], "p", {}, "c1"), + ("two bot commits on a failed commit are refused", ["bot2", "bot1", "c1:failure", "p"], "p", {}, "none"), + ("an inner commit below a refused head stays refused", ["c2:failure", "inner", "p"], "p", {}, "none"), + ("below a refused head and its inner commit, an older green commit is taken", + ["c3:failure", "inner", "c1:success", "p"], "p", {}, "c1"), + ("a candidate that is not ahead of the pin is skipped, not moved back", + ["c2:success", "side:success", "p"], "p", {"behind": ("c2",)}, "side"), + ("the newest run of a workflow counts: a success after a failure", ["c1:success/failure", "p"], "p", {}, "c1"), + ("the newest run of a workflow counts: a failure after a success", ["c1:failure/success", "p"], "p", {}, "none"), + ("a cancelled run refuses", ["c1:cancelled", "p"], "p", {}, "none"), + ("a run still in progress refuses", ["c1:running", "p"], "p", {}, "none"), + ("skipped and neutral runs count as green", ["c1:a=skipped,b=neutral", "p"], "p", {}, "c1"), + ("one failed workflow of two refuses", ["c1:a=success,b=failure", "p"], "p", {}, "none"), + ("a pin at the tip is up to date", ["p:success", "c0:success"], "p", {}, "none"), + ("a pin beyond the window, a repository without workflows: its commits are green", + ["c2", "c1"], "old", {"workflows": 0}, "c2"), + ("a pin beyond the window, a repository with workflows and no runs: an error, not a guess", + ["c2", "c1"], "old", {}, "error: no usable commit among the last 2 on main"), + ("a pin beyond the window: a commit without runs inherits the nearest older run", + ["bot", "c1:success"], "old", {}, "bot"), + ("unpinned: the green tip is taken", ["c1:success"], None, {}, "c1"), + ("unpinned, no run anywhere, workflows exist: an error", ["c1"], None, {}, "error: no usable commit among the last 1 on main"), + ("a branch without commits is an error", [], "p", {}, "error: no commits on main"), + ("a green run on a side branch is not CI of the tracked branch", + ["m:running", "f:success@feature", "p:success"], "p", {}, "none"), + ("a green run of a tag push does not hide a failed run of the branch", ["t:w=success@v1/failure", "p:success"], "p", {}, "none"), +] + + +def git(*args: str, cwd: Path) -> str: + return subprocess.run(["git", "-c", "user.name=t", "-c", "user.email=t@t", "-c", "commit.gpgsign=false", *args], + cwd=cwd, check=True, capture_output=True, text=True).stdout.strip() + + +def resolve_cases(tmp: Path) -> None: + """The resolve step in a brain with mounts: `good` moves, `new` (not pinned yet) gets its first pin, + `bad` fails (401), `odd` gets an answer without the expected key, `else` is not on GitHub.""" + old = os.getcwd() + os.environ["GITHUB_REPOSITORY"] = "Own/brain" + try: + empty = tmp / "empty" + empty.mkdir() + os.chdir(empty) + git("init", "-q", cwd=empty) + check(bump.submodules() == [], "without a .gitmodules there is nothing to move", bump.submodules()) + (empty / ".gitmodules").write_text("") + check(bump.submodules() == [], "an empty .gitmodules (the last mount removed) is nothing to move, no crash", bump.submodules()) + + b = tmp / "brain" + b.mkdir() + os.chdir(b) + git("init", "-q", "-b", "main", cwd=b) + (b / ".gitmodules").write_text( + '[submodule "good"]\n\tpath = brains/good\n\turl = ../good.git\n\tignore = all\n' + '[submodule "bad"]\n\tpath = brains/bad\n\turl = git@github.com:own/bad.git\n\tbranch = .\n' + '[submodule "else"]\n\tpath = brains/else\n\turl = https://gitlab.com/x/else.git\n' + '[submodule "new"]\n\tpath = brains/new\n\turl = ../new.git\n' + '[submodule "new-again"]\n\tpath = brains/new-again\n\turl = ../new.git\n' + '[submodule "odd"]\n\tpath = brains/odd\n\turl = ../odd.git\n\tbranch = main\n' + '[submodule "other"]\n\tpath = brains/other\n\turl = ../../them/other.git\n') + for path, pin in (("brains/good", "g0"), ("brains/bad", "b0"), ("brains/odd", "o0")): + git("update-index", "--add", "--cacheinfo", f"160000,{sha(pin)},{path}", cwd=b) + git("add", ".gitmodules", cwd=b) + git("commit", "-q", "-m", "brain", cwd=b) + subs = {s["name"]: s for s in bump.submodules()} + check([subs["good"]["slug"], subs["bad"]["slug"], subs["bad"]["branch"], subs["good"]["branch"], subs["else"]["slug"], subs["other"]["slug"]] + == ["Own/good", "own/bad", "main", None, None, "them/other"], + "slugs: ../x is a sibling under the brain's owner, ../../o/x another owner's, ssh URLs parse, " + "`branch = .` is main, no branch is the default branch, another host has none", subs) + out = subprocess.run([sys.executable, str(REPO / "src/km/templates/team/bump-mounted-pins.py"), "--list-repos", "own"], + cwd=b, capture_output=True, text=True) + check(out.stdout.strip() == "good,bad,new,odd", "--list-repos names the mounts of the owner, in any case", out.stdout + out.stderr) + + (b / ".gitmodules").write_text((b / ".gitmodules").read_text().split('[submodule "other"]')[0]) + git("add", ".gitmodules", cwd=b) + git("commit", "-q", "-m", "drop other", cwd=b) + fakes = {"Own/good": Fake(["g1:w=success@trunk/failure@main", "g0:success@trunk"], default="trunk"), "Own/new": Fake(["n1:success"])} + + def api(path, token, method="GET", payload=None): # bad is refused, odd answers without workflow_runs + slug = "/".join(path.split("/")[1:3]) + if slug == "Own/odd": + return [{"sha": sha("o1")}, {"sha": sha("o0")}] if "/commits?" in path else {} + if slug not in fakes: + raise bump.ApiError("HTTP 401", "Bad credentials") + return fakes[slug].api(path.replace(slug, "o/r", 1), token) + bump.api = api + msg, body = tmp / "msg", tmp / "body" + rc = bump.resolve(SimpleNamespace(write=True, emit_message=str(msg), emit_body=str(body), hold_if_moved=None), "t") + staged = dict(reversed(ln.split()[1:4:2]) for ln in git("ls-files", "-s", "brains", cwd=b).splitlines()) + text = body.read_text(encoding="utf-8") + check(rc == 1, "a mount that cannot be resolved fails the run", rc) + check(staged.get("brains/good") == sha("g1"), "the other mounts still move", staged) + check(staged.get("brains/new") == sha("n1"), "a mount not pinned yet gets its first pin", staged) + check(staged.get("brains/odd") == sha("o0") and "odd: KeyError" in text, "an unexpected answer fails that mount only", text) + check(staged.get("brains/bad") == sha("b0"), "the failed mount stays at main's pin", staged) + check(not re.search(r"[0-9a-f]{40}", text + msg.read_text(encoding="utf-8")) and f"`{sha('g1')[:8]}`" in text, + "the body and the message carry short shas only", text) + check("bad: HTTP 401" in text and "else: no path or no GitHub url" in text, "the body names what did not move", text) + check("--keep-from" not in REPO.joinpath("src/km/templates/team/bump-mounted-pins.py").read_text(encoding="utf-8"), + "nothing is carried over from the pull request branch", "") + failed_diff = subprocess.run(["git", "diff", "--cached", "--quiet", "HEAD", "--", "brains/good"], cwd=b).returncode + none_diff = subprocess.run(["git", "diff", "--cached", "--quiet", "--ignore-submodules=none", "HEAD", "--", "brains/good"], cwd=b).returncode + check(failed_diff == 0 and none_diff == 1, "with `ignore = all` only --ignore-submodules=none sees the moved gitlink", + (failed_diff, none_diff)) + def held(change: str) -> bool: + """Build a bump commit on main changing `change` (path=pin), then main moves bad by hand; + resolve with --hold-if-moved on that commit: True when it wrote nothing.""" + git("reset", "-q", "--hard", "HEAD", cwd=b) + base = git("rev-parse", "HEAD", cwd=b) + path, pin = change.split("=") + git("update-index", "--add", "--cacheinfo", f"160000,{sha(pin)},{path}", cwd=b) + git("commit", "-q", "-m", "bump", cwd=b) + git("update-ref", "refs/bump/proposed", "HEAD", cwd=b) + git("reset", "-q", "--hard", base, cwd=b) + git("update-index", "--cacheinfo", f"160000,{sha('b5')},brains/bad", cwd=b) # by hand on main + git("commit", "-q", "-m", "by hand", cwd=b) + msg.unlink(missing_ok=True) + args = SimpleNamespace(write=True, emit_message=str(msg), emit_body=str(body), hold_if_moved="refs/bump/proposed") + rc = bump.resolve(args, "t") + git("reset", "-q", "--hard", base, cwd=b) + return rc == 1 and not msg.exists() + check(held("brains/bad=b1"), "a failing mount the open pull request moves: nothing is written", "") + check(not held("brains/good=g1"), "a failing mount the pull request does not move (main moved it since) holds nothing back", "") + check(set(fakes["Own/good"].branches) == {None}, "a mount without a branch line lists its default branch, and its runs count", + fakes["Own/good"].branches) + finally: + os.chdir(old) + + +def api_errors() -> None: + """An answer that is not JSON, or a connection dropped mid-read, fails the call as ApiError (one + mount), not with a traceback that stops every mount.""" + import http.client + import io + real = bump.urllib.request.urlopen + for what, answer in (("an answer that is not JSON", lambda *a, **k: io.BytesIO(b"")), + ("a connection dropped mid-read", lambda *a, **k: (_ for _ in ()).throw(http.client.RemoteDisconnected("x")))): + bump.urllib.request.urlopen = answer + try: + REAL_API("repos/o/r", "t") + got = "no error" + except bump.ApiError as e: + got = "ApiError" + except Exception as e: # noqa: BLE001 + got = type(e).__name__ + finally: + bump.urllib.request.urlopen = real + check(got == "ApiError", f"{what} is an ApiError", got) + + +def branch_head_cases(tmp: Path) -> None: + """--branch-prs takes a head a pull request was opened for as current only while it is one + commit on main that changes gitlinks and nothing else; branch names are sent quoted.""" + import contextlib + import io + old, old_repo, old_api = os.getcwd(), os.environ.get("GITHUB_REPOSITORY"), bump.api + os.environ["GITHUB_REPOSITORY"] = "own/brain" + os.chdir(tmp) + (tmp / ".gitmodules").write_text('[submodule "m"]\n\tpath = brains/m\n\turl = ../m.git\n') + try: + for what, ahead, files, want in ( + ("a head that only moves gitlinks is current", 1, ["brains/m"], "true true"), + ("a head that also changes another file is regenerated", 1, ["brains/m", "README.md"], "true false"), + ("a branch with a commit pushed below the head is regenerated", 2, ["brains/m"], "true false"), + ("a compare answer without its files is not taken as gitlinks only", 1, [], "true false")): + calls = [] + + def api(path, token, method="GET", payload=None): + calls.append(path) + if path.startswith("repos/own/brain/pulls?"): + return [{"state": "open", "head": {"sha": "h"}}] + if path == "repos/own/brain/compare/main...h": + return {"ahead_by": ahead, "files": [{"filename": f} for f in files]} + raise AssertionError(path) + bump.api = api + out = io.StringIO() + with contextlib.redirect_stdout(out): + bump.report_branch_prs("chore/bump-mounted-pins", "h", "t") + check(out.getvalue().split() == want.split(), what, out.getvalue()) + check("head=own%3Achore%2Fbump-mounted-pins&" in calls[0], "the branch is sent quoted", calls[0]) + fake = Fake(["c1:success@rel/1.x", "p:success@rel/1.x"]) + bump.api = fake.api + got, _ = bump.newest_commit("o/r", "rel/1.x", sha("p"), "t") + check(got == sha("c1") and fake.branches == ["rel%2F1.x"], "a branch name with a slash is sent quoted and its runs count", + (got, fake.branches)) + finally: + os.chdir(old) + bump.api = old_api + if old_repo is None: + os.environ.pop("GITHUB_REPOSITORY", None) + else: + os.environ["GITHUB_REPOSITORY"] = old_repo + + +def main() -> int: + for what, commits, pin, kw, want in CASES: + got = pick(commits, pin, **kw) + check(got == want, what, f"got {got}, want {want}") + api_errors() + with tempfile.TemporaryDirectory() as tmp: + resolve_cases(Path(tmp)) + with tempfile.TemporaryDirectory() as tmp: + branch_head_cases(Path(tmp)) + print(f"mounted_pins: {passed} passed, {failed} failed") + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/mounted_pins_smoke.sh b/tests/mounted_pins_smoke.sh new file mode 100644 index 0000000..2eb1fd0 --- /dev/null +++ b/tests/mounted_pins_smoke.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash +# Smoke test for the bump-mounted-pins workflow and script a team brain gets: the commit selection +# against a stand-in GitHub API (tests/mounted_pins_check.py), the workflow's shape (which job holds +# which token), and its resolve step run against a local origin with a stand-in script. +# Run: bash tests/mounted_pins_smoke.sh +set -u +source "$(dirname "$0")/lib.sh" +WF="$REPO/src/km/templates/team/bump-mounted-pins.yml" +T="$(mktemp -d)"; trap 'rm -rf "$T"' EXIT + +out="$(kmpy "$REPO/tests/mounted_pins_check.py" 2>&1)"; rc=$? +echo "$out" | grep -E "FAIL|Traceback|Error" ; echo "$out" | tail -1 +[ "$rc" = 0 ] && ok "commit selection and resolve against a stand-in API" || no "mounted_pins_check.py" + +# --- the workflow's shape --- +kmpy - "$WF" <<'PY' && ok "the app token lives in resolve alone, behind the main-only environment; publish alone writes" || no "workflow shape" +import sys, yaml +wf = yaml.safe_load(open(sys.argv[1], encoding="utf-8")) +jobs, text = wf["jobs"], open(sys.argv[1], encoding="utf-8").read() +problems = [] +def need(cond, what): + cond or problems.append(what) +need(wf["defaults"]["run"]["shell"] == "bash", "defaults.run.shell is bash (a container job runs sh otherwise)") +need(wf["permissions"] == {"contents": "read"}, "read-only by default") +for name, job in jobs.items(): + need(job["runs-on"] == "${{ vars.MOUNTED_PINS_RUNNER || 'ubuntu-latest' }}", f"{name}: ubuntu-latest unless the brain names a runner") + need("@sha256:" in job.get("container", ""), f"{name}: the image is pinned by digest") + uses = [s["uses"].split("@")[0] for s in job["steps"] if "uses" in s] + allowed = ["actions/checkout", "actions/create-github-app-token"] if name == "resolve" else ["actions/checkout"] + need(set(uses) <= set(allowed), f"{name}: runs no other action ({uses})") + need(all(s.get("with", {}).get("persist-credentials") is False for s in job["steps"] if s.get("uses", "").startswith("actions/checkout")), + f"{name}: checkout keeps no credentials") + need(not any("pip " in s.get("run", "") for s in job["steps"]), f"{name}: installs no Python package") + secrets = "secrets." in yaml.safe_dump(job) + need(secrets == (name == "resolve"), f"{name}: {'reads' if name == 'resolve' else 'never reads'} the app's secrets") + need(("environment" in job) == (name == "resolve"), f"{name}: environment only where the key is") +need(jobs["resolve"]["environment"] == "mounted-pins", "the key's environment is mounted-pins") +need(jobs["resolve"]["if"] == "needs.mounts.outputs.names != ''", "no mounts: resolve does not run, no environment is created") +token = next(s for s in jobs["resolve"]["steps"] if s.get("uses", "").startswith("actions/create-github-app-token")) +need(token["with"]["owner"] == "${{ github.repository_owner }}", "the token's owner is the brain's owner") +need(token["with"]["permission-contents"] == "read" and token["with"]["permission-actions"] == "read", "a read-only token") +need(jobs["publish"]["permissions"] == {"contents": "write", "pull-requests": "write", "actions": "write"}, "publish permissions") +need(jobs["resolve"]["permissions"] == {"contents": "read", "pull-requests": "read"}, "resolve permissions") +need("--force-with-lease=" in text and "--ignore-submodules=none" in text, "lease push, gitlink diff that sees `ignore = all` mounts") +need("://x-access-token" not in text and "GIT_CONFIG_VALUE_0=" in text, "the token reaches git through the environment, not a URL") +for p in problems: + print(f" {p}") +sys.exit(1 if problems else 0) +PY + +# --- the resolve step against a local origin --- +# A stand-in for the script: --branch-prs prints $PRS ("open seen"), --write sets the gitlink of +# brains/m to $NEW; $FAILED (paths) could not be resolved, and with --hold-if-moved REF it writes +# nothing when REF's commit moves one of them, as the script does. +if [ "${BASH_VERSINFO[0]}" -lt 4 ]; then + echo " skip: the resolve step needs bash 4 or newer (mapfile), this is ${BASH_VERSION}" +else + kmpy -c 'import sys, yaml +step = next(s for s in yaml.safe_load(open(sys.argv[1]))["jobs"]["resolve"]["steps"] if s.get("id") == "bump") +print(step["run"].replace("/tmp/pin-", sys.argv[2] + "/pin-"))' "$WF" "$T" > "$T/step.sh" # files of its own, as in a fresh container + mkdir -p "$T/bin" && printf '#!/bin/sh\n[ "$1" = -w0 ] && shift\nexec %s "$@"\n' "$(command -v base64)" > "$T/bin/base64" && chmod +x "$T/bin/base64" + O="$T/origin.git"; W="$T/work" + git init -q --bare -b main "$O" && git init -q -b main "$W" + mkdir -p "$W/.github/scripts" + printf '[submodule "m"]\n\tpath = brains/m\n\turl = ../m.git\n\tignore = all\n' > "$W/.gitmodules" + cat > "$W/.github/scripts/bump-mounted-pins.py" <<'PY' +import os, subprocess, sys +args = sys.argv[1:] +if args[0] == "--branch-prs": + print(os.environ["PRS"]) + sys.exit(0) +failed = os.environ.get("FAILED", "").split() +ls = lambda ref, p: subprocess.run(["git", "ls-tree", ref, "--", p], capture_output=True, text=True, check=True).stdout +hold = "--hold-if-moved" in args and any(ls(args[args.index("--hold-if-moved") + 1], p) != ls(args[args.index("--hold-if-moved") + 1] + "^", p) + for p in failed) +if os.environ["NEW"] != os.environ["MAIN_PIN"] and not hold: + subprocess.run(["git", "update-index", "--cacheinfo", f"160000,{os.environ['NEW']},brains/m"], check=True) + open(args[args.index("--emit-message") + 1], "w").write("move\n") + open(args[args.index("--emit-body") + 1], "w").write("body\n") +sys.exit(1 if failed else 0) +PY + A="$(printf 'a%.0s' {1..40})"; B="$(printf 'b%.0s' {1..40})"; C="$(printf 'c%.0s' {1..40})" + ( cd "$W" && git update-index --add --cacheinfo "160000,$A,brains/m" && git add .gitmodules .github && git "${G[@]}" commit -q -m main \ + && git remote add origin "$O" && git push -q origin main ) && git clone -q "$O" "$T/side" + branch_with(){ # the bump branch on origin, proposing pin $1; none: no branch. Made in a clone of its + # own, since the step's fetch makes the work tree's repository shallow. + git -C "$O" update-ref -d refs/heads/chore/bump-mounted-pins + [ "$1" = none ] && return + ( cd "$T/side" && git checkout -q -B tmp origin/main && git update-index --cacheinfo "160000,$1,brains/m" && git "${G[@]}" commit -q -m p \ + && git push -q -f origin HEAD:refs/heads/chore/bump-mounted-pins ) + } + step(){ # PRS NEW [FAILED]: run the step, print changed= and problems= + ( cd "$W" && git reset -q --hard main && : > "$T/out" && rm -f "$T"/pin-* \ + && PATH="$T/bin:$PATH" PRS="$1" NEW="$2" FAILED="${3:-}" MAIN_PIN="$A" GITHUB_OUTPUT="$T/out" BRANCH=chore/bump-mounted-pins \ + GH_TOKEN=t GH_PIN_TOKEN=t bash --noprofile --norc -eo pipefail "$T/step.sh" >/dev/null 2>&1 ) || echo "step failed" + grep -o "^changed=.*" "$T/out"; grep -o "^problems=.*" "$T/out" + } + r(){ echo $(step "$@"); } + branch_with none; [ "$(r "false false" "$B")" = "changed=true problems=0" ] && ok "no bump branch: the move is proposed" || no "no branch ($(r "false false" "$B"))" + branch_with "$B"; [ "$(r "true true" "$B")" = "changed=false problems=0" ] && ok "an open pull request already proposing these pins: nothing is pushed" || no "open PR, same pins ($(r "true true" "$B"))" + branch_with "$B"; [ "$(r "true true" "$C")" = "changed=true problems=0" ] && ok "an open pull request with older pins is moved on" || no "open PR, newer pins ($(r "true true" "$C"))" + branch_with "$B"; [ "$(r "true true" "$C" brains/m)" = "changed=false problems=1" ] && ok "a mount that fails while the open pull request moves it: nothing is pushed, the run fails" || no "open PR moving a failing mount ($(r "true true" "$C" brains/m))" + branch_with "$B"; [ "$(r "true true" "$C" brains/x)" = "changed=true problems=1" ] && ok "a mount that fails and the open pull request does not move: the others are still proposed, the run fails" || no "open PR, a failing mount it does not move ($(r "true true" "$C" brains/x))" + branch_with none; [ "$(r "false false" "$B" brains/m)" = "changed=true problems=1" ] && ok "a mount that fails with no pull request open: the others are still proposed, the run fails" || no "no PR, a failing mount ($(r "false false" "$B" brains/m))" + branch_with "$B"; [ "$(r "false false" "$B")" = "changed=true problems=0" ] && ok "a branch whose pull request never opened is pushed again, so the pull request opens" || no "M1 retry ($(r "false false" "$B"))" + branch_with "$B"; [ "$(r "false true" "$B")" = "changed=false problems=0" ] && ok "a pull request closed on these pins is not proposed again" || no "closed PR ($(r "false true" "$B"))" + branch_with none; [ "$(r "false false" "$A")" = "changed=false problems=0" ] && ok "nothing to move: no change" || no "nothing to move ($(r "false false" "$A"))" +fi +summary mounted_pins_smoke diff --git a/tests/team_smoke.sh b/tests/team_smoke.sh index f22dc8c..9fae3bd 100644 --- a/tests/team_smoke.sh +++ b/tests/team_smoke.sh @@ -148,7 +148,7 @@ python3 -c "import json,sys; c=json.load(open('$m'))['counts']; sys.exit(0 if c= G checkout -q -- schema.local.yaml process/cust.md # --- workflows --- -grep -q "scripts/" "$T"/.github/workflows/*.yml && no "workflows call km, not copied scripts" || ok "workflows call km, not copied scripts" +grep -h "scripts/" "$T"/.github/workflows/*.yml | grep -qv "\.github/scripts/bump-mounted-pins\.py" && no "workflows call km, not copied scripts" || ok "workflows call km, not copied scripts" python3 -c "import yaml,glob; [yaml.safe_load(open(f)) for f in glob.glob('$T/.github/workflows/*.yml')]" 2>/dev/null \ && ok "workflows are valid YAML" || no "workflows are valid YAML" python3 - "$T/.github/workflows/staleness.yml" <<'PY' && ok "staleness validates after demoting, even after a failed step" || no "staleness validates after demoting" @@ -383,7 +383,9 @@ def pins(paths): return out own = pins([f"{repo}/action.yml", *glob.glob(f"{repo}/.github/workflows/*.yml")]) tpl = pins(glob.glob(f"{repo}/src/km/templates/team/*.yml")) -sys.exit(0 if all(name in own and shas == own[name] for name, shas in tpl.items()) else 1) +# Actions only a template uses: no Dependabot here sees them; each brain's own Dependabot moves them. +TEMPLATE_ONLY = {"actions/create-github-app-token"} +sys.exit(0 if all(shas == own[name] if name in own else name in TEMPLATE_ONLY for name, shas in tpl.items()) else 1) PY # --- non-ASCII file names on a system whose encoding is not UTF-8 (Windows, cp1252) --- @@ -458,6 +460,55 @@ printf 'on:\n pull_request:\njobs:\n v:\n steps:\n - run: make\n' > "$ out="$(km upgrade --root "$D" 2>&1)" [ "$(on_keys "$D/.github/workflows/ci.yml")" = "pull_request" ] && ok "a ci.yml that does not install km gets no trigger" || no "own ci.yml gets no trigger ($out)" +# --- the workflow that moves the mounted pins, and its script --- +PW=.github/workflows/bump-mounted-pins.yml; PS=.github/scripts/bump-mounted-pins.py +{ cmp -s "$T/$PW" "$REPO/src/km/templates/team/bump-mounted-pins.yml" && cmp -s "$T/$PS" "$REPO/src/km/templates/team/bump-mounted-pins.py"; } \ + && ok "a team brain gets the mounted-pins workflow and its script" || no "team brain mounted-pins files" +{ [ -f "$O/$PW" ] && [ -f "$O/$PS" ]; } && ok "upgrade adds them to a team brain without them" || no "upgrade adds the mounted-pins files" +[ ! -e "$P/$PW" ] && [ ! -e "$P/$PS" ] && ok "a personal brain does not get them" || no "personal brain has no mounted-pins files" +kmpy - <<'PY' && ok "SHIPPED lists the current mounted-pins templates" || no "a mounted-pins template changed: add its hash to SHIPPED in upgrade.py" +import sys, km.upgrade as u +from km.init import template +sys.exit(any(u.shipped_hash(template(t, {})) not in h for t, h in u.SHIPPED.items())) +PY +kmpy -c 'import km.upgrade as u +h = u.shipped_hash +assert h(" - uses: a/b@1 # v1\n") == h(" - uses: a/b@2 # v2\n") and h(" uses: \"a/b@1\"\n") == h(" uses: \"a/b@2\"\n") +assert h(" - uses: a/b@1\n") != h(" - uses: a/c@1\n")' && ok "an action pin and its comment do not count as an edit, the action does" || no "shipped_hash masking" +kmpy -c 'import km.upgrade as u +new = " - uses: a/b@k # v2.0.0\n - uses: a/c@k # v2.0.0\n" +got = u.carry_pins(" - uses: a/b@n # v2.1.0\n - uses: a/c@o # v1.9.9\n", new) +assert got == " - uses: a/b@n # v2.1.0\n - uses: a/c@k # v2.0.0\n", got' \ + && ok "an upgrade keeps an action pin the brain moved past the template, and takes the template pin where that is newer" || no "carry_pins direction" +# an earlier version km shipped is replaced, also after the brain's Dependabot moved its action +# pins: a stand-in for one (a line ahead of the template), registered as shipped for this run +MOVE='s/\(uses: actions\/checkout@\)[^ ]*.*/\1aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa # v99/' +for f in "$PW:bump-mounted-pins.yml" "$PS:bump-mounted-pins.py"; do + { echo "# an earlier km version"; sed "$MOVE" "$REPO/src/km/templates/team/${f#*:}"; } > "$O/${f%%:*}" +done +upgrade_older(){ out="$(cd "$O" && kmpy - 2>&1 <<'PY' +import sys, km.upgrade as u +from km.init import template +for t, h in u.SHIPPED.items(): + h.add(u.shipped_hash("# an earlier km version\n" + template(t, {}))) +sys.argv = ["km upgrade"] +sys.exit(u.main()) +PY +)"; } +upgrade_older +{ cmp -s "$O/$PW" <(sed "$MOVE" "$REPO/src/km/templates/team/bump-mounted-pins.yml") && cmp -s "$O/$PS" "$REPO/src/km/templates/team/bump-mounted-pins.py" \ + && out_has "updated $PW to this km's version" && out_has "updated $PS to this km's version"; } \ + && ok "upgrade replaces a copy that is an earlier version km shipped, keeping the action pins its Dependabot moved" || no "upgrade replaces an unchanged earlier copy ($out)" +sed "$MOVE" "$O/$PW" > "$F/moved.yml" && mv "$F/moved.yml" "$O/$PW"; cp "$O/$PW" "$F/moved.yml"; upgrade_older +{ cmp -s "$O/$PW" "$F/moved.yml" && ! out_has "mounted-pins"; } \ + && ok "the current version with pins the brain's Dependabot moved is left as it is" || no "current copy with moved pins ($out)" +echo "# the brain's edit" >> "$O/$PW"; echo "# the brain's edit" >> "$O/$PS"; out="$(km upgrade --root "$O" 2>&1)" +{ tail -1 "$O/$PW" | grep -q "the brain's edit" && tail -1 "$O/$PS" | grep -q "the brain's edit" \ + && out_has "left $PW: the brain changed it" && out_has "left $PS: the brain changed it"; } \ + && ok "upgrade leaves an edited copy, with a note" || no "upgrade keeps edited mounted-pins files ($out)" +rm "$O/$PW"; touch "$O/.gitmodules"; out="$(km upgrade --root "$O" 2>&1)"; rm "$O/.gitmodules" +out_has "create the environment mounted-pins restricted to main" && ok "upgrade adding the workflow to a brain with mounts says to restrict the environment first" || no "environment note on upgrade ($out)" + # --- main guard: the hook that refuses a commit on main, and the workflow that flags a direct push --- hooks_of(){ kmpy -c 'import sys, yaml print(" ".join(r["repo"].rsplit("/", 1)[-1] + ":" + h["id"] + ":" + ",".join(h.get("args", []))