Skip to content

ci: persist golangci-lint analysis cache between runs - #11505

Merged
Roasbeef merged 1 commit into
masterfrom
ci-persist-lint-cache
Nov 5, 2026
Merged

Roasbeef merged 1 commit into
masterfrom
ci-persist-lint-cache

Conversation

@Roasbeef

@Roasbeef Roasbeef commented Nov 5, 2026

Copy link
Copy Markdown
Member

Problem

The lint job re-analyzes the entire repository on every run because nothing persists golangci-lint's per-file analysis cache between runs. The lint and lint submodules steps bind mount ~/.cache/golangci-lint into the tools container as /root/.cache/golangci-lint, but that directory is never saved or restored, so every job starts cold. The log confirms it: every run ends with File cache stats: 0 entries of total size 0B, and the main-repo lint pass takes ~8 minutes (measured in job 126979734038, 01:46:55 to 01:55:12).

The Go module cache and build cache are already restored through the setup-go composite action, so dependency downloads aren't the issue here. The missing piece is the analysis cache that lets golangci-lint skip packages whose sources haven't changed. This PR is separate from #11504, which caches the compiled linter binary itself (a ~3.6 minute saving); this one addresses the ~8 minute analysis pass.

Solution

In this PR, we add an actions/cache step to the lint job that restores and saves ~/.cache/golangci-lint.

The cache is keyed as follows:

  • The exact key is lint-analysis-<os>-go-<version>-<config/tools hash>-<commit SHA>. The commit SHA makes the key unique per run so every run saves a fresh snapshot (a static key would save once and then never update).
  • The restore-key prefix drops the SHA: lint-analysis-<os>-go-<version>-<config/tools hash>-. A source-only change therefore restores the most recent previous snapshot and only re-analyzes the changed packages.
  • The config/tools hash covers .golangci.yml, tools/go.mod, tools/go.sum, tools/.custom-gcl.yml, and tools/linters/**/*.go. Any change to the lint config, the pinned linter version, its dependencies, or the custom linter plugins falls back to a clean analysis rather than restoring results produced by a different linter.

The existing cache cleanup job (deletes caches older than 12 hours) bounds how many snapshots accumulate, and GitHub's 10GB LRU eviction applies on top of that.

Expected impact

Locally, running the same image the CI lint job uses (lnd-tools), a warm analysis cache cut the main-repo lint pass from 1m36s to 1m22s on a warm tree with Docker overhead dominating. The CI saving is larger because the CI pass is cold-typechecking everything with GOGC=50, which is what makes the cold 8 minutes. The exact CI saving can't be measured without a real runner round-trip; the expected effect is that packages whose sources didn't change are skipped.

Verification

  • actionlint v1.7.12 on .github/workflows/main.yml: clean.
  • YAML parse of the workflow: OK; the lint job now lists the golangci-lint analysis cache step before the lint steps.
  • git diff --check: clean.
  • scripts/check-release-notes.sh with the PR ref: passes.
  • Local cache behavior verified: running custom-gcl cache status against the bind-mounted directory after a lint pass shows a populated cache (Size: 4.4MiB), and a second run is faster than the first.
  • Not verified: a real GHA cache restore on a runner. The first run after this merges populates the cache; the second run should show Cache restored from key: lint-analysis-... and a File cache stats line with nonzero entries in the lint log.

Out of scope

  • The 12 submodule lints still run as 12 sequential docker run invocations. They share the same analysis cache directory, so submodule packages reused from the main pass benefit too, but a parallel fan-out or single-pass lint is a separate change.
  • The GOGC=50 setting and the runner disk-cleanup step (~3 minutes) are untouched; whether they belong in the lint job is a separate question.


## Tooling and Documentation

* CI now [persists `golangci-lint`'s per-file analysis

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't think we need release notes for this kind of changes

@github-actions github-actions Bot added the severity-low Best-effort review label Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

🟢 PR Severity: LOW

CI | 2 files | 33 lines changed

🟢 Low (2 files)
  • .github/workflows/main.yml - CI/CD workflow configuration change (adds a golangci-lint analysis cache step)
  • docs/release-notes/unreleased/11505.md - release notes entry

Analysis

This PR only touches CI/CD workflow configuration (.github/workflows/main.yml) and a release-notes markdown file. No Go source files, no core packages, and no database migrations are involved. The change adds an actions/cache step to persist the golangci-lint analysis cache between runs, which is a build/tooling optimization with no runtime or protocol impact. File count (2) and line count (33) are both well under the bump thresholds.


To override, add a severity-override-{critical,high,medium,low} label.

@Roasbeef

Roasbeef commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Addressed the review feedback in c99498b:

  • Switched to actions/cache/restore@v4 for the restore step, and a separate actions/cache/save@v4 gated on inputs.tier == 'nightly'. Only the nightly master run writes a snapshot; every other run restores only. This avoids filling the repo's 10GB cache space with snapshots no other PR can read (a PR's cache is readable only by that same PR and by master).
  • Removed the three stray blank lines.
  • The save gate uses the workflow_call tier input rather than github.event_name == 'schedule', because main.yml runs under workflow_call when the nightly workflow calls it, and never sees schedule directly. That meant adding the workflow_call trigger and tier input to main.yml, the same hunk build: cache custom linter build in CI #11504 adds; whichever merges second needs a trivial rebase.
  • Kept the SHA in the key: with saves now nightly-only, each nightly run saves one new snapshot keyed by that master commit, and the restore-key prefix picks up the most recent one. Nightly snapshots accumulate slowly (one per day) and the 12-hour cache cleanup job plus GitHub's LRU eviction bound them.

Verification: actionlint clean (with the workflow_call input defined), YAML parses, check-release-notes.sh passes for PR 11505, git diff --check clean.

@Roasbeef

Roasbeef commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Correction to the previous commit (509f1c3): the nightly-only save gate was wrong for this branch's base. The tier/workflow_call input only exists in the unmerged #11489 stack; on master no nightly caller exists, so the gate would have left the analysis cache unsaved by anyone. Per-PR saves also have real reuse value: later pushes to the same PR restore them.

Now: single actions/cache@v4 step, restore+save in every run, keyed only on the config/tools hash (lint config, pinned linter version, custom plugin sources). No commit SHA in the key, so there's exactly one snapshot per configuration variant that later runs overwrite instead of one snapshot per commit. golangci-lint's own cache metadata determines which entries are valid for the current sources, so stale entries are re-analyzed rather than trusted. The 12-hour cache cleanup job still bounds total cache age. No workflow_call hunk remains, so the rebase concern against #11489 is gone.

@Roasbeef

Roasbeef commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Correction to my previous comment (and to 509f1c3): the claim that actions/cache lets later runs overwrite the config-hash-only key was wrong. actions/cache never overwrites an existing key, so that key would have saved the first snapshot and then restored the same frozen snapshot forever.

Fixed in 7140830: the exact key now ends in the run id (github.run_id), so every run saves a fresh snapshot, and the restore-keys prefix keeps the config/tools hash so a run restores the newest snapshot produced with the same linter configuration. Snapshots are a few MiB each and the 12-hour cache cleanup job bounds the accumulation.

@Roasbeef
Roasbeef force-pushed the ci-persist-lint-cache branch from 7140830 to bf3feb6 Compare November 5, 2026 02:58
@Roasbeef

Roasbeef commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Rebased onto current master and changed in bf3feb6:

  • Split into actions/cache/restore@v4 (every run) and actions/cache/save@v4 gated on inputs.tier == 'nightly'. Only the nightly master run writes a new analysis snapshot; PR and merge queue runs read it.
  • The nightly run also lints master, so its snapshot covers the sources most PRs share.
  • The restore key still ends in the run id, and the restore-keys prefix is the config/tools hash, so a run restores the newest snapshot produced with the same linter configuration. Snapshots accumulate one per nightly run, not one per PR push.
  • Also rebased on top of the tier stack that merged today, so inputs.tier and the nightly caller exist on this branch's base.

To verify after the nightly seeds the cache: a PR's lint log should show Cache restored from key: lint-analysis-..., a nonzero File cache stats count, and a shorter main lint pass. The post-job save step runs as the same runner user that owns the mounted directory, and the nightly run will confirm its size in the log.

@Roasbeef

Roasbeef commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Fixed in b1b597a: the save step now runs after lint submodules, and its condition is ${{ !cancelled() && inputs.tier == 'nightly' }}, so a failed lint step still keeps the analysis it did produce. You're right about the nightly too: it runs master's workflow, so the first cache write for this PR only happens on the first nightly run after this merges. The earliest real check is a manual workflow_dispatch of nightly on master after merge, then watching the next PR lint job for Cache restored from key: lint-analysis-... and a nonzero File cache stats.

In this commit, we persist golangci-lint's per-file analysis cache in
the GitHub Actions cache store so the lint job stops re-typechecking the
whole repository from scratch on every run.

The lint targets bind mount ~/.cache/golangci-lint into the tools
container, but nothing persisted that directory between runs, so each job
started with an empty analysis cache ("File cache stats: 0 entries of
total size 0B" in the logs) and re-analyzed every package.

Every run restores the most recent snapshot, keyed by the lint config,
the tools module and the custom linter plugin sources, so a run never
reuses analysis results produced by a different linter. Only the nightly
master run saves a new snapshot: entries created by PR and merge queue
runs are only readable by that same PR and by master, and the repo's
shared cache space is already at its size limit. The save runs after
both lint steps and also when they failed, so the analysis that did run
isn't lost. A cancelled run saves nothing.
@Roasbeef
Roasbeef force-pushed the ci-persist-lint-cache branch from b1b597a to 0d5172b Compare November 5, 2026 03:20
@Roasbeef

Roasbeef commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Rebased onto master (now includes #11504) in 0d5172b. The branch is a single commit again: the restore step sits after the Buildx setup from #11504, and the nightly-gated save sits after lint submodules with ${{ !cancelled() && inputs.tier == 'nightly' }}. The conflict with #11504's lint-job steps is resolved; CI is running on the new head.

@Roasbeef
Roasbeef merged commit 7669a85 into master Nov 5, 2026
11 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

severity-low Best-effort review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants