Skip to content

perf: do not fetch the repo index when --skip-reindex is set - #7

Merged
vaclav-dvorak merged 1 commit into
exa-masterfrom
vd/20261006/skip-index-fetch-on-skip-reindex
Oct 6, 2026
Merged

vaclav-dvorak merged 1 commit into
exa-masterfrom
vd/20261006/skip-index-fetch-on-skip-reindex

Conversation

@vaclav-dvorak

Copy link
Copy Markdown

Summary

  • push no longer downloads the repo index when --skip-reindex is set. Bumps plugin.yaml to 0.1.1.

Details

FetchRaw sat outside the if !act.skipReindex guard, and the bytes it returns are read only inside that guard:

b, err := storage.FetchRaw(ctx, repoEntry.IndexURL())   // always ran
if err != nil { ... }
if !act.skipReindex {
    idx := helmutil.NewIndex()
    if err := idx.UnmarshalBinary(b); err != nil { ... } // only reader of b
    ...
} else {
    act.printer.Printf("[DEBUG] Skipping reindex ...")
}

So with --skip-reindex every push did a full s3manager download of index.yaml and threw it away.

Measured on the prod chart repo (exa-helm-charts-prod-c42d): index.yaml is 22.3 MB (41,987 chart versions across 130 charts). v2-helm-charts-s3-prod pushes 129 charts sequentially with --skip-reindex --force, so one build spent ~2.8 GB on GETs it never read. From that build's own log:

step time
helm package ×129 12s
helm s3 push ×129 440s
helm s3 reindex + index re-stamp 22s

Per-push gaps were tightly clustered (median 4.0s, min 2s, max 5s) — the signature of a fixed-size download, not of the chart uploads, which vary a lot in size. It also explains a steady slowdown with no step change: median build time went from ~240–330s in late August to ~460–490s now, tracking index growth (16.6 MB → 22.3 MB in four days) while chart count only went 111 → 130.

The comment above the fetch — "The gap between index fetching and uploading should be as small as possible to make the best effort to avoid race conditions" — is about the index read-modify-write, which does not happen on the skip-reindex path. It moves with the code it describes.

Unit tests and go vet are unchanged from the base commit (internal/awsutil's TestDynamicBucketRegion fails identically before and after — it reads the ambient AWS region). The minio-backed e2e suite could not run here: the minio/minio image pull is denied without a Docker Hub login.

FetchRaw sat outside the skipReindex guard, so every push downloaded the whole
index.yaml and discarded it. On the prod chart repo that index is 22 MB and the
build pushes 129 charts in sequence, so a run spent ~440s of its ~478s build
phase on 2.8 GB of GETs it never read. The race the surrounding comment guards
is about the index read-modify-write, which does not happen on this path.
@vaclav-dvorak
vaclav-dvorak merged commit 69ac764 into exa-master Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant