feat(nvsnap): download each model once per cluster into a shared volume for Helm functions

[balajinvda]

Why

Customers deploy Helm charts of N identical GPU workers, and every worker
downloads and compiles on its own. The gate-and-promote election in #2104
cannot serve multi-node instances (LWS, StatefulSet groups, Dynamo
podgangs): holding any member deadlocks the group. The invariant we want
is simpler and storage-independent: the model is downloaded once per
cluster and every pod attaches it; compile caches are shared the same
way. Design and the decisions taken with the product owner:
docs/proposals/helm-shared-model-volume.md.

What changed

One model-volume lifecycle on every storage. The primary volume lives in
the nvsnap namespace; completion is a labelled, retained PV and a released
claim; each function namespace gets a per-namespace view of it (read-only
for readers, read-write for the download Job). The storage mode decides
only two things: the claim's access mode and how the download reaches
the primary (block: staged into a pod-local emptyDir and copied by the
agent, sized from the measured bytes; shared filesystem: the Job writes
through its namespace's read-write view while readers already hold their
views). Admission never waits for a volume to bind.

• internal/modelid: identity and landing volume for any chart. Identity
  as a URI (hf://, ngc://, s3://, nim://) from engine args
  (including positional vllm serve <x> and $VAR expansion), engine
  env, download init containers (NGC CLI, hf download, S3, KServe
  storage-initializer), NIM images, and LeaderWorkerSet workers via the
  leader template. Landing volume kind decides substitutability; a
  customer's shared PVC, hostPath or image is left alone.
• internal/modelvolume + webhook: one download Job per identity per
  cluster (nvsnap-model-dl-<key>, idempotent create, so no election),
  built from the chart's own download init wrapped to touch a marker, or
  from hf download on the engine image with credentials forwarded.
  Every workload pod is a reader: its download init becomes a wait for
  the marker (deadline, then self-download), the engine is started
  offline when it used to download itself. Charts whose engine script
  fetches an artifact nvsnap has no recipe for (an NGC model pulled in
  the main container) keep their download; the agent on that node
  captures the finished tree after Ready and later pods read the copy.
  Every container the feature creates gets requests and limits, dropped
  capabilities, no privilege escalation and the chart's own posture, so
  enforced Kyverno baselines pass.
• Views are sized from the bytes the download measured. The Job reports
  the tree size in its termination message (kubelet unmounts a finished
  Job pod's volumes at once), the staged copy and the capture measure the
  same number, and the agent stamps it on the primary; a 62 GB model on a
  4Ti nominal filesystem claim gets 62Gi views.
• Compile-cache volume, same shape: per configuration key and rank, the
  first Ready group is collected (startup plans, kernel and inductor
  caches, the transformers modules cache), later pods seed their local
  cachedir from it. Compile caches stay in a local cachedir on every
  storage; the model view is read-only. Capture-source pods collect a set
  too, so the first deployment hands the second one its startup plan.
  Credential paths under HOME (NGC CLI config, Hub tokens, and the like)
  never travel in a set.
• Failure handling on the shared-filesystem Job path: a fresh filesystem
  root is root-owned and the Job runs as the function's user (uid 1000
  for Dynamo and NIM images), so the agent on the Job's node opens the
  primary root through the Job pod's kubelet mount and the writer waits
  until its landing is writable. A failed Job is recorded with its reason
  and the download container's last words, the writer view is retired,
  the primary claim is kept for the retry, and a failure marker next to
  the completion marker sends readers already waiting to their fallback
  at once instead of at their deadline. A successful retry clears the
  record.
• Reaper on every agent (agent.modelVolume.reapInterval, default 10m)
  for model and cache volumes alike: unreferenced views are retired with
  their holders, view PVs whose claim is gone are removed, incomplete
  primaries released for 15 min are removed with their storage; complete
  primaries are kept.
• nvsnap.io/cache-salt folds an operator-chosen value into the capture
  and cache identity for changes the identity cannot see (a wheel
  installed at start, a floating tag re-pointed).
• The blob store tier is removed (binary, package, agent uploader, server
  endpoints, chart component, build targets); no current path used it.
• Chart: agent.modelVolume.{enabled,waitDeadline,reapInterval}, storage
  profile modelVolume: {mode, storageClass, size, minSize, readerMode}
  (block by default on shared-volume strategies, rwx must be declared),
  agent.runtime with the runtime socket mounted at its host path for
  CRI-O, a fully qualified smoke-hook image for short-name enforcement,
  and smoke-hook and rollout timeouts that fit a serial agent roll on
  large clusters. The agent build fails loudly when the base image is
  missing from the target registry.

Customer Release Notes

Helm chart functions download each model once per cluster; every other
worker, later version, scale-up and redeploy attaches the downloaded
volume, and compile caches are shared between pods of one configuration.
No chart change. Requires NVMesh or a shared filesystem class such as
OCI File Storage.

Plan Summary

Helm: new values (off by default), one extra Bidirectional hostPath on
the agent DaemonSet, ClusterRole verbs (PVC update/patch, pods patch,
namespaces get, PV create/update/delete). Primaries and the agents' copy
holders live in the release namespace; views live in function
namespaces. No new controllers or CRDs.

Usage

helm upgrade nvsnap deploy/helm/nvsnap --set agent.modelVolume.enabled=true
kubectl get jobs,pvc -A -l nvsnap.io/model
kubectl get pv -l nvsnap.io/model-complete=true
kubectl get pods -A -l nvsnap.io/model-pending=true

Testing

NVMesh (block mode), stock vllm-workers chart, Qwen2.5-32B TP=4, two
replicas: first deploy one Job, 65 GB downloaded once, both Ready +591 s
with 0 downloads in the pods; reinstall both Ready +136 s; cold 325 s.
PVC reader mode, Qwen2.5-14B: first deploy both Ready +344 s, reinstall
+118 s. Real NVCF Helm function with the chart's own NGC download
(954 MB): claim sized at 2Gi from the measured bytes instead of 512Gi,
both Ready +230 s; compile-cache seeding took torch.compile from 23 s to
3 s per pod.

OCI File Storage (shared-filesystem mode), GB300 arm64 nodes, CRI-O,
NVCF storage class, 2026-10-02 and 2026-10-03:

• Qwen2.5-32B vLLM, 62 GB: cold 6 min 37 s (Job writes through the
  writer view, cache set collected after Ready); warm 1 min 38 s;
  scaling 1 to 4 replicas on fresh nodes, each Ready 72 to 78 s after
  creation reading weights at about 2.3 GB/s per node from one
  filesystem, no downloads, no attach wait.
• kimi-k3, 1.56 TB, two pods TP 8, engine-script download captured after
  Ready (image pulls excluded): cold admission to Ready 46 min 30 s
  (download 31 min, capture into the primary 24 min in the background);
  second deployment, unseeded, 20 min 42 s; third deployment, seeded,
  12 min 20 s (weights 425 s from the filesystem view, profiling 0 s
  with the saved plan, graph capture 80 s, no restarts). Against the
  same chart on NVMesh: 14 min 21 s warm.
• Dynamo function with an HF model as a non-root user on a fresh
  filesystem primary: this is the run that found the root-permission and
  failed-Job gaps fixed above; the fixes are covered by unit tests and
  the function redeploy after the next agent roll is the e2e check.

Unit: identity forms and classifier; Job spec derivation and
idempotence; webhook matrix (block and shared-filesystem readers,
complete-at-admission minting, the same flow asserted on every storage,
plain pods keep every annotation, Job volume carry-over, hardening and
posture inheritance, capture sources, injected download, customer
volumes left alone, writer and reader scripts); controller completion,
release, detach gate, per-namespace view minting, bytes recording from
every fill path, capture from an engine download, root opening on the
Job's node, failed Job handling and the failure marker, cache-set
collection with credential exclusion, reaper rules. go test ./...
green.

Notes

Stacked on #2104 (base nvsnap/helm-election). Measured on the way and
recorded in the design doc: the CuTeDSL warmup cannot be cached from
outside the engine (cute.compile() forces no cache); CUDA graph capture
is not cacheable; on both NVMesh and NFS the weight read sits at about
2 GB/s per node against 4.4 GB/s from local NVMe, which points at a
node-local tier under the shared primary as the next step. The first
start of a multi-worker engine can race in transformers' dynamic-module
copy; seeded starts avoid it because the modules cache arrives populated.

References

Relates to #2099

Related Pull Requests

#2104, #2101

Dependencies

None

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/nvcf/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: de7d48ad-9e18-4cc0-a042-6508dbe8383d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[FamousDirector]

Reviewed the incremental change against declared base nvsnap/helm-election (7642bfc). Four P1 availability/data-integrity blockers are detailed inline: cross-namespace concurrent writers, releasing the writer claim after a failed completion update, fallback downloads into a read-only reader view, and writes escaping a partially extracted rank directory on retry.

Validation: existing modelid/modelvolume tests passed; webhook, hostlibs, and tarstream tests passed with a temporary Darwin portability overlay for unchanged treecopy code. Focused checkpointstore shared-volume/view and rootfsonly composer tests passed. Five isolated regression probes reproduced the four findings. Linux agent tests cross-compiled successfully; Helm lint and incremental diff whitespace checks passed. Full checkpointstore/rootfsonly filesystem tests could not run on macOS because the unchanged Linux sendfile path panics there. No live GPU/CSI/cluster validation was performed. All probes used temporary overlays; tracked files remain unchanged.

[FamousDirector]

[P1] Elect the download owner across namespaces

When two Helm functions in different namespaces first request the same incomplete model URI, each namespace creates its own Job here. ensureDownload reuses the cluster-wide primary but gives both Jobs read-write views of the same CSI handle. A fake-client admission probe produced two Jobs sharing that handle. Running the generated writer scripts concurrently also left the completion marker present while the slower writer truncated a model file: readers can start against partially overwritten weights after the first Job reports completion. This breaks both the once-per-cluster requirement and model integrity.

Acquire one atomic, cluster-wide owner for the model identity before creating the namespaced Job. Only that owner should download/write; other namespaces should remain readers, with ownership released or recovered on failure.

[balajinvda]

Fixed in 8e09512. The webhook now claims a cluster-wide download owner before creating any Job: a Lease in the nvsnap namespace whose create is atomic, so concurrent admissions agree on one owning namespace. Other namespaces create no Job and no writer view and read the primary like any other reader. An owner whose Job is gone past a 2 minute grace period is retired with UID and resourceVersion preconditions and the download claimed again; a recorded failure or the completion releases the record. I reused the Lease-create primitive rather than LeaseElector itself, because nvsnap-server lists and reconciles the capture-election Leases. Tests: TestClaimDownload_OneOwnerAcrossNamespaces (concurrent claims from two namespaces, grace period, stale takeover, release on failure) and TestModelVolume_SharedFilesystem_OneWriterAcrossNamespaces (two namespaces admitted, one Job, no second writer view; it failed with 2 Jobs before the fix).

[FamousDirector]

[P1] Persist completion and Retain before releasing the writer claim

A PV resource-version conflict is ignored here, so the following PVC deletion runs even though neither the completion labels nor Retain were persisted. This can occur when a CSI controller or another agent updates the PV between Get and Update. With a dynamically provisioned block PV whose reclaim policy is Delete, releasing the last writer claim then allows the completed model volume to be destroyed. The injected-conflict probe confirmed this method returns nil and deletes the claim while the PV remains incomplete with reclaim policy Delete; the completed artifact is also no longer discoverable through Lookup.

Retry conflicts against a freshly fetched PV, and delete the writer claim only after the required completion metadata and Retain policy have successfully persisted, or a reread confirms them.

[balajinvda]

Fixed in a1060fd. MarkCompleteClaim now retries PV update conflicts against a freshly read PV (bounded at 5 attempts) and deletes the writer claim only once the stored PV is labelled complete and set to Retain. Persistent conflicts return an error and leave the claim in place. Test: TestMarkComplete_ConflictKeepsClaimUntilPersisted, with one injected conflict (retried, then released) and permanent conflicts (error, claim kept, PV still Delete). Both cases failed before the fix.

[FamousDirector]

[P1] Give failed readers a writable fallback landing

For PVC readers admitted while an RWX download is in flight, readerLanding replaces the original landing with a CSI view whose PV has CSI.ReadOnly=true. This script then runs the original download against that same landing when the failure marker appears or the wait deadline expires. Any fallback that creates model files fails on the read-only filesystem, leaving the function in an init-container restart loop rather than recovering. A generated-script probe with a failed marker and a non-writable landing confirmed the write fails. Recording the shared failure only bypasses mutation for later admissions; already admitted readers retain this mount.

Route fallback into a writable per-pod landing that the engine also consumes, or recreate the affected readers on their original download path after recording the failure. Reusing the read-only shared view cannot implement the advertised local fallback.

[balajinvda]

Fixed in 2c36257, taking your second option. A writable per-pod landing would need the engine to read a second path, which this design cannot provide without rewriting the chart. PVC readers no longer fall back in place: on the failure marker or past the deadline their wait stops with the reason and a non-zero exit, instead of downloading into the read-only view. When the download Job fails, the agent on the Job node records the failure and deletes every waiting reader of that model that is not Ready. Their controllers recreate them, and admission sees the failure record and leaves them on their own download path. hostPath readers keep the in-place fallback because their landing is writable until the agent binds. Tests: TestReaderScript_NoInPlaceFallbackOnReadOnlyView, TestModelVolume_SharedFilesystem_ReaderHasNoInPlaceFallback and TestModelVolumeController_FailedSharedDownloadRecreatesWaitingReaders. The design doc failure table is updated to match. One limit remains: a writer that hangs without failing still leaves readers waiting, as before; the Job failing is what releases them.

[FamousDirector]

[P1] Confine extraction writes when retrying partial rank streams

fetchRank retries into the same destination after an interrupted response, retaining symlinks from the first extraction. A later regular-file entry uses os.OpenFile here, which follows those existing links. The lexical link check below does not account for intermediate symlink resolution, so writes can escape the rank directory and truncate other files. A temporary-directory probe using two archives emitted by the actual Write function reproduced this: an interrupted first stream left relative links; after the source entry changed to a regular file, the retry overwrote a sibling file outside the extraction destination. Cache collection must not corrupt neighboring cache or agent filesystem data when a transfer retries.

Use filesystem operations confined to the destination root for all extraction paths, rejecting escaping symlink resolution. Extract retries into fresh staging directories and publish only a complete tree.

[balajinvda]

Fixed in b0ea0ee. Extract now refuses any entry beneath a symlinked directory, replaces a symlink at an entry path instead of following it, and opens files with O_NOFOLLOW. The rank fetch now uses the new ExtractFresh: each attempt extracts into a fresh staging directory next to the destination and replaces it only when the whole stream extracted, so a retry never merges into what an interrupted attempt left. Tests: TestExtract_RefusesWritesThroughSymlinkedParents (your d -> . then d/f -> ../victim chain), TestExtract_ReplacesSymlinkAtFilePath (both overwrote the victim before the fix) and TestExtractFresh_PublishesOnlyCompleteTrees.