refactor(runtime): retire Community image dependencies by akram · Pull Request #3386 · NVIDIA/OpenShell

akram · 2026-09-16T16:42:02Z

Summary

Retire the OpenShell Community workload-image dependency by making NVIDIA Ubuntu Noble the product default and removing Community catalog shorthand. Close the runtime gaps exposed by the smaller base image: shell selection now happens inside the workload boundary, SFTP runs as a native Rust sandbox helper, and E2E tests use explicit workload fixtures when they require extra tools.

Related Issue

Part of #3116.

Changes

Default Docker, Podman, Kubernetes, VM, Helm, RPM, and local development paths to nvcr.io/nvidia/base/ubuntu:24.04.
Remove Community image alias expansion and OPENSHELL_COMMUNITY_REGISTRY; non-path --from values are now treated as explicit OCI image references.
Remove Community-image references from the README, published docs, examples, provider profiles, skills, tests, and runtime configuration.
Resolve SSH login shells in openshell-sandbox, where the workload filesystem is visible, while preserving direct exec without a shell.
Serve the SSH SFTP subsystem from a native Rust adapter inside openshell-sandbox, removing the workload image's sftp-server dependency.
Keep SFTP as the standards-compatibility path for OpenSSH scp, VS Code Remote-SSH bootstrap, and external clients. Native bulk-transfer RPCs remain a separate follow-up.
Confine SFTP operations to the workspace with fd-relative Linux filesystem operations and openat2 resolution, including traversal and symlink-escape protection.
Carry trusted runtime-helper intent across the authenticated sandbox boundary without exposing a forgeable executable sentinel.
Keep the boundary wire format backward compatible by making shell and runtime-helper intent optional.
Use a pinned GHCR tool image for general E2E coverage while retaining dedicated coverage for the product's Noble default.
Replace VM and host-gateway tests' implicit Community-image and curl assumptions with explicit fixtures and a Bash /dev/tcp HTTP client.
Preserve the existing non-root fallback, workspace ownership, Landlock /bin allowance, and Podman diagnostics from this PR.

Testing

The manual run found and fixed root-level create, read-only open, and recursive-upload realpath compatibility bugs. After the fix, both native lanes pass the functional, confinement, cancellation, and concurrency matrix. In the same-image 1 GiB comparison, native median throughput was 184.84 MiB/s upload and 135.81 MiB/s download, versus 300.29 MiB/s and 166.23 MiB/s on the legacy server. Upload throughput remains above the manual plan's 20% investigation threshold.

mise run ci completed all change-related checks locally. Its Go gateway test lane reads the real /etc/openshell/gateways directory and failed because this development host contains a system default gateway; clean GitHub runners do not have that host state.

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Infrastructure changes include relevant E2E coverage

copy-pr-bot · 2026-09-16T16:42:05Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

drew

gator-agent

PR Review Status

This rebased change is project-valid and implements the maintainer-authored Alpine-default issue, but the initial review found five concrete blockers across default sandbox execution, Podman identity resolution, published guidance, multi-architecture builds, and build-input integrity.

Action required: @akram, please address the five findings below and push an updated head for a focused follow-up review.

Blocking findings:

GATOR-b4c97627-01: Stock Alpine's default shell is outside the fallback Landlock allowlist.
GATOR-b4c97627-02: Podman rejects a USER-less image before the new fallback identity is applied.
GATOR-b4c97627-03: Published docs and bundled skills still describe the retired community-image contract.
GATOR-b4c97627-04: The new multistage builders always produce x86-64 binaries, including for ARM64 images.
GATOR-b4c97627-05: The new builders execute an unpinned, unverified network installer as root.

Carried findings:

None

Gator metadata

Validation: Project-valid because this PR implements maintainer-authored issue #3116 and its Alpine-default acceptance criteria.
Docs: Missing for the direct CLI, default-image, policy, and deployment UX changes.
Checks: DCO is green; required workflows are not yet dispatched for this head and review blockers remain.
E2E: test:e2e and Kubernetes-specific coverage will be required after review feedback is resolved; not dispatched yet.
Head SHA: b4c9762725ee9773632dfa82a643156e62e5a3e9
Base SHA: 4cd5e547802c53112c8d5326b8ef8ce4c987ba7e
Merge base SHA: 4cd5e547802c53112c8d5326b8ef8ce4c987ba7e
Patch ID: ad4df3e0b3dd28a9376be3eab9c80ae3efce35a4
Gator payload: 9
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no
Next state: gator:in-review

github-actions · 2026-09-21T10:13:57Z

Label test:e2e applied, but pull-request/3386 does not exist yet. A maintainer needs to comment /ok to test dd617491dd2dc08c2b9f3efca3b6a542e7095eae to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

elezar · 2026-09-21T10:15:26Z

/ok-to-test dd61749

elezar · 2026-09-21T13:24:35Z

/ok-to-test 863ebd7

elezar · 2026-09-21T19:08:17Z

The failing VM and Kubernetes lanes appear to have two separate causes:

VM: the failing run tested PR head 863ebd78 directly and does not include chore(vm): refresh runtime defaults and hardening #3446 from current main. That change gives the VM driver and VM E2E lane a VM-specific default of nvcr.io/nvidia/base/ubuntu:24.04. Without it, this PR's global Alpine default reaches VM rootfs preparation and is rejected because Alpine does not provide /bin/bash. Please rebase onto current main and preserve the VM-specific Ubuntu defaults when resolving any conflicts.
Kubernetes: the v1alpha1 and v1beta1 jobs show docker.io/library/alpine:3.22 failing with Docker Hub 429 Too Many Requests, followed by ErrImagePull / ImagePullBackOff and the 300-second provisioning timeout. The Branch E2E kind setup currently preloads only the OpenShell gateway, sandbox, and supervisor images. It did not previously preload the community workload image either; that image was pulled from public GHCR and happened not to encounter this Docker Hub rate limit.

We should update the kind setup to pull/save/load docker.io/library/alpine:3.22 into the kind node before running the Kubernetes tests (or otherwise provide an authenticated/cache-backed mirror). Since 3.22 uses IfNotPresent, preloading it should keep sandbox pods from contacting Docker Hub.

The managed/operator jobs surface only ConfigurationPending timeouts because their diagnostics inspect the gateway namespace rather than the workspace namespaces where the sandbox pull events occur; they are consistent with the same pull failure.

Failing jobs:

default_sandbox_image() now returns docker.io/library/alpine:3.22, a generic version-qualified official image, so a fresh install no longer depends on the community sandbox image catalog. All compute drivers (docker, podman, kubernetes, vm) inherit this fallback. Part of NVIDIA#3116. Signed-off-by: Akram Signed-off-by: Akram <akram.benaissi@gmail.com>

…ox image Update the shared gateway default_image, Helm chart values, the standalone Kubernetes manifest, and the dev gateway task scripts to use docker.io/library/alpine:3.22 instead of the community base image, consistent with default_sandbox_image(). GPU e2e image-build base is left unchanged (CUDA needs a glibc base). Part of NVIDIA#3116. Signed-off-by: Akram Signed-off-by: Akram <akram.benaissi@gmail.com>

With the default sandbox image now Alpine, images that declare no OCI USER must start instead of being rejected. When the image declares no USER and the policy requests none, the Podman and Docker drivers now supply a numeric non-root identity (DEFAULT_SANDBOX_UID/GID = 1000) instead of rejecting, matching the numeric-identity behavior of the Kubernetes and VM drivers. The supervisor's resolved-identity path runs the sandbox as a synthesized non-root account without the account existing in the image. Images that declare a USER keep the OCI resolution path unchanged. Part of NVIDIA#3116. Signed-off-by: Akram <akram.benaissi@gmail.com> Signed-off-by: Evan Lezar <elezar@nvidia.com>

Signed-off-by: Evan Lezar <elezar@nvidia.com>

The restrictive default policy granted read-only access to /app, a directory that only existed in the community base image. A generic Alpine default has no /app, so remove it. Landlock best-effort already ignores absent paths; this just stops advertising a community-specific layout in the default. Part of NVIDIA#3116. Signed-off-by: Akram Signed-off-by: Akram <akram.benaissi@gmail.com>

Signed-off-by: Evan Lezar <elezar@nvidia.com>

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew · 2026-09-22T08:26:35Z

/ok to test 06ea61e

drew

gator-agent

PR Review Status

Thanks @drew. I reviewed the Community-retirement and provider-readiness fixture delta on the latest head in critical-only mode. It preserves the previously resolved findings and introduces no new Critical security, data-loss, or correctness defects.

Blocking findings:

No blocking findings remain.

Carried findings:

None; all seven prior findings remain resolved.

Gator metadata

Validation: Project-valid because this PR implements the maintainer-authored Community-image retirement work in #3116.
Docs: Published docs, examples, skills, and migration guidance are updated for explicit OCI references and the NVIDIA Ubuntu Noble default.
Checks: The current-head mirror is synchronized; Branch Checks, Helm Lint, Windows, and required E2E workflows are dispatched and running.
E2E: Required test:e2e, test:e2e-kubernetes, and test:e2e-centos coverage is dispatched on the current head.
Head SHA: 06ea61e8a2316716d3fe8ca6c2acca7e3e3d9878
Base SHA: 96c08f111b7ccf3d048a876c23e724014babb585
Merge base SHA: 96c08f111b7ccf3d048a876c23e724014babb585
Patch ID: 73b295b581bc122bbc2a63e2aa077be399329893
Gator payload: 10
Review mode: critical_only
Previous reviewed SHA: 8ad3d42ab7d931c00009a3bc5137da65987af21b
Review budget exhausted: yes
Maintainer decision required: no; all prior findings are resolved, no qualifying scope-growth decision is outstanding, and no new Critical was found.
Next state: gator:watch-pipeline

akram · 2026-09-22T10:08:43Z

@drew The rust-docker and python e2e tests are failing because they are referencing a sandbox user that was definied and existent in the community base:latest. The new image does not have such a user, so every policy with run_as_user: sandbox hits IdentityResolutionFailed.

akram · 2026-09-22T10:11:20Z

            "read_only": ["/usr", "/lib", "/proc", "/dev/urandom", "/etc", "/opt", "/var/log"],
            "read_write": ["/sandbox", "/tmp", "/dev/null"]},
        "landlock": {"compatibility": "best_effort"},
        "process": {"run_as_user": "sandbox", "run_as_group": "sandbox"},


The user sandbox does not exist in the new base image

I have added a commit that adds a test image. We also need uv and python for the tests. I don't think pulling these back into the default sandbox image makes sense.

Signed-off-by: Evan Lezar <elezar@nvidia.com>

elezar · 2026-09-22T11:47:06Z

/ok-to-test c1bb9fd

drew

gator-agent

PR Review Status

Thanks @akram for identifying that the Docker and Python E2E policies still expected the retired sandbox user, and thanks @elezar for the focused fixture update. I reviewed that latest delta in critical-only mode: the dedicated Noble fixture is confined to Docker E2E paths, explicit --from and --template selections remain intact, all seven prior findings remain resolved, and no newly introduced Critical defect was found.

Blocking findings:

No blocking findings remain.

Carried findings:

None; all seven prior findings remain resolved.

Gator metadata

Validation: Project-valid because this PR implements the maintainer-authored Community-image retirement work in #3116.
Docs: The dedicated E2E workload fixture and its separation from the product default are documented.
Checks: Helm Lint and GPU disposition are green; Branch Checks and required E2E are running on the current head.
E2E: Required test:e2e, test:e2e-kubernetes, and test:e2e-centos coverage is dispatched on the current head.
Head SHA: c1bb9fdc0d9bedb30d2096880dfeabeb4b8f8a69
Base SHA: 96c08f111b7ccf3d048a876c23e724014babb585
Merge base SHA: 96c08f111b7ccf3d048a876c23e724014babb585
Patch ID: b56eab79a052c0715c8019f4e8ee8a97b06e6116
Gator payload: 10
Review mode: critical_only
Previous reviewed SHA: 06ea61e8a2316716d3fe8ca6c2acca7e3e3d9878
Review budget exhausted: yes
Maintainer decision required: no; all prior findings are resolved, there is no qualifying scope growth, and no new Critical was found.
Next state: gator:watch-pipeline

elezar · 2026-09-22T12:43:40Z

I am merging this. The vm failure seems like a race that I will address in a follow-up. The same tests have passed in https://github.com/NVIDIA/OpenShell/actions/runs/35712151180/job/106698284871 using the same images.

drew · 2026-09-22T12:56:54Z

gator-agent

Monitoring Complete

Monitoring is complete because this PR has merged.

Final status: All seven Gator findings were resolved, the critical-only review found no new blocker, and maintainer @elezar accepted the remaining VM failure as a follow-up race before merging.

I removed the active gator:* label because there is nothing left for gator to monitor on this PR.

Gator metadata

Head SHA: c1bb9fdc0d9bedb30d2096880dfeabeb4b8f8a69
Gator payload: 10
Final state: merged

elezar · 2026-09-22T12:57:59Z

Tracking the VM restart failure in #3546, with reproduction instructions and investigation evidence.

Both CI attempts failed in vm_gateway_restart_preserves_running_and_stopped_intent. After merging this PR, a control run with the gateway built from main (50230616d) passed locally. Adding a diagnostic two-second delay immediately before supervisor ownership cleanup during shutdown reproduced the same failure: the gateway exited before cleanup completed, the old owner record survived in SQLite, and the replacement supervisor hit its 10-second acceptance timeout.

The passing and failing local runs used the same workload image. This confirms a gateway shutdown race under controlled timing; it does not establish the exact scheduling in CI or rule out image-dependent timing exposing the race. The issue includes the proposed bounded cleanup-drain direction. No production fix has been applied yet.

jgarciao · 2026-09-22T15:06:34Z

@elezar Related to the failing jobs due to Docker Hub 429 Too Many Requests one possible solution could be to use the Alpine ERC mirror mentioned here

akram mentioned this pull request Sep 16, 2026

refactor(sandbox)!: default to Alpine and drop community image resolution #3236

Closed

drew force-pushed the refactor/retire-community-images-alpine branch from fd27bc3 to b4c9762 Compare September 20, 2026 04:32

drew marked this pull request as ready for review September 20, 2026 04:32

drew requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners September 20, 2026 04:32

drew reviewed Sep 20, 2026

View reviewed changes

drew added the gator:in-review Gator is reviewing or awaiting PR review feedback label Sep 20, 2026

lunarwhite mentioned this pull request Sep 21, 2026

fix(policy)!: reject unsupported TLS endpoint values #3414

Merged

6 tasks

elezar force-pushed the refactor/retire-community-images-alpine branch 2 times, most recently from 1055e48 to b4c9762 Compare September 21, 2026 09:47

elezar changed the title ~~refactor(sandbox)!: retire community images and default to Alpine~~ feat(sandbox): default to official Alpine sandbox image Sep 21, 2026

elezar force-pushed the refactor/retire-community-images-alpine branch from b4c9762 to dd61749 Compare September 21, 2026 10:09

elezar added the test:e2e Requires end-to-end coverage label Sep 21, 2026

elezar mentioned this pull request Sep 21, 2026

refactor(cli)!: remove community image resolution #3503

Draft

5 tasks

elezar force-pushed the refactor/retire-community-images-alpine branch 2 times, most recently from 779d434 to 863ebd7 Compare September 21, 2026 13:18

elezar mentioned this pull request Sep 21, 2026

bug: surface supervisor startup failures to sandbox commands #3519

Open

4 tasks

akram and others added 6 commits September 21, 2026 21:33

test(conformance): use Alpine workload image

2b6bd9c

Signed-off-by: Evan Lezar <elezar@nvidia.com>

docs(config): document Alpine default images

628c641

Signed-off-by: Evan Lezar <elezar@nvidia.com>

drew added the gator:watch-pipeline Gator is monitoring PR CI/CD status label Sep 22, 2026

refactor(runtime): remove community image dependencies

9b2de64

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew changed the title ~~feat(sandbox): switch default sandbox image~~ refactor(runtime): retire Community image dependencies Sep 22, 2026

test(e2e): build provider readiness tool fixture

06ea61e

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew reviewed Sep 22, 2026

View reviewed changes

drew added gator:blocked Gator is blocked by process or repository gates and removed gator:watch-pipeline Gator is monitoring PR CI/CD status labels Sep 22, 2026

akram commented Sep 22, 2026

View reviewed changes

drew added gator:in-review Gator is reviewing or awaiting PR review feedback and removed gator:blocked Gator is blocked by process or repository gates labels Sep 22, 2026

fix(e2e): use a dedicated Noble fixture for Docker tests

c1bb9fd

Signed-off-by: Evan Lezar <elezar@nvidia.com>

drew reviewed Sep 22, 2026

View reviewed changes

drew added gator:watch-pipeline Gator is monitoring PR CI/CD status and removed gator:in-review Gator is reviewing or awaiting PR review feedback labels Sep 22, 2026

elezar approved these changes Sep 22, 2026

View reviewed changes

elezar enabled auto-merge September 22, 2026 12:09

drew added gator:blocked Gator is blocked by process or repository gates and removed gator:watch-pipeline Gator is monitoring PR CI/CD status labels Sep 22, 2026

elezar disabled auto-merge September 22, 2026 12:43

elezar merged commit 5023061 into NVIDIA:main Sep 22, 2026
149 of 156 checks passed

drew removed the gator:blocked Gator is blocked by process or repository gates label Sep 22, 2026

elezar mentioned this pull request Sep 22, 2026

bug(vm): gateway restart can retain stale supervisor ownership and fail sandbox recovery #3546

Closed

6 tasks

drew mentioned this pull request Sep 22, 2026

docs: add 0.1.0 upgrade guide outline #3540

Merged

7 tasks

Conversation

akram commented Sep 16, 2026 • edited by drew Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Sep 16, 2026

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 21, 2026

Uh oh!

elezar commented Sep 21, 2026

Uh oh!

elezar commented Sep 21, 2026

Uh oh!

elezar commented Sep 21, 2026

Uh oh!

drew commented Sep 22, 2026

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

akram commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akram Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

elezar Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

elezar commented Sep 22, 2026

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

elezar commented Sep 22, 2026

Uh oh!

Uh oh!

drew commented Sep 22, 2026

Monitoring Complete

Uh oh!

elezar commented Sep 22, 2026

Uh oh!

jgarciao commented Sep 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

akram commented Sep 16, 2026 •

edited by drew

Loading

akram commented Sep 22, 2026 •

edited

Loading