Skip to content

TL/CUDA: fall back for alltoall if buffer has no IPC handle - #1380

Open
0z5a wants to merge 3 commits into
openucx:masterfrom
0z5a:tl-cuda-ipc-fallback
Open

0z5a wants to merge 3 commits into
openucx:masterfrom
0z5a:tl-cuda-ipc-fallback

Conversation

@0z5a

@0z5a 0z5a commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

CLA: I have signed the UCF CLA (confirmed earlier on openucx/ucx#12223).

What

Part of #1378: alltoall only.

If cudaIpcGetMemHandle() rejects a buffer in ucc_tl_cuda_mem_info_get(), return UCC_ERR_NOT_SUPPORTED with a debug message instead of UCC_ERR_INVALID_PARAM with an error message. Also clear the CUDA error state. For alltoall, ucc_coll_init() then moves on to the next TL in the score map (TL/UCP in my GH200 build; TL/NCCL in my B300 build, which includes it). Alltoallv maps the result back to UCC_ERR_INVALID_PARAM with an error message, as on master.

Why ?

Legacy IPC handles exist only for cudaMalloc memory. With buffers from cuMemCreate (PyTorch expandable segments, ncclMemAlloc) or cudaMallocAsync, TL/CUDA alltoall failed at init, even though TL/UCP can run it.

Alltoallv keeps failing: whether its buffers are checked depends on the rank's own counts. A rank that sends nothing skips the check (and, without a proxy, the destination check too), so it would stay on TL/CUDA while the other ranks fall back, and hang at its barrier. Alltoall counts are the same on every rank, so all ranks decide the same way.

Reproducer from #1378, 2 ranks, alltoall with 1 MiB per peer, results checked on the host:

buffers master 9d14898 this PR
cudaMalloc TL/CUDA, PASS TL/CUDA, PASS (unchanged)
cudaMallocAsync init: Invalid parameter next TL, PASS
cuMemCreate (POSIX fd handle) init: Invalid parameter next TL, PASS
  • GH200 (aarch64, sm_90, CUDA 13.2), 2 ranks on one GPU: as in the table; the fallback is TL/UCP.
  • B300 (sm_103, CUDA 13.3, driver 610.x), 2 GPUs: as in the table; the fallback is TL/NCCL.
  • These runs were on 0460041. The alltoallv change in fe05b72 has not been run on a GPU yet.

With UCC_LOG_LEVEL=debug the fallback shows as (GH200):

tl_cuda_coll.c:75   UCC  DEBUG cudaIpcGetMemHandle() failed for 0x...: 1(invalid argument)
ucc_coll_score_map.c:140  UCC  DEBUG coll Alltoall is not supported for TL_CUDA, fallback TL_UCP

How ?

  • ucc_tl_cuda_mem_info_get() changes only in its error path. The team scratch buffer goes through the same function, but it is always cudaMalloc memory, and its caller still logs failed to get scratch memory info on any error.
  • alltoallv_ce.c wraps the call and turns UCC_ERR_NOT_SUPPORTED back into UCC_ERR_INVALID_PARAM.
  • All ranks need to use the same kind of allocator, which holds for expandable segments and for the CUDA mempool since both are process-wide settings. With mixed allocators, ranks would choose different TLs, and that case isn't handled by this PR.
  • A full fix, which alltoallv needs, would export VMM and pool buffers through cuMemExportToShareableHandle/cudaMemPoolExportPointer and keep them on TL/CUDA. I can follow up with that if it is preferred.
  • No overlap with TL/CUDA: add support for ucc mem map #1315: its push algorithms are opt-in, and this copy-engine path is unchanged there.

@Sergei-Lebedev

Copy link
Copy Markdown
Contributor

@0z5a pls update AUTHORS file

@greptile-apps

greptile-apps Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge within its stated same-allocator scope.

Summary

This PR lets CUDA all-to-all try another transport when a buffer cannot provide a CUDA IPC handle, while keeping all-to-all-v’s existing error behavior.

  • Alltoall can try another transport when CUDA IPC handles are unavailable.
  • Alltoallv keeps buffers without CUDA IPC handles as errors.
  • The contributor list now includes Dezhen Lu.

0z5a explicitly leaves mixed-allocator fallback unsupported and defers all-to-all-v support until handle export is available. No new actionable issues were found.

Reviews (6) · Last reviewed commit: "TL/CUDA: keep alltoallv error path witho..." · Reviewed by Greptile

mi->ptr,
cuda_st,
cudaGetErrorString(cuda_st));
return UCC_ERR_NOT_SUPPORTED;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Zero-count ranks hang

Returning UCC_ERR_NOT_SUPPORTED here can split an alltoallv across different transports, even when every rank uses cudaMallocAsync. On two directly connected GPUs, let rank 0 send nothing and rank 1 send data to rank 0. Rank 0 skips the IPC check because its source length is zero, and proxy_needed == 0 skips its destination check. Rank 1 hits this return and falls back, while rank 0 stays on TL/CUDA and waits forever at its shared barrier.

Make the fallback decision agree across ranks, including ranks with zero source counts.

Knowledge Base Used:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, a rank that sends nothing never runs the check. fe05b72 keeps the fallback for alltoall only, where counts are the same on every rank. Alltoallv returns UCC_ERR_INVALID_PARAM again, as on master, and needs the handle-export follow-up.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You’re right. The fe05b72 change keeps the fallback limited to alltoall and maps UCC_ERR_NOT_SUPPORTED back to UCC_ERR_INVALID_PARAM for alltoallv, avoiding the rank-divergent selection and barrier hang. The alltoallv limitation is therefore intentional pending handle-export support; my earlier finding no longer applies.

Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.

@0z5a

0z5a commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

@Sergei-Lebedev Added in bd11694, thanks.

@0z5a 0z5a changed the title TL/CUDA: fall back if buffer has no IPC handle TL/CUDA: fall back for alltoall if buffer has no IPC handle Nov 9, 2026
@Sergei-Lebedev

Copy link
Copy Markdown
Contributor

/build

@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — Lint (codestyle) · commit fe05b72e

TL;DR: The codestyle lint failed because the HEAD commit's title is 63 characters, exceeding the repo's 50-character limit; shorten/reword the commit subject and force-push.

Full analysis

Summary: The "Check commit title" step of the Lint (codestyle) workflow exited 1 — commit fe05b72e has a title longer than 50 chars.

Root cause: .github/workflows/codestyle.yaml enforces a 50-character maximum on every non-merge commit subject in the PR range. The title TL/CUDA: keep alltoallv on the error path without an IPC handle is 63 characters, so check_title() returned 1 ("Commit title is too long: 63") and the step exited with code 1. The other two commits in the PR (DOCS: add Dezhen Lu to AUTHORS, TL/CUDA: fall back if buffer has no IPC handle) passed, confirming this is purely a commit-message policy violation, not a code or formatting problem. The clang-format step never ran because the title check failed first.

Implicated commit: [REDACTED:Hex High Entropy String] — "TL/CUDA: keep alltoallv on the error path without an IPC handle" (PR #1380, branch tl-cuda-ipc-fallback)

File: .github/workflows/codestyle.yaml:39 (the [ ${#msg} -gt 50 ] check); offending object is the commit message of fe05b72e, not a source file.

Suggested fix: Reword the commit subject to ≤50 characters, keeping the TL/CUDA: prefix and no trailing period, and move the detail into the commit body. For example:

  • TL/CUDA: keep a2av on error path w/o IPC handle (47 chars)
  • TL/CUDA: no IPC handle fallback on error path (45 chars)

Then:

git commit --amend -m "TL/CUDA: keep a2av on error path w/o IPC handle"   # or interactive rebase if it is not HEAD
git push --force-with-lease origin tl-cuda-ipc-fallback

Since this commit appears to be a follow-up to TL/CUDA: fall back if buffer has no IPC handle, squashing the two into one commit would also clear the check. Note the check validates every commit in origin/<base>..HEAD, so all commits must satisfy the 50-char limit, the H1/H2 prefix regex, and the no-trailing-dot rule.

Related: PR #1380 (openucx/ucc); no existing issue found for this failure.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 4c901ed7-0d48-4097-9766-3b6dfee56037 in the triage console for the audit trail.

@0z5a
0z5a force-pushed the tl-cuda-ipc-fallback branch from fe05b72 to 4175e2b Compare November 10, 2026 08:39
@0z5a

0z5a commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Shortened the last commit's title to 46 characters (TL/CUDA: keep alltoallv error path without IPC) to pass the codestyle check; the code is unchanged.

@Sergei-Lebedev

Copy link
Copy Markdown
Contributor

/build

@janjust
janjust force-pushed the tl-cuda-ipc-fallback branch from 4175e2b to 00ede25 Compare November 12, 2026 14:35
@janjust

janjust commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

/build

@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — ASAN · commit 00ede25a

TL;DR: The ASAN job's gtest shards fail on test_cfg_file.opts_applied (shard 0) and test_cfg_file.env_preference (shard 2) — the two tests that apply UCC config‑file values to a config table via ucc_config_parser_fill_opts(); the config‑file→options plumbing in src/utils/ucc_parser.c is broken against the live openucx/ucx master this job builds, and it is unrelated to PR #1380 (TL/CUDA isn't even compiled here).

Full analysis

Summary: ASAN gtest shards exited 1 with [ FAILED ] test_cfg_file.opts_applied (shard 0) and [ FAILED ] test_cfg_file.env_preference (shard 2); build and all 3386+ other tests passed.

Root cause: Evidence from the logs:

  • The build job succeeded completely (CXXLD gtest, artifact ucc-asan-build uploaded). Failure is only in the sharded test jobs.
  • Of the four test_cfg_file tests, exactly the two that call init_cfg() → ucc_config_parser_fill_opts() fail. parse_existing / parse_non_existing pass, so test/gtest/utils/ucc_test.conf is found and parsed correctly (path/GTEST_UCC_TOP_SRCDIR is fine). The breakage is in applying parsed file values to a config table (ucc_apply_file_cfg() → ucc_config_parser_set_value()), or in that call returning non‑UCC_OK (EXPECT_EQ(UCC_OK, status) in init_cfg). Failure is deterministic, not flaky — both tests fail in whichever shard runs them.
  • That code path is a thin wrapper over UCX's ucs_config_parser_fill_opts() / ucs_config_parser_set_value(), compiled under UCX feature macros (UCS_HAVE_CONFIG_GLOBAL_LIST_ENTRY_FLAGS, UCS_HAVE_PARSER_SET_VALUE_TABLE_PREFIX). The workflow resolves UCX live: git ls-remote https://github.com/openucx/ucx refs/heads/master (OPEN_UCX_BRANCH: master, cache key ucx-asan-[REDACTED:Hex High Entropy String], "Detected UCX version: 1.24"), so an upstream UCX master change silently changes this behaviour.
  • PR TL/CUDA: fall back for alltoall if buffer has no IPC handle #1380 ("TL/CUDA: fall back for alltoall if buffer has no IPC handle") cannot be implicated: configure in this job reports CUDA support: no, TL CUDA: disabled (CUDA not available), TL modules: < self ucp > — the PR's files are not compiled.

Implicated commit: unknown (not PR #1380 / 00ede25a); the moving dependency is openucx/ucx master at [REDACTED:Hex High Entropy String] as pulled by this run. Last UCC-side change in this area: c3e8ede4 (Sergey Lebedev, "UTIL: fix mranged parameters parser", #1319).

File: src/utils/ucc_parser.c:660 (ucc_config_parser_fill_opts) and src/utils/ucc_parser.c:536-571 (ucc_apply_file_cfg_value), via src/utils/ucc_parser.h:215 (ucc_config_parser_set_value passes NULL as table_prefix); test at test/gtest/utils/test_cfg_file.cc:91 and :103.

Suggested fix:

  1. Re-run the ASAN workflow on master (or git checkout master && ./test/gtest/gtest --gtest_filter='test_cfg_file.*' against the same UCX build) to confirm this is a pre-existing master/UCX breakage, and unblock PR TL/CUDA: fall back for alltoall if buffer has no IPC handle #1380 — it is not the cause.
  2. Stop tracking a moving dependency: change OPEN_UCX_BRANCH from master to a released tag (or pin an explicit UCX sha) in the ASAN workflow so upstream UCX changes can't red the UCC gate.
  3. Real fix in UCC: make ucc_config_parser_set_value() pass the real table prefix instead of NULL in the UCS_HAVE_PARSER_SET_VALUE_TABLE_PREFIX branch (thread entry->prefix through ucc_apply_file_cfg() → ucc_apply_file_cfg_value()), and add a ucc_warn/ucc_debug of the variable name + status when ucc_apply_file_cfg_value() fails so this class of failure is diagnosable from the log instead of only showing up as a bare EXPECT_EQ mismatch.
  4. While in this file: ucc_add_team_sections() (src/utils/ucc_parser.c:657) returns status which is uninitialized when no section matched but found was set by an earlier iteration path — worth initializing to UCC_OK.

Related: PR #1380 (#1380) — failing check is unrelated to its change; no existing issue found for test_cfg_file failures.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id e3309a2d-ad41-4f5f-92ee-fb37fbf8b6ba in the triage console for the audit trail.

0z5a and others added 3 commits November 19, 2026 13:00
ucc_tl_cuda_mem_info_get() exports the user buffer of the alltoall(v)
copy-engine algorithms with the legacy cudaIpcGetMemHandle(). That call
only accepts cudaMalloc memory and fails with cudaErrorInvalidValue for
buffers from cuMemCreate (e.g. PyTorch expandable segments,
ncclMemAlloc) or from a stream-ordered memory pool (cudaMallocAsync).
The error was mapped to UCC_ERR_INVALID_PARAM, which ucc_coll_init()
does not treat as a reason to try the next TL, so the collective failed
although TL/UCP could run it.

Return UCC_ERR_NOT_SUPPORTED with a debug message instead, and clear
the CUDA error state, so that collective init falls back to the next
TL in the score map.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
For alltoallv, whether a buffer is checked depends on the rank's own
counts: a rank that sends nothing skips the IPC check. If the other ranks
fell back to another TL, that rank would stay on TL/CUDA and hang at its
barrier. Map NOT_SUPPORTED back to INVALID_PARAM for alltoallv, as on
master, and keep the fallback for alltoall, where counts are the same on
every rank.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
@janjust
janjust force-pushed the tl-cuda-ipc-fallback branch from 00ede25 to 5ad8565 Compare November 19, 2026 19:00
@janjust

janjust commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

/build

@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — UCC/ucc-test-nvls · commit 1ba6d2fa

TL;DR: The x86_64 build_helper Docker image build failed because the scctl installer in Dockerfile.build_helper could not query Artifactory (HTTP 401 on the "latest version" lookup) — an external Artifactory auth/permission change, not a code problem in PR #1380. Fix by supplying Artifactory credentials (or pinning/caching the scctl binary) in that RUN step and re-running.

Full analysis

Summary: Stage "Build Docker" (node 78, x86_64 build_helper) failed at STEP 12/18 of .ci/dockerfiles/Dockerfile.build_helper; the pipeline then aborted (pipeline stop on build_helper) before any NVLS test ran.

Root cause: The build step RUN curl -s https://artifactory.nvidia.com/artifactory/sw-swx-lab-infra-generic-local/scctl/get-app.sh | bash -s -- --output-dir /usr/local/bin failed with:

Error: Failed to get latest version: curl: (22) The requested URL returned error: 401
subprocess exited with status 1
Error: building at STEP "RUN curl -s .../scctl/get-app.sh | bash ..."

The installer script itself downloads, but its unauthenticated Artifactory query for the latest scctl version now returns 401 (unauthorized). This is an infrastructure/credentials regression, not a source change: build #405 of the same job, ~10 hours earlier on the same Dockerfile, ran the identical step successfully ("Downloading scctl version v0.14.1 … Successfully installed scctl version v0.14.1"). The aarch64 UCC image build in the same parallel stage succeeded, so compilation of the PR's code is fine. Because the job forces --no-cache rebuild of build_helper on every run ("Forcing building file per user request"), any Artifactory read-permission change breaks the whole NVLS pipeline immediately.

Implicated commit: No UCC commit — external Artifactory ACL/token change. The step was introduced/relocated in 8c1a64f ("CI: update scctl repository location (#1365)", Daniel Pressler), which is the code that depends on anonymous Artifactory read access.

File: .ci/dockerfiles/Dockerfile.build_helper:23

Suggested fix:

  1. Short term: re-run build TEST: add gtest ucc on BF2 #406 — if anonymous access was revoked permanently, it will keep failing.
  2. Make the scctl install authenticated: pass an Artifactory identity token into the build (e.g. --build-arg/--secret sourced from a Jenkins credential, never echoed into the log) and have get-app.sh use it; confirm with the sw-swx-lab-infra-generic-local repo owners whether anonymous read on scctl/ was removed.
  3. Make the step robust: pin an explicit scctl version (e.g. v0.14.1) so the "latest version" API lookup is skipped, add curl --retry 3 --retry-all-errors, and/or pre-bake scctl into the base image so build_helper doesn't hit Artifactory on every --no-cache rebuild.
  4. Consider dropping the forced --no-cache rebuild of build_helper (it's tagged nvls/latest and rarely changes) so a transient registry/Artifactory outage can't block all NVLS testing.
  5. If a token is already configured for this step, rotate it — a 401 can also mean an expired credential.

Related: #1365 (introduced the current scctl Artifactory location)

@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — UCC/ucc-test-nvls · commit 5ad8565e

TL;DR: The build_helper Docker image build failed at the step that pipes the scctl installer from artifactory.nvidia.com into bash — Artifactory returned HTTP 401 (unauthorized), so the install script aborted. This is an infrastructure/credentials problem on the sw-swx-lab-infra-generic-local repo, unrelated to the tl-cuda-ipc-fallback PR code; the fix is to supply Artifactory credentials (or restore anonymous read) and make the step fail loudly with curl -fsSL.

Full analysis

Summary: Stage "Build Docker" (node 78) of UCC/ucc-test-nvls #406 failed building harbor.mellanox.com/torch-ucc/x86_64/build_helper:nvls at STEP 12/18; the pipeline then stopped (pipeline stop on build_helper).

Root cause: The Dockerfile step
RUN curl -s https://artifactory.nvidia.com/artifactory/sw-swx-lab-infra-generic-local/scctl/get-app.sh | bash -s -- --output-dir /usr/local/bin
produced:
Error: Failed to get latest version: curl: (22) The requested URL returned error: 401 → subprocess exited with status 1 → Error: building at STEP "RUN curl -s ...": exit status 1.
The installer script fetched (anonymously) is itself unable to query the artifact repository — Artifactory now rejects unauthenticated access to sw-swx-lab-infra-generic-local (permission/repo change or expired anonymous read). The build ran with --no-cache ("Forcing building file per user request"), so the previously cached layer could not hide the problem. Build #405 of the same job built this image successfully 39 s, so nothing in the source tree changed to cause this — the remote endpoint changed. Note also that curl -s without -f means an auth failure on the installer URL itself would silently pipe an error page into bash, masking the real cause.

Implicated commit: Not the triggering commit 5ad8565 (tl-cuda-ipc-fallback). Last change to this step: 8c1a64ff "CI: update scctl repository location (#1365)" — Daniel Pressler; the failure itself is an external Artifactory authorization change.

File: .ci/dockerfiles/Dockerfile.build_helper:23

Suggested fix:

  1. Restore read permission for the CI identity on sw-swx-lab-infra-generic-local/scctl (or move scctl to a repo with anonymous read). If the repo is intentionally protected now, pass an Artifactory token into the build as a secret/build-arg and have the installer use it, e.g.:
    RUN --mount=type=secret,id=artifactory_token curl -fsSL -H "Authorization: Bearer $(cat /run/secrets/artifactory_token)" <url> | bash -s -- --output-dir /usr/local/bin
    (inject via the existing Jenkins credentials mechanism; do not echo the token — any token already printed in logs should be rotated).
  2. Harden the step so failures are explicit and not pipe-into-bash silent: use curl -fsSL ... -o /tmp/get-app.sh && bash /tmp/get-app.sh ..., and pin an explicit scctl version instead of "latest" so the build doesn't depend on a live version lookup.
  3. Short-term unblock: re-run with the cached build_helper:nvls image (drop the force-rebuild/--no-cache), since only this layer is affected.

Related: PR #1365 (CI: update scctl repository location) — the change that introduced the current Artifactory path; PR #1380 is the build trigger but not the cause.

@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — UCC/ucc-test-mpi · commit 1ba6d2fa

TL;DR: The ucc-test-mpi pipeline failed while building the build_helper container image: step 12 of .ci/dockerfiles/Dockerfile.build_helper downloads the scctl installer from artifactory.nvidia.com anonymously and Artifactory returned HTTP 401, aborting the image build (and therefore the whole pipeline). This is an infrastructure/credentials problem, not a code problem in PR #1380 — the fetch needs authenticated access (or a publicly readable repo path).

Full analysis

Summary: Stage "Build Docker" (node 63) failed after 46s: podman build of Dockerfile.build_helper aborted at RUN curl -s https://artifactory.nvidia.com/.../scctl/get-app.sh | bash ... with Error: Failed to get latest version: curl: (22) The requested URL returned error: 401, which then triggered pipeline stop on build_helper.

Root cause: The scctl install step performs an unauthenticated download from the Artifactory generic repo sw-swx-lab-infra-generic-local. That repo now rejects anonymous requests (401 Unauthorized), so get-app.sh fails to resolve the latest version and exits non-zero, failing the Docker build. Note the UCC image build in the parallel branch (node 54) completed successfully, confirming the failure is isolated to the helper-image provisioning step, not to the commit under test. Also, the job forces --no-cache ("Forcing building file per user request"), so there is no cached layer to mask the outage. No timeout/hang involved — failure occurred ~6s after the step started.

Implicated commit: 8c1a64f — "CI: update scctl repository location (#1365)", Daniel Pressler (last change to the failing line; the repo it points at now requires auth)

File: .ci/dockerfiles/Dockerfile.build_helper:23

Suggested fix:

  1. Make the download authenticated: bind the Artifactory service credentials in Jenkins and pass them into the build via BuildKit secrets (preferred) or a build arg, e.g. RUN --mount=type=secret,id=artifactory_token curl -s -H "Authorization: Bearer $(cat /run/secrets/artifactory_token)" <url> | bash -s -- --output-dir /usr/local/bin. Do not hardcode the token in the Dockerfile or echo it in logs.
  2. Verify with the SWX lab infra owners whether sw-swx-lab-infra-generic-local/scctl should allow anonymous reads; if anonymous access was intentionally revoked, either restore it or publish scctl into a repo the CI already authenticates against (same path the base images in harbor.mellanox.com use).
  3. Longer term, bake scctl into the harbor.mellanox.com/hpcx/${ARCH}/ubuntu24.04/base image so every PR build doesn't depend on a live external download, and add set -o pipefail / an explicit error message so this failure mode is obvious.
  4. Re-run build Added valgrind suppression file #130 once access is restored; no change to PR TL/CUDA: fall back for alltoall if buffer has no IPC handle #1380 is needed.

Related: PR #1365 (#1365) changed the scctl repository location; PR #1231 (#1231) introduced the scctl/Slurm-based CI flow.

@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — UCC/ucc-test-gtest · commit 5ad8565e

TL;DR: The build_helper Docker image build failed at the scctl install step because https://artifactory.nvidia.com/artifactory/sw-swx-lab-infra-generic-local/scctl/get-app.sh now returns HTTP 401 (anonymous access/credential no longer valid); fix by fetching scctl with authenticated credentials (or baking it into the base image), not by changing PR #1380.

Full analysis

Summary: Stage "Build Docker" (node 54) of UCC/ucc-test-gtest #216 failed while building .ci/dockerfiles/Dockerfile.build_helper at STEP 12/18; the pipeline then aborted ("pipeline stop on build_helper").

Root cause: Infrastructure/credential regression, not a code defect. In build #216:

STEP 12/18: RUN curl -s https://artifactory.nvidia.com/artifactory/sw-swx-lab-infra-generic-local/scctl/get-app.sh | bash -s -- --output-dir /usr/local/bin
Error: Failed to get latest version: curl: (22) The requested URL returned error: 401
Error: building at STEP "RUN curl -s ...": exit status 1

The identical step with the identical Dockerfile succeeded one day earlier in build #215 ("Downloading scctl version v0.14.1 / Successfully installed scctl version v0.14.1"), so the Artifactory repo sw-swx-lab-infra-generic-local has stopped serving unauthenticated requests (anonymous read revoked or the service account/token used by the agent expired). Because the image never built, scctl was absent and the downstream Slurm cleanup also failed with a cascading symptom: scancel: error: No job identification provided (empty ${SLURM_JOB_ID}). Nothing in the failure touches the tl-cuda-ipc-fallback changes of PR #1380.

Implicated commit: No code commit caused this. The step was last touched by 8c1a64ff — "CI: update scctl repository location (#1365)", Daniel Pressler — which introduced the current Artifactory URL that is now returning 401.

File: .ci/dockerfiles/Dockerfile.build_helper:23

Suggested fix:

  1. Short term: verify/renew the Artifactory credential for sw-swx-lab-infra-generic-local (rotate the service-account identity token if it expired) and confirm anonymous read is still intended for that repo; then re-run DOCS: Component diagram update #216.
  2. Make the step authenticated and fail loudly instead of relying on anonymous access — pass a Jenkins-managed Artifactory token into the build as a BuildKit secret/--build-arg and use it in the curl call (e.g. an Authorization/X-JFrog-Art-Api header sourced from the secret, never echoed), and add curl -fsSL --retry 3 --retry-all-errors so a 401/5xx produces an immediate explicit error.
  3. More robust: pre-install a pinned scctl version (v0.14.1 is what CORE: Change mt pq default value #215 used) into harbor.mellanox.com/hpcx/${ARCH}/ubuntu24.04/base, or mirror the installer into the internal harbor/registry so gtest CI no longer depends on an external Artifactory endpoint at image-build time.
  4. Separately, guard the pipeline teardown so slurm.stop is skipped when SLURM_JOB_ID is empty, to avoid the misleading scancel: No job identification provided error masking the real failure.

Related: PR #1365 (introduced the scctl repository location); no existing issue tracks the 401 — worth opening one against the CI infra. PR #1380 is unrelated to this failure.

@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — UCC/ucc-test-gtest · commit 1ba6d2fa

TL;DR: The build_helper Docker image build failed at RUN curl ... scctl/get-app.sh because Artifactory returned HTTP 401 (unauthenticated) for sw-swx-lab-infra-generic-local; the fix is to pass Artifactory credentials into that docker build (or use an anonymously readable repo path) rather than an un-authenticated curl.

Full analysis

Summary: Stage "Build Docker" (node 54) for harbor.mellanox.com/torch-ucc/x86_64/build_helper:gtest failed at STEP 12/18 of .ci/dockerfiles/Dockerfile.build_helper, which aborted the pipeline (pipeline stop on build_helper, exit code 1).

Root cause: Infrastructure/credential failure, not a code regression from PR #1380. The Dockerfile fetches the scctl installer anonymously:

STEP 12/18: RUN curl -s https://artifactory.nvidia.com/artifactory/sw-swx-lab-infra-generic-local/scctl/get-app.sh | bash -s -- --output-dir /usr/local/bin
Error: Failed to get latest version: curl: (22) The requested URL returned error: 401
Error: building at STEP ... exit status 1

The ...-generic-local repository requires authentication (or anonymous read was revoked), so the installer's version lookup got a 401 and exited 1. The build was run with --no-cache ("Forcing building file per user request"), so the previously cached layer could no longer mask the problem. The UCC image itself built, compiled and pushed successfully (gtest binary installed at 19:13), and the SLURM scancel: error: No job identification provided in the stop stage is a downstream side-effect of never reaching the test stage (no SLURM_JOB_ID was ever set).

Implicated commit: 8c1a64f — "CI: update scctl repository location (#1365)", Daniel Pressler (changed the scctl Artifactory URL; the new location is not anonymously readable). The PR head commit 1ba6d2f is unrelated.

File: .ci/dockerfiles/Dockerfile.build_helper:23

Suggested fix:

  1. Supply Artifactory credentials to the scctl download instead of anonymous access — e.g. mount a token at build time (RUN --mount=type=secret,id=artifactory ... or curl -u "$ART_USER:$ART_TOKEN" fed from a Jenkins credential/--secret), and do not bake the token into a layer or --build-arg. Rotate the credential if any token was ever echoed in a build log.
  2. Alternatively point line 23 at a virtual/remote repo path that permits anonymous read (or vendor a pinned scctl tarball), which keeps the image buildable from any agent.
  3. Harden the step regardless: use curl -fsSL --retry 3 ... -o /tmp/get-app.sh && bash /tmp/get-app.sh ... so an HTTP error fails with a clear message instead of piping an error page into bash.
  4. Make the pipeline stop on build_helper stage skip slurm.stop when SLURM_JOB_ID is empty, so image-build failures aren't obscured by the scancel error.

Related: PR #1365 (CI: update scctl repository location) is the change that introduced this URL; no existing issue found for the 401.

@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — UCC/ucc-test-mpi · commit 5ad8565e

TL;DR: The Build Docker stage for build_helper failed because STEP 12/18 (curl ... scctl/get-app.sh | bash) got an HTTP 401 Unauthorized from artifactory.nvidia.com, aborting the image build — this is an infra/credential problem, not a code problem in PR #1380; fix it by supplying Artifactory credentials to that build step (or mirroring scctl into harbor).

Full analysis

Summary: UCC/ucc-test-mpi #130 failed in stage Build Docker (node 63) while building .ci/dockerfiles/Dockerfile.build_helper; the pipeline then short-circuited at pipeline stop on build_helper, so no MPI tests ever ran.

Root cause: The image is rebuilt with --no-cache, so every build re-runs:

STEP 12/18: RUN curl -s https://artifactory.nvidia.com/artifactory/sw-swx-lab-infra-generic-local/scctl/get-app.sh | bash -s -- --output-dir /usr/local/bin
Error: Failed to get latest version: curl: (22) The requested URL returned error: 401
Error: building at STEP "RUN curl -s ...": exit status 1

The get-app.sh helper is fetched and executed, but its own "get latest version" request to the sw-swx-lab-infra-generic-local generic repo is rejected with 401, i.e. anonymous/unauthenticated read access to that Artifactory repo is no longer granted (credentials revoked/expired or repo permissions changed). No Artifactory auth is passed into the docker build at all. Evidence it is environmental, not the PR: build #129 ran the identical Dockerfile (unchanged since 8c1a64f, 2026-10-19) and the same Build Docker stage succeeded in 64s; the failure is in a container-image build step that never touches the tl-cuda-ipc-fallback sources. The two secondary symptoms in the log (podman system reset umount error, cgroups-v1 deprecation warnings) are benign and not the failure.

Implicated commit: Not the PR commit 5ad8565. The fragile step was introduced/changed by 8c1a64ff — "CI: update scctl repository location (#1365)", Daniel Pressler. The actual trigger is an external Artifactory permission/credential change.

File: .ci/dockerfiles/Dockerfile.build_helper:23

Suggested fix:

  1. Immediate: pass an Artifactory identity token into the build and use it, e.g. RUN --mount=type=secret,id=artifactory_token curl -sfL -H "Authorization: ..." <url> | bash -s -- --output-dir /usr/local/bin with the token injected from the existing Jenkins credentials store (do not bake it into a layer or echo it). Verify/rotate the service account used for sw-swx-lab-infra-generic-local — a 401 usually means the token behind it expired.
  2. More robust: stop pulling scctl from Artifactory at image-build time. Either bake it into the harbor.mellanox.com/hpcx/${ARCH}/ubuntu24.04/base image or mirror a pinned scctl version into harbor, so the test pipeline no longer depends on a live authenticated Artifactory call.
  3. Hardening regardless: add -f/--fail and an explicit version pin to the curl so a future auth/HTML error fails loudly instead of piping an error page into bash, and consider retry logic so a transient 5xx/401 doesn't kill the whole MPI test job.
  4. Re-run Added valgrind suppression file #130 once credentials are restored; the PR itself has not actually been tested yet.

Related: #1365 (introduced the current scctl repository URL); PR under test: #1380

@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — UCC/ucc-test-nvls · commit 1ba6d2fa

TL;DR: The build_helper Docker image build failed because the Artifactory scctl installer returns HTTP 401 (unauthenticated) — an infra/credentials problem in .ci/dockerfiles/Dockerfile.build_helper, not a code regression from PR #1380. Fix by supplying Artifactory credentials/token to that RUN step (or hosting scctl in an anonymously-readable location).

Full analysis

Summary: Stage "Build Docker" (x86_64 build_helper:nvls image) failed at STEP 12/18 with Error: Failed to get latest version: curl: (22) The requested URL returned error: 401, which aborted the whole ucc-test-nvls pipeline before any NVLS test ran.

Root cause: Line 23 of .ci/dockerfiles/Dockerfile.build_helper pipes https://artifactory.nvidia.com/artifactory/sw-swx-lab-infra-generic-local/scctl/get-app.sh into bash with no authentication. The request for the scctl "latest version" metadata is rejected with HTTP 401, so the installer exits 1 and podman build fails (building at STEP "RUN curl -s ... | bash -s -- --output-dir /usr/local/bin": exit status 1). The repo path was changed in commit 8c1a64f ("CI: update scctl repository location") and that generic repo now requires (or always required) credentials — anonymous reads are being denied. Note the build also uses --no-cache, so no cached layer can mask the failure. The aarch64 UCC image in the parallel branch built and pushed fine, confirming this is isolated to the unauthenticated Artifactory fetch, not to the commit under test (1ba6d2f / PR #1380).

Secondary (symptom only): the post-failure slurm.stop step then threw java.nio.file.NoSuchFileException: /mnt/pvc/job-id-jenkins-UCC-ucc-test-nvls-407.txt because no Slurm job was ever submitted — that masks the real error in the build summary.

Implicated commit: 8c1a64f — Daniel Pressler, "CI: update scctl repository location (#1365)" (changed the scctl Artifactory URL); the triggering commit 1ba6d2f is not implicated.

File: .ci/dockerfiles/Dockerfile.build_helper:23

Suggested fix:

  1. Pass Artifactory credentials into the build instead of fetching anonymously, e.g. inject a token via BuildKit secret / build-arg from a Jenkins credential and use it in the header, something like:
    RUN --mount=type=secret,id=artifactory_token curl -fsSL -H "Authorization: Bearer $(cat /run/secrets/artifactory_token)" <url> | bash -s -- --output-dir /usr/local/bin
    (store the token in Jenkins credentials; never bake it into a layer).
  2. Alternatively point at an anonymously readable mirror of scctl (harbor/hpcx base image can pre-install it) so the CI image build has no external auth dependency.
  3. Harden the step regardless: use curl -fsSL and set -o pipefail so a 401 on the script itself fails immediately with a clear message, and pin a specific scctl version instead of querying "latest".
  4. Make slurm.stop tolerate a missing jobIdFile (fileExists guard) so the real root cause isn't buried by the NoSuchFileException.
  5. Verify whether the Artifactory token/service account used by svcnbu-swx-hpcx expired; if so rotate/renew it in the credential store.

Related: PR #1365 (scctl repository location change, the implicated CI commit). No existing issue tracks this 401; worth opening one. PR #1380 is unrelated to the failure and can be re-run once the image build is fixed.

@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — UCC/ucc-test-nvls · commit 5ad8565e

TL;DR: The build died while building the build_helper Docker image — the unauthenticated curl of the scctl installer from artifactory.nvidia.com returned HTTP 401, aborting docker build. Fix by supplying Artifactory credentials to that RUN step (or moving scctl to an anonymously-readable location); the PR's code changes are unrelated.

Full analysis

Summary: Stage "Build Docker" (node 64, x86_64 build_helper image) failed at STEP 12/18 of .ci/dockerfiles/Dockerfile.build_helper.

Root cause: The Dockerfile pipes an Artifactory-hosted installer into bash with no authentication:
RUN curl -s https://artifactory.nvidia.com/artifactory/sw-swx-lab-infra-generic-local/scctl/get-app.sh | bash -s -- --output-dir /usr/local/bin
The log shows Error: Failed to get latest version: curl: (22) The requested URL returned error: 401 → subprocess exited with status 1 → Error: building at STEP "RUN curl -s https://artifactory.nvidia.com/...". The sw-swx-lab-infra-generic-local repo no longer permits anonymous reads (or the path moved), so the image build aborts. Because the image is built with --no-cache and gates the Slurm submission, nothing downstream ran — the trailing java.nio.file.NoSuchFileException: /mnt/pvc/job-id-jenkins-UCC-ucc-test-nvls-407.txt in slurm.stop is a secondary symptom (no Slurm job was ever submitted), not the cause. The parallel aarch64 UCC image built and pushed fine, confirming this is infrastructure, not PR #1380 code.

Implicated commit: 8c1a64f — "CI: update scctl repository location (#1365)", Daniel Pressler (last change to this curl URL; combined with an Artifactory permission/path change it now 401s)

File: .ci/dockerfiles/Dockerfile.build_helper:23

Suggested fix:

  1. Verify the current scctl location/visibility in Artifactory. If it requires auth, inject a token into the build instead of an anonymous fetch, e.g. pass it via BuildKit secret or build-arg from a Jenkins credential and use curl -H "Authorization: ..." / --netrc-file, keeping the value out of image layers and logs. Do not hardcode the token in the Dockerfile.
  2. Make the step fail loudly and early rather than mid-build: add curl -sSfL so the HTTP error is reported by curl itself, and set -o pipefail semantics (RUN bash -o pipefail -c '...').
  3. As a short-term unblock, revert the URL to the previously working repository from before 8c1a64f, or bake scctl into the harbor.mellanox.com/hpcx/x86_64/ubuntu24.04/base image so the per-build network fetch disappears.
  4. Separately, guard slurm.stop against a missing jobIdFile (fileExists check) so failures before submission don't mask the real error with a NoSuchFileException.

Related: PR #1365 (CI: update scctl repository location) — the change that introduced this URL; triggering PR #1380 is unaffected and should pass once the image build is fixed.

@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — UCC/ucc-test-nvls · commit 1ba6d2fa

TL;DR: The Build Docker stage for the build_helper image failed because curl to https://artifactory.nvidia.com/artifactory/sw-swx-lab-infra-generic-local/scctl/get-app.sh now returns HTTP 401 Unauthorized, so the scctl install step in Dockerfile.build_helper exits 1. This is an infrastructure/credentials issue unrelated to PR #1380 — the Artifactory repo path needs working anonymous access or the build needs to pass Artifactory credentials.

Full analysis

Summary: Stage 54 "Build Docker" (build_helper image) aborted at STEP 12/18 of .ci/dockerfiles/Dockerfile.build_helper; the pipeline then stopped (pipeline stop on build_helper), failing the whole Parallel stage even though the aarch64 UCC image build (stage 71) completed successfully.

Root cause: From the stage-54 log:

STEP 12/18: RUN curl -s https://artifactory.nvidia.com/artifactory/sw-swx-lab-infra-generic-local/scctl/get-app.sh | bash -s -- --output-dir /usr/local/bin
Error: Failed to get latest version: curl: (22) The requested URL returned error: 401
Error: building at STEP "RUN curl -s ...get-app.sh | bash ...": exit status 1

The Artifactory generic repo sw-swx-lab-infra-generic-local rejects the unauthenticated request (401). The bootstrap script is fetched and executed with no credentials and no --fail/retry handling, so the docker build dies at that layer. Nothing in the failure touches UCC source code — the parallel UCC build/configure/install (CUDA 13.1, NVLS enabled) ran clean to completion and pushed its image.

Implicated commit: 8c1a64ff — "CI: update scctl repository location (#1365)", Daniel Pressler (changed the scctl URL to the now-401 sw-swx-lab-infra-generic-local path). The 401 itself is an Artifactory permission/retention change, not a code regression.

File: .ci/dockerfiles/Dockerfile.build_helper:23

Suggested fix:

  1. Verify the repo path/permissions with the Artifactory owners — either restore anonymous read on sw-swx-lab-infra-generic-local/scctl/ or get the correct current path for get-app.sh.
  2. If the repo now requires auth, inject an Artifactory identity token into the build via a Jenkins credential and a BuildKit/podman build secret (never a plain --build-arg, which bakes it into image history), e.g. mount the secret and pass it as a header to curl. Rotate any token that ends up in logs.
  3. Harden the step so a registry hiccup produces a clear error instead of a silently piped script: use curl -fsSL --retry 3 --retry-all-errors -o /tmp/get-app.sh <url> && bash /tmp/get-app.sh --output-dir /usr/local/bin. Better still, pin a specific scctl version/checksum rather than "latest".
  4. Short term, to unblock PR CI: re-run once access is restored, or make the build_helper image use a cached/published tag instead of --no-cache rebuilding it on every PR run.

Related: PR #1365 (CI: update scctl repository location) is the change that introduced this URL; no existing issue tracks the 401.

@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — UCC/ucc-test-nvls · commit 5ad8565e

TL;DR: The build failed while building the build_helper Docker image: the scctl install step curls https://artifactory.nvidia.com/artifactory/sw-swx-lab-infra-generic-local/scctl/get-app.sh and Artifactory returned HTTP 401, aborting the image build. This is a CI/infra auth-or-path problem in .ci/dockerfiles/Dockerfile.build_helper, unrelated to the PR's code changes.

Full analysis

Summary: Stage "Build Docker" (x86_64 build_helper image) failed at STEP 12/18, which caused the Parallel stage and then the whole ucc-test-nvls build (plus the slurm job stop) to fail.

Root cause: The Dockerfile pipes an Artifactory-hosted installer into bash:
RUN curl -s https://artifactory.nvidia.com/artifactory/sw-swx-lab-infra-generic-local/scctl/get-app.sh | bash -s -- --output-dir /usr/local/bin
The log shows:

Error: Failed to get latest version: curl: (22) The requested URL returned error: 401
subprocess exited with status 1
Error: building at STEP "RUN curl -s .../scctl/get-app.sh | bash ..." : exit status 1

A 401 means anonymous read on sw-swx-lab-infra-generic-local is no longer permitted (or the repo/path moved and now resolves to a protected endpoint). The build was run with --no-cache ("Forcing building file per user request"), so no cached layer masked it. The parallel aarch64 branch, which doesn't need this step for the main UCC image, built and pushed fine — confirming the failure is isolated to this unauthenticated download, not to the tl-cuda-ipc-fallback code (TL/CUDA compiled and linked cleanly in the aarch64 branch). No large timestamp gaps: the failure happened ~23s into the step, so this is not a hang or timeout.

Implicated commit: 8c1a64ff — "CI: update scctl repository location (#1365)", Daniel Pressler (last change to this Dockerfile / this URL)

File: .ci/dockerfiles/Dockerfile.build_helper:23

Suggested fix:

  1. Restore authenticated access for the download — inject an Artifactory identity token at build time via a Jenkins credential and a BuildKit/podman secret (do not bake it into a layer), e.g. RUN --mount=type=secret,id=artifactory_token curl -fsSL -H "Authorization: Bearer $(cat /run/secrets/artifactory_token)" <url> | bash -s -- --output-dir /usr/local/bin. Any existing token in the Jenkins credential store that is returning 401 should be checked and rotated if expired/leaked.
  2. Or confirm with the lab-infra owners that sw-swx-lab-infra-generic-local/scctl/get-app.sh is still the correct, anonymously readable path (the 401 suggests the repo permissions changed after CI: update scctl repository location #1365) and update the URL accordingly.
  3. Harden the step regardless: use curl -fsSL --retry 3 --retry-all-errors and check the exit status before piping to bash, so a transient 4xx/5xx produces a clear error instead of a confusing "Failed to get latest version".
  4. Re-run build CORE: clarify ctx_cfg_modify semantics (#403) #408 after the fix; no change to PR TL/CUDA: fall back for alltoall if buffer has no IPC handle #1380 is required.

Related: #1365 (introduced current scctl URL), PR under test: #1380

@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — UCC/ucc-test-mpi · commit 1ba6d2fa

TL;DR: The ucc-test-mpi build failed while building the build_helper Docker image: step 12 (curl ... scctl/get-app.sh | bash) aborted because Artifactory returned HTTP 401 when the script queried the latest scctl version — an auth/infra problem in the CI Dockerfile, not in PR #1380's code. Fix by giving the image build an authenticated Artifactory credential (or pinning/mirroring scctl in an anonymously readable location).

Full analysis

Summary: Stage "Build Docker" (node 63) failed building .ci/dockerfiles/Dockerfile.build_helper; the pipeline then stopped with "pipeline stop on build_helper".

Root cause: Log evidence from stage 63:

STEP 12/18: RUN curl -s https://artifactory.nvidia.com/artifactory/sw-swx-lab-infra-generic-local/scctl/get-app.sh | bash -s -- --output-dir /usr/local/bin
Error: Failed to get latest version: curl: (22) The requested URL returned error: 401
Error: building at STEP "RUN curl -s .../scctl/get-app.sh ...": exit status 1

The get-app.sh bootstrap script itself downloaded fine, but its internal "latest version" lookup against artifactory.nvidia.com/artifactory/sw-swx-lab-infra-generic-local/scctl/... was rejected with 401 Unauthorized — the repository/path no longer permits anonymous reads (or the path moved). The container build has no Artifactory credentials, so the RUN layer fails deterministically. The build was invoked with --no-cache ("Forcing building file per user request"), so the previously cached layer no longer hid the problem. No timeout/hang is involved: timestamps are continuous (22:24:24 → 22:25:00) and failure is an explicit non-zero exit. The parallel "Build Docker" stage (node 56) that doesn't use this step succeeded, confirming the failure is isolated to this Dockerfile step and unrelated to commit 1ba6d2f / PR #1380.

Implicated commit: 8c1a64f — "CI: update scctl repository location (#1365)", Daniel Pressler (introduced the current sw-swx-lab-infra-generic-local/scctl URL)

File: .ci/dockerfiles/Dockerfile.build_helper:23

Suggested fix:

  1. Short term: make the scctl install authenticated — inject an Artifactory identity token from a Jenkins credential into the build (e.g. docker build --secret id=artifactory,env=ARTIFACTORY_TOKEN and have the RUN step export it so get-app.sh sends the auth header). Do not bake the token into a layer or echo it in the log; if any token was previously printed in CI output, rotate it.
  2. Alternative/robustness: pin a specific scctl version artifact URL instead of relying on the "latest version" lookup endpoint that now 401s, or pre-install scctl into the harbor.mellanox.com/hpcx/x86_64/ubuntu24.04/base image / mirror it in Harbor so the UCC Dockerfile needs no external auth.
  3. Confirm with the infra owners whether sw-swx-lab-infra-generic-local was made private or the scctl path relocated again, and update line 23 accordingly.
  4. Re-run PR TL/CUDA: fall back for alltoall if buffer has no IPC handle #1380's CI after the fix; its commit did not cause this failure.

Related: PR #1365 (CI: update scctl repository location) is the change that introduced this URL; no existing issue found for the 401.

@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — UCC/ucc-test-mpi · commit 5ad8565e

TL;DR: The ucc-test-mpi pipeline never ran any tests — the build_helper Docker image build failed at STEP 12/18 because the unauthenticated curl of the scctl installer from artifactory.nvidia.com returned HTTP 401. This is a CI/infra auth problem in .ci/dockerfiles/Dockerfile.build_helper, unrelated to PR #1380.

Full analysis

Summary: Parallel stage "Build Docker" (harbor.mellanox.com/torch-ucc/x86_64/build_helper:mpi) failed, then "pipeline stop on build_helper" aborted the Slurm job, so no MPI tests executed.

Root cause: Line 23 of the build_helper Dockerfile downloads the scctl bootstrap script anonymously:
RUN curl -s https://artifactory.nvidia.com/artifactory/sw-swx-lab-infra-generic-local/scctl/get-app.sh | bash -s -- --output-dir /usr/local/bin
The build log shows:

Error: Failed to get latest version: curl: (22) The requested URL returned error: 401
subprocess exited with status 1
Error: building at STEP "RUN curl -s https://artifactory.nvidia.com/.../get-app.sh | bash ..." : exit status 1

The sw-swx-lab-infra-generic-local repo now rejects anonymous reads (401 Unauthorized), so the inner "get latest version" request from get-app.sh fails. The build was run with --no-cache ("Forcing building file per user request"), so the previously cached layer could not mask the problem. Note the curl -s | bash pattern also swallows the HTTP status, which is why this surfaces as an opaque mid-script failure. The failure is deterministic infrastructure/credentials breakage, not a code regression from the tl-cuda-ipc-fallback branch (the CUDA IPC fallback change cannot affect a Docker image build).

Implicated commit: 8c1a64f — "CI: update scctl repository location (#1365)", Daniel Pressler (last change to this line; the referenced artifactory path is the one now returning 401). The triggering commit 5ad8565 is not implicated.

File: .ci/dockerfiles/Dockerfile.build_helper:23

Suggested fix:

  1. Short term: restore access by supplying credentials to the download instead of anonymous curl — mount a Jenkins-managed artifactory token as a build secret (e.g. RUN --mount=type=secret,id=art_token curl -s -H "Authorization: Bearer $(cat /run/secrets/art_token)" ...) or pass an identity token via --secret/netrc. Do not bake the token into a layer or an ARG, and have whoever owns the sw-swx-lab-infra-generic-local repo confirm/rotate the credential if the old one expired.
  2. Make the failure legible and avoid silent 401s: replace curl -s | bash with curl -fsSL --retry 3 -o /tmp/get-app.sh <url> && bash /tmp/get-app.sh --output-dir /usr/local/bin so a non-2xx response fails immediately with a clear message.
  3. Alternative if scctl is not needed for every image: pin and vendor a specific scctl version into the internal harbor.mellanox.com base image (or make the install conditional/non-fatal) so test pipelines don't depend on a live external artifactory fetch at image-build time.
  4. Re-run build CORE: get lib attr implementation #131 after the credential fix; no change to PR TL/CUDA: fall back for alltoall if buffer has no IPC handle #1380 is required.

Related: PR #1365 (CI: update scctl repository location) — the commit that introduced this artifactory path; PR #1380 is the trigger but not the cause.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants