Repository navigation
Conversation
|
@0z5a pls update AUTHORS file |
[Medium impact] The PR appears safe to merge within its stated same-allocator scope.
|
| mi->ptr, | ||
| cuda_st, | ||
| cudaGetErrorString(cuda_st)); | ||
| return UCC_ERR_NOT_SUPPORTED; |
There was a problem hiding this comment.
Returning UCC_ERR_NOT_SUPPORTED here can split an alltoallv across different transports, even when every rank uses cudaMallocAsync. On two directly connected GPUs, let rank 0 send nothing and rank 1 send data to rank 0. Rank 0 skips the IPC check because its source length is zero, and proxy_needed == 0 skips its destination check. Rank 1 hits this return and falls back, while rank 0 stays on TL/CUDA and waits forever at its shared barrier.
Make the fallback decision agree across ranks, including ranks with zero source counts.
Knowledge Base Used:
There was a problem hiding this comment.
Right, a rank that sends nothing never runs the check. fe05b72 keeps the fallback for alltoall only, where counts are the same on every rank. Alltoallv returns UCC_ERR_INVALID_PARAM again, as on master, and needs the handle-export follow-up.
There was a problem hiding this comment.
You’re right. The fe05b72 change keeps the fallback limited to alltoall and maps UCC_ERR_NOT_SUPPORTED back to UCC_ERR_INVALID_PARAM for alltoallv, avoiding the rank-divergent selection and barrier hang. The alltoallv limitation is therefore intentional pending handle-export support; my earlier finding no longer applies.
Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.
|
@Sergei-Lebedev Added in bd11694, thanks. |
|
/build |
|
🤖 CI Triage Agent — TL;DR: The codestyle lint failed because the HEAD commit's title is 63 characters, exceeding the repo's 50-character limit; shorten/reword the commit subject and force-push. Full analysisSummary: The "Check commit title" step of the Root cause: Implicated commit: [REDACTED:Hex High Entropy String] — "TL/CUDA: keep alltoallv on the error path without an IPC handle" (PR #1380, branch File: .github/workflows/codestyle.yaml:39 (the Suggested fix: Reword the commit subject to ≤50 characters, keeping the
Then: Since this commit appears to be a follow-up to Related: PR #1380 (openucx/ucc); no existing issue found for this failure.
|
fe05b72 to
4175e2b
Compare
|
Shortened the last commit's title to 46 characters ( |
|
/build |
4175e2b to
00ede25
Compare
|
/build |
|
🤖 CI Triage Agent — TL;DR: The ASAN job's gtest shards fail on Full analysisSummary: Root cause: Evidence from the logs:
Implicated commit: unknown (not PR #1380 / File: Suggested fix:
Related: PR #1380 (#1380) — failing check is unrelated to its change; no existing issue found for
|
ucc_tl_cuda_mem_info_get() exports the user buffer of the alltoall(v) copy-engine algorithms with the legacy cudaIpcGetMemHandle(). That call only accepts cudaMalloc memory and fails with cudaErrorInvalidValue for buffers from cuMemCreate (e.g. PyTorch expandable segments, ncclMemAlloc) or from a stream-ordered memory pool (cudaMallocAsync). The error was mapped to UCC_ERR_INVALID_PARAM, which ucc_coll_init() does not treat as a reason to try the next TL, so the collective failed although TL/UCP could run it. Return UCC_ERR_NOT_SUPPORTED with a debug message instead, and clear the CUDA error state, so that collective init falls back to the next TL in the score map. Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
For alltoallv, whether a buffer is checked depends on the rank's own counts: a rank that sends nothing skips the IPC check. If the other ranks fell back to another TL, that rank would stay on TL/CUDA and hang at its barrier. Map NOT_SUPPORTED back to INVALID_PARAM for alltoallv, as on master, and keep the fallback for alltoall, where counts are the same on every rank. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
00ede25 to
5ad8565
Compare
|
/build |
|
🤖 CI Triage Agent — TL;DR: The x86_64 Full analysisSummary: Stage "Build Docker" (node 78, x86_64 Root cause: The build step The installer script itself downloads, but its unauthenticated Artifactory query for the latest scctl version now returns 401 (unauthorized). This is an infrastructure/credentials regression, not a source change: build #405 of the same job, ~10 hours earlier on the same Dockerfile, ran the identical step successfully ("Downloading scctl version v0.14.1 … Successfully installed scctl version v0.14.1"). The aarch64 UCC image build in the same parallel stage succeeded, so compilation of the PR's code is fine. Because the job forces Implicated commit: No UCC commit — external Artifactory ACL/token change. The step was introduced/relocated in 8c1a64f ("CI: update scctl repository location (#1365)", Daniel Pressler), which is the code that depends on anonymous Artifactory read access. File: .ci/dockerfiles/Dockerfile.build_helper:23 Suggested fix:
Related: #1365 (introduced the current scctl Artifactory location) |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage "Build Docker" (node 78) of UCC/ucc-test-nvls #406 failed building Root cause: The Dockerfile step Implicated commit: Not the triggering commit 5ad8565 (tl-cuda-ipc-fallback). Last change to this step: File: Suggested fix:
Related: PR #1365 (CI: update scctl repository location) — the change that introduced the current Artifactory path; PR #1380 is the build trigger but not the cause. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage "Build Docker" (node 63) failed after 46s: Root cause: The Implicated commit: 8c1a64f — "CI: update scctl repository location (#1365)", Daniel Pressler (last change to the failing line; the repo it points at now requires auth) File: Suggested fix:
Related: PR #1365 (#1365) changed the scctl repository location; PR #1231 (#1231) introduced the scctl/Slurm-based CI flow. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage "Build Docker" (node 54) of UCC/ucc-test-gtest #216 failed while building Root cause: Infrastructure/credential regression, not a code defect. In build #216: The identical step with the identical Dockerfile succeeded one day earlier in build #215 ("Downloading scctl version v0.14.1 / Successfully installed scctl version v0.14.1"), so the Artifactory repo Implicated commit: No code commit caused this. The step was last touched by File: Suggested fix:
Related: PR #1365 (introduced the scctl repository location); no existing issue tracks the 401 — worth opening one against the CI infra. PR #1380 is unrelated to this failure. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage "Build Docker" (node 54) for Root cause: Infrastructure/credential failure, not a code regression from PR #1380. The Dockerfile fetches the scctl installer anonymously: The Implicated commit: 8c1a64f — "CI: update scctl repository location (#1365)", Daniel Pressler (changed the scctl Artifactory URL; the new location is not anonymously readable). The PR head commit 1ba6d2f is unrelated. File: Suggested fix:
Related: PR #1365 (CI: update scctl repository location) is the change that introduced this URL; no existing issue found for the 401. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Root cause: The image is rebuilt with The Implicated commit: Not the PR commit 5ad8565. The fragile step was introduced/changed by File: Suggested fix:
Related: #1365 (introduced the current scctl repository URL); PR under test: #1380 |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage "Build Docker" (x86_64 Root cause: Line 23 of Secondary (symptom only): the post-failure Implicated commit: 8c1a64f — Daniel Pressler, "CI: update scctl repository location (#1365)" (changed the scctl Artifactory URL); the triggering commit 1ba6d2f is not implicated. File: Suggested fix:
Related: PR #1365 (scctl repository location change, the implicated CI commit). No existing issue tracks this 401; worth opening one. PR #1380 is unrelated to the failure and can be re-run once the image build is fixed. |
|
🤖 CI Triage Agent — TL;DR: The build died while building the Full analysisSummary: Stage "Build Docker" (node 64, x86_64 Root cause: The Dockerfile pipes an Artifactory-hosted installer into bash with no authentication: Implicated commit: 8c1a64f — "CI: update scctl repository location (#1365)", Daniel Pressler (last change to this File: Suggested fix:
Related: PR #1365 ( |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage 54 "Build Docker" (build_helper image) aborted at Root cause: From the stage-54 log: The Artifactory generic repo Implicated commit: File: Suggested fix:
Related: PR #1365 ( |
|
🤖 CI Triage Agent — TL;DR: The build failed while building the Full analysisSummary: Stage "Build Docker" (x86_64 Root cause: The Dockerfile pipes an Artifactory-hosted installer into bash: A 401 means anonymous read on Implicated commit: File: Suggested fix:
Related: #1365 (introduced current scctl URL), PR under test: #1380 |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage "Build Docker" (node 63) failed building Root cause: Log evidence from stage 63: The Implicated commit: 8c1a64f — "CI: update scctl repository location (#1365)", Daniel Pressler (introduced the current File: Suggested fix:
Related: PR #1365 ( |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Parallel stage "Build Docker" ( Root cause: Line 23 of the build_helper Dockerfile downloads the scctl bootstrap script anonymously: The Implicated commit: 8c1a64f — "CI: update scctl repository location (#1365)", Daniel Pressler (last change to this line; the referenced artifactory path is the one now returning 401). The triggering commit 5ad8565 is not implicated. File: .ci/dockerfiles/Dockerfile.build_helper:23 Suggested fix:
Related: PR #1365 (CI: update scctl repository location) — the commit that introduced this artifactory path; PR #1380 is the trigger but not the cause. |
CLA: I have signed the UCF CLA (confirmed earlier on openucx/ucx#12223).
What
Part of #1378: alltoall only.
If
cudaIpcGetMemHandle()rejects a buffer inucc_tl_cuda_mem_info_get(), returnUCC_ERR_NOT_SUPPORTEDwith a debug message instead ofUCC_ERR_INVALID_PARAMwith an error message. Also clear the CUDA error state. For alltoall,ucc_coll_init()then moves on to the next TL in the score map (TL/UCP in my GH200 build; TL/NCCL in my B300 build, which includes it). Alltoallv maps the result back toUCC_ERR_INVALID_PARAMwith an error message, as on master.Why ?
Legacy IPC handles exist only for
cudaMallocmemory. With buffers fromcuMemCreate(PyTorch expandable segments,ncclMemAlloc) orcudaMallocAsync, TL/CUDA alltoall failed at init, even though TL/UCP can run it.Alltoallv keeps failing: whether its buffers are checked depends on the rank's own counts. A rank that sends nothing skips the check (and, without a proxy, the destination check too), so it would stay on TL/CUDA while the other ranks fall back, and hang at its barrier. Alltoall counts are the same on every rank, so all ranks decide the same way.
Reproducer from #1378, 2 ranks, alltoall with 1 MiB per peer, results checked on the host:
cudaMalloccudaMallocAsyncInvalid parametercuMemCreate(POSIX fd handle)Invalid parameterWith
UCC_LOG_LEVEL=debugthe fallback shows as (GH200):How ?
ucc_tl_cuda_mem_info_get()changes only in its error path. The team scratch buffer goes through the same function, but it is alwayscudaMallocmemory, and its caller still logsfailed to get scratch memory infoon any error.alltoallv_ce.cwraps the call and turnsUCC_ERR_NOT_SUPPORTEDback intoUCC_ERR_INVALID_PARAM.cuMemExportToShareableHandle/cudaMemPoolExportPointerand keep them on TL/CUDA. I can follow up with that if it is preferred.pushalgorithms are opt-in, and this copy-engine path is unchanged there.