Skip to content

tcu: WGMMA shared-memory descriptor contract; CTA LMEM slot and DXA stride alignment - #488

Open
tinebp wants to merge 3 commits into
masterfrom
fix/tcu-desc-contract
Open

tinebp wants to merge 3 commits into
masterfrom
fix/tcu-desc-contract

Conversation

@tinebp

@tinebp tinebp commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

1. WGMMA shared-memory descriptor contract (fixes #454)

  • Before, an unaligned descriptor base was silently rounded down in the RTL.
  • The alignment rule is defined once, in sw/common/tensor_cfg.h (smem_desc_check, wgmma_slice_words):
    • a strided operand's base is a multiple of the K-slice, and so is its stride;
    • a block-major base starts an LMEM bank row.
  • SimX aborts with a message naming the broken rule. wgmma_sync asserts both descriptors in debug kernel builds. The RTL tile buffers gain simulation-only asserts for the block-major rule.

2. CTA LMEM slot alignment

  • When an LMEM DMA agent is present (DXA or WGMMA), each CTA's LMEM slot is aligned to a bank row. Otherwise the alignment stays MEM_BLOCK_SIZE.
  • This matters where a bank row is wider than 64 B. There, the second co-resident CTA's operands started mid-row, and the RTL computed wrong results.
  • There is one constant per side: LMEM_SLOT_ALIGN in VX_gpu_pkg, and sw/common/lmem_slot.h for SimX and the runtime.
  • Default configs, and the perf baselines, are unchanged.

3. DXA multicast stride

  • The RTL now rounds the multicast smem_stride up to the slot pitch when it latches the descriptor, as SimX already did.

Impact and evidence

Impact class: R3 (ci/change_impact.py classify). Maintainer approval needed.

Contracts touched:

  • sw/common/tensor_cfg.h, installed: adds smem_desc_check, smem_desc_fault and wgmma_slice_words.
  • vx_tensor.h: the wgmma_context layout changes; asserts only.
  • dxa.h: smem_stride is now the per-CTA LMEM size, rounded up by the device. Callers that already pass the pitch see no change. No in-tree, PoCL or Mesa caller passes anything else.
evidence status detail
functional tests (simx / rtlsim / xrt) run New sgemm_tcu_wg-lmem-pad passes on SimX and rtlsim. With the KMU rounding reverted, it fails the tile buffer's bank-row assert. sgemm_tcu_wg_dxa_mcast-pad passes on SimX and rtlsim. Also passing on rtlsim: sgemm_tcu_wg-9, -b-offset, sgemm_tcu_wg_sp-8, sgemm_tcu_wg_dxa-lmem128k. Also passing on SimX: a-offset, -2, -10, flash_attn_tcu_wg-fp16, cta_cluster-1.
model_parity n/a the RTL now matches the SimX stride rounding
perf_gate not run the pitch is unchanged in every baseline config (XLEN 32, NT ≤ 16)
Beacon logic_gate run DXA stride: +11 LUT, depth unchanged. The asserts are simulation-only.
synthesis not run
XLEN 64 build run rtlsim -Wall lint at XLEN 32 and 64 (standard matrix, TCU WGMMA, sparse, DXA) and the 32-thread WGMMA config; SimX with -DSIMX_FUNCTIONAL
  • No test silenced.
  • No golden data changed.
  • RTL and SimX agree on the pitch and the stride rounding.
  • No toolchain pin change.

🤖 Generated with Claude Code

tinebp and others added 3 commits November 19, 2026 03:37
…contract

#475 made row-major A and K-major B honour a base inside a bank row, but
the tile buffers still mishandle some descriptors silently, while SimX
reads the exact byte address, so the two disagreed with no error:
- a row-major/K-major row is fetched one LMEM access per row, and words of
  its K-slice past the bank row are dropped (abuf/bbuf write guards), so a
  base or stride that is not a multiple of the slice reads stale data;
- the in-row offset is the byte offset >> 2, so a 2-byte (fp16/bf16) base
  is truncated;
- a block-major operand (stride 0) uses only the bank-row address, so an
  unaligned base is rounded down.

The rule, defined once in tensor_cfg.h (smem_desc_check) and shared by
SimX and the kernel library: for a strided operand the base is a multiple
of 4*slice bytes and the stride a multiple of slice words, where slice is
fedpK words dense, TC_K (A) or 2*TC_K (B) sparse; a block-major base is
LMEM bank-row aligned. The slice divides a bank row in every legal
configuration (static_assert in SimX). The rule is slightly stricter than
the RTL needs in some geometries, so that legality does not depend on bank
geometry; any multiple of one tile row satisfies it everywhere.

SimX now aborts with a message naming the rule (both descriptor decode
paths share decode_descs), wgmma_sync asserts both descriptors in debug
kernel builds, vx_make_smem_desc asserts the documented 4-byte stride
multiple, and the WGMMA design doc gains the alignment table.
sgemm_tcu_wg gains an A_OFF_ELEMS knob (default 0, no codegen change):
a 2-byte and a 4-byte misaligned fp16 base now abort in SimX, a one-slice
offset passes. The legal CI cases still pass (a-offset, b-offset,
sgemm_tcu_wg-2, flash_attn_tcu_wg-fp16, sgemm_tcu_wg_sp-fedp2k-ss,
sgemm_tcu_wg_dxa-1, SimX XLEN 32).

The RTL gains simulation-only asserts for the block-major part of the rule
in the A and B tile buffers, whose fetch drops the in-row offset, so a
kernel that breaks it fails loudly on rtlsim as it does on SimX instead of
reading the wrong rows. No synthesized logic changes. The rtlsim cases
sgemm_tcu_wg-9, -b-offset, sgemm_tcu_wg_sp-8, sgemm_tcu_wg_dxa-lmem128k and
sgemm_tcu_wg_dxa_mcast-pad pass with them.

Fixes #454.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…is present

The KMU and the SimX dispatcher pitch the per-CTA LMEM slots at the CTA's
LMEM size rounded up to MEM_BLOCK_SIZE (64 B), so a CTA's __local_mem()
base was only 64 B aligned. The LMEM DMA agents move whole bank rows
(LMEM_NUM_BANKS x XLEN/8 bytes): a block-major WGMMA operand must start a
bank row, and DXA multicast steps receivers by whole rows. Where a bank row
is wider than 64 B (more than 16 banks at XLEN 32, 8 at XLEN 64), the second
co-resident CTA's operands started mid-row: the RTL rounded them down and
computed wrong results, and SimX now rejects the descriptor.

The slot alignment is now max(MEM_BLOCK_SIZE, bank row) when DXA or WGMMA
is enabled (LMEM_DMA_EN), and MEM_BLOCK_SIZE otherwise, so every default
and narrow configuration keeps its pitch. One constant per side: RTL
LMEM_SLOT_ALIGN in VX_gpu_pkg (used by the KMU rounding), and
sw/common/lmem_slot.h for the SimX dispatcher, the SimX DXA multicast stride
and the runtime occupancy estimate. Derived from existing config defines,
so the resolved config and the perf baselines are unchanged.

sgemm_tcu_wg gains an SMEM_PAD_BYTES knob (default 0) and a CI case on
SimX and rtlsim, sgemm_tcu_wg-lmem-pad: 32 threads, two CTAs per core,
64 B pad. It aborted on the descriptor check before and passes now; on
rtlsim it passes with the same instruction count, and with the KMU
rounding reverted to MEM_BLOCK_SIZE it fails the tile buffer's bank-row
assert. sgemm_tcu_wg-2/-10, sgemm_tcu_wg_dxa_mcast-1/-3 and
cta_cluster-1 still pass on SimX.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… RTL

The host programs the multicast stride as the per-CTA LMEM size, and the
receivers sit one CTA slot pitch apart. SimX rounded the stride up to the
slot pitch at DCR write, but the RTL used the value as given (truncated to
bank-row words), so the two agreed only when the host happened to pass an
already-aligned size. With a 400 B footprint (448 B pitch) the RTL stepped
receivers by 400 B and every receiver CTA computed garbage, while SimX
passed.

VX_dxa_setup now rounds the stride to LMEM_SLOT_ALIGN when the descriptor
is latched, the same rounding the KMU applies to the slot pitch. The adder
sits on the latch from the descriptor RAM's registered output, off the
per-beat path. Beacon (dxa): depth unchanged (15, mean-50 9.4, 40 registers
at 8+ levels), +11 LUT. dxa.h and the multicast design doc define the
stride as the per-CTA LMEM size.

sgemm_tcu_wg_dxa_mcast gains a runtime -p LMEM pad (default 0) and now
passes its LMEM size as the stride; new CI case sgemm_tcu_wg_dxa_mcast-pad
(SimX and rtlsim, -p 16) fails on rtlsim before this change and passes
after, with SimX's instruction count. sgemm_tcu_wg_dxa_mcast-1/-2 and
cta_cluster-1 still pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WGMMA SS, row-major A: SimX and RTL disagree when the A descriptor's base is not aligned to an LMEM bank row

1 participant