Skip to content

WGMMA SS, row-major A: SimX and RTL disagree when the A descriptor's base is not aligned to an LMEM bank row #454

Closed

Description

@eyoon1131

Summary

For a WGMMA with A in shared memory (SS) and row-major (ldm != 0), SimX and the RTL read different data when the A descriptor's base address is not a multiple of one LMEM bank row (NUM_THREADS * XLEN/8 bytes, so 32 B at NT=4, XLEN=64):

  • SimX reads A from the exact byte address in the descriptor.
  • The RTL rounds the base down to the start of its bank row, so every row of A is read from the wrong columns.

So the same kernel passes on simx and gets wrong results on rtlsim (in the reproducer below, every output), with no error or assert. There is no documented alignment rule for SMEM descriptors, and vx_make_smem_desc accepts these bases, so it isn't clear which side is "right". I've listed a few ideas for potential fixes below.

Reproducer

On current master (48fc0069d)

The command, patch and results in this section were all run on current master 48fc0069d, XLEN=64, toolchain v3.0.1. The test is sgemm_tcu_wg with the patch below, which pads A's rows to tileK + A_PAD elements and starts each row A_OFF elements in. A tf32 element is 4 B, so A_OFF=4 is a 16 B base offset.

make -C tests/regression/sgemm_tcu_wg clean
CONFIGS="-DVX_CFG_NUM_THREADS=4 -DVX_CFG_EXT_TCU_ENABLE -DVX_CFG_TCU_WGMMA_ENABLE -DVX_CFG_TCU_TF32_ENABLE -DITYPE=tf32 -DOTYPE=fp32 -DWGMMA_SS -DWGMMA_RMAJOR_A -DA_PAD=28 -DA_OFF=4" \
  make -C tests/regression/sgemm_tcu_wg run-rtlsim
A base offset simx rtlsim
0 B PASS PASS
16 B (A_OFF=4) PASS FAIL, 256 / 256 outputs wrong
32 B (A_OFF=8, one bank row) PASS PASS
Repro patch (tests only)
--- a/tests/regression/sgemm_tcu_wg/kernel.cpp
+++ b/tests/regression/sgemm_tcu_wg/kernel.cpp
@@ -27,8 +27,15 @@
 
   // Shared memory layout: A [cta_M × tileK] then B [tileK × per_warp_N]
   auto smem   = reinterpret_cast<ctx::input_t *>(__local_mem());
+#ifndef A_PAD
+#define A_PAD 0
+#endif
+#ifndef A_OFF
+#define A_OFF 0
+#endif
+  constexpr uint32_t lda = ctx::tileK + A_PAD;
   auto A_smem = smem;
-  auto B_smem = smem + cta_M * ctx::tileK;
+  auto B_smem = smem + cta_M * lda;
 
   ctx::fragment_acc fragC;
   ctx::fill_fragment(fragC, 0);
@@ -46,7 +53,7 @@
       uint32_t r = idx / ctx::tileK;
       uint32_t c = idx % ctx::tileK;
     #ifdef WGMMA_RMAJOR_A
-      A_smem[r * ctx::tileK + c] = pA[(tile_row + r) * K + (k + c)];
+      A_smem[r * lda + A_OFF + c] = pA[(tile_row + r) * K + (k + c)];
     #else
       A_smem[ctx::a_blockmajor_idx(r, c)] = pA[(tile_row + r) * K + (k + c)];
     #endif
@@ -68,14 +75,14 @@
     //   block-major: warp_rank * a_warp_elems (per a_blockmajor_idx packing).
     //   row-major:   warp_rank * xtileM * tileK  (per kernel's m*tileK + k).
   #ifdef WGMMA_RMAJOR_A
-    auto A_warp = A_smem + warp_rank * ctx::xtileM * ctx::tileK;
+    auto A_warp = A_smem + warp_rank * ctx::xtileM * lda + A_OFF;
   #else
     auto A_warp = A_smem + warp_rank * ctx::a_warp_elems;
   #endif
     auto desc_b = vt::vx_make_smem_desc(B_smem, 0); // block-major: stride field unused
 
   #ifdef WGMMA_RMAJOR_A
-    constexpr uint32_t a_ldm = ctx::tileK;  // row-major A: ldm = tileK elements
+    constexpr uint32_t a_ldm = lda;
   #else
     constexpr uint32_t a_ldm = 0;           // block-major A
   #endif
--- a/tests/regression/sgemm_tcu_wg/main.cpp
+++ b/tests/regression/sgemm_tcu_wg/main.cpp
@@ -761,7 +761,10 @@
   auto time_start = std::chrono::high_resolution_clock::now();
 
   std::cout << "start device" << std::endl;
-  uint32_t smem_size = (cta_M * wg_cfg::tileK + wg_cfg::tileK * per_warp_N) * sizeof(itype_t);
+#ifndef A_PAD
+#define A_PAD 0
+#endif
+  uint32_t smem_size = (cta_M * (wg_cfg::tileK + A_PAD) + wg_cfg::tileK * per_warp_N) * sizeof(itype_t);
   vx_event_h launch_ev = nullptr;
   {
     vx_launch_info_t li = {};

Earlier runs (90a626217, not current master)

These runs were made on the older commit 90a626217, not on current master, using a longer version of the same patch. hw/rtl/tcu/VX_tcu_abuf.sv is identical between 90a626217 and 48fc0069d. On the SimX side, sim/simx/tcu/tcu_unit.cpp differs by one line, a perf-counter reset unrelated to operand addressing. So these results should carry over to master. They show the same pattern in other configurations:

  • XLEN=64: fp16 with 4 B and 16 B offsets fails, 1024/1024 outputs wrong.
  • XLEN=32, where a bank row is 16 B: 4 B and 8 B offsets fail for tf32 and fp16, and 16 B offsets pass.
  • Advancing one A descriptor along K (several wgmma_syncs over tileK-wide windows of the same rows, 16 B per step) fails at XLEN=64. A DBG_TRACE_TCU trace shows desc_a=0x200000 and 0x200010 issuing identical LMEM reads.

In every one of these runs, simx passed.

Where the two models differ

  • RTL, hw/rtl/tcu/VX_tcu_abuf.sv:
    • Lines 172–176: desc_a_word_base keeps only the bank-row bits of the base (desc_a_row_base). The low BANK_SEL_BITS are explicitly marked UNUSED.
    • Line 244: in row-major mode, row_word_off_init = step_k * row_k_words. Nothing adds the base's offset within its bank row back in.
    • Line 380: the stripe write takes each row's words from a single bank-row response, and drops any word with src >= BANK_ROW_WORDS. If the offset were added back, a row's slice could cross into the next bank row, and those words would be lost too.
  • SimX, sim/simx/tcu/tcu_unit.cpp:1444: elem_off = row * ldm + col, applied to the full byte base. Nothing is truncated.

The row stride (ldm) is part of the per-row offset, so the RTL does apply it. But a stride that is not a multiple of the 2-word fetch slice (TCU_WG_FEDP_K) fails the same way even with an aligned base, because some rows' slices then cross a bank row and hit the same guard. On current master with the patch above (A_OFF=0, XLEN=64), strides of 5 and 7 words (A_PAD=1, 3) pass on simx and fail on rtlsim with 192 / 256 outputs wrong. Strides of 6 and 8 words (A_PAD=2, 4) pass on both. The October runs on 90a626217 show the same pattern at XLEN=32.

Why it matters

  • Advancing a descriptor along K is the standard pattern, as with Hopper's descriptors and CUTLASS. It steps the base by tileK * sizeof(input_t): 16 B at NT=4/8 and 32 B at NT=16/32. That is a whole bank row only at NT=4, XLEN=32. In every other configuration some steps are misaligned; at NT=4, XLEN=64, for example, every other step lands mid-bank-row.
  • Hit in practice: an earlier version of the WGMMA FlashAttention kernel (flash_tcu_wg, in tests: FlashAttention regression tests (SIMT and WGMMA tiers) #452) staged Q and P with a row stride and pointed each A descriptor at row + k*tileK. It got up to 0.46 relative error on rtlsim at XLEN=64, but was correct on simx. The kernel in tests: FlashAttention regression tests (SIMT and WGMMA tiers) #452 works around this by staging Q and P as dense per-k-tile tiles ([kt][Br][tileK]), so every A base is aligned.
  • CI doesn't cover it:
    • tests/regression/common.mk adds -DWGMMA_RS to every WGMMA build unless SS is asked for. RS loads A into registers and never goes through the abuf.
    • The 7 -DWGMMA_SS cases in ci/testcases/ all use block-major A, which is aligned by construction.
    • No case builds -DWGMMA_RMAJOR_A.

Possible fixes

These are just a few potential fixes for the issue. SimX and the RTL should accept the same descriptors, and both should reject what they don't support rather than diverge silently.

I'm opening this as an issue rather than a PR because choosing between these sets the descriptor contract for WGMMA, and that decision belongs to the WGMMA engine's main contributor.

  1. Require base and stride to be multiples of the K-slice, tileK * sizeof(input_t).

    • RTL change: add the in-row offset to row_word_off_init. A bank row is a power-of-two multiple of the slice, so an aligned slice never crosses a bank row. No second read and no straddle logic.
    • Enforcement: assert the rule in vx_make_smem_desc, using ctx::tileK, and in SimX.
    • Pro:
      • The rule is stated in API terms kernels already use.
      • It doesn't expose the bank count.
      • Advancing a descriptor along K always complies.
    • Con: a little stricter than the hardware needs for strides. The real stride granularity is the internal TCU_WG_FEDP_K slice, 2 words in the configurations tested here.
  2. Honor any 4-byte-aligned base in the RTL, matching SimX.

    • Change: carry the base's word offset within its bank row into row_word_off_init. Fetch a second bank row when a row's slice crosses the boundary.
    • Pro: most general. No contract change, and SimX stays as it is.
    • Con: the straddling case costs an extra LMEM read per affected row, plus the logic to merge it, for no clear use case beyond what option 1 allows.
  3. Require bank-row-aligned bases, and leave the RTL as it is.

    • Change: document the rule, assert it in vx_make_smem_desc, and have SimX assert or model the truncation.
    • Pro: no RTL change.
    • Con:
      • The bank count (NUM_THREADS * XLEN/8 bytes) leaks into the API.
      • Advancing a descriptor along K is ruled out in every configuration except NT=4, XLEN=32.
      • The stride rule still needs stating separately.

With any of these, an SS row-major regression case with an offset base, and ideally a model_parity case, would keep the two models in step. Open PR #436 rewrites VX_tcu_abuf.sv and the descriptor encoding (an 18-bit offset, with the stride in words) but keeps the truncation, so the fix could land on top of that or alongside it.

Environment

Ubuntu 22.04, Vortex master 48fc0069d, TOOLCHAIN_REV=v3.0.1 prebuilt via ci/toolchain_install.sh, Verilator from the same prebuilt drop.

🤖 Written with help from Claude Code

Activity

  1. tinebp commented on Nov 19, 2026

    @tinebp
    Collaborator

    Fixed by #488.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions