Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions ci/testcases/dxa.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,17 @@ tests:
app: sgemm_tcu_wg_dxa_mcast
configs: -DVX_CFG_EXT_TCU_ENABLE -DVX_CFG_EXT_DXA_ENABLE -DVX_CFG_TCU_TYPE=DPI
args: -m 128 -n 64 -k 32
# A per-CTA LMEM size that is not a slot-pitch multiple, passed as the multicast
# stride: the device rounds it to the pitch the receivers sit at.
- id: sgemm_tcu_wg_dxa_mcast-pad
touches: [hw/rtl/dxa, sim/simx/dxa, hw/rtl/VX_kmu.sv, sim/simx/cta_dispatcher.cpp, sw/common/lmem_slot.h]
via: blackbox
drivers:
- simx
- rtlsim
app: sgemm_tcu_wg_dxa_mcast
configs: -DVX_CFG_EXT_TCU_ENABLE -DVX_CFG_EXT_DXA_ENABLE -DVX_CFG_TCU_TYPE=DPI
args: -p 16
- id: sgemm_tcu_wg_dxa_mcast-mc
touches: [hw/rtl/VX_kmu.sv, hw/rtl/interfaces/VX_kmu_bus_if.sv,
hw/rtl/core/VX_kmu_bus_arb.sv, hw/rtl/core/VX_cta_dispatch.sv,
Expand Down
16 changes: 16 additions & 0 deletions ci/testcases/tensor_wg.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,22 @@ tests:
app: sgemm_tcu_wg
configs: -DVX_CFG_NUM_THREADS=8 -DVX_CFG_NUM_WARPS=8 -DVX_CFG_ISSUE_WIDTH=4 -DVX_CFG_EXT_TCU_ENABLE -DVX_CFG_TCU_WGMMA_ENABLE
-DITYPE=fp16 -DOTYPE=fp32 -DWGMMA_NRC=8 -DWGMMA_SS -DWGMMA_KMAJOR_B -DB_PAD_K=1 -DB_OFF_K=1 -DVX_CFG_NUM_ALU_BLOCKS=1 -DVX_CFG_NUM_LSU_BLOCKS=1 -DVX_CFG_NUM_FPU_BLOCKS=1
# Two co-resident CTAs whose LMEM footprint is not a bank-row multiple (128 B
# at 32 threads): the second CTA's block-major operands must still start a bank
# row, so the dispatcher's slot pitch keeps the bank-row alignment.
- id: sgemm_tcu_wg-lmem-pad
via: blackbox
drivers:
- simx
- rtlsim
xlen:
- 32
touches: [hw/rtl/VX_kmu.sv, hw/rtl/VX_gpu_pkg.sv, sim/simx/cta_dispatcher.cpp, sw/common/lmem_slot.h,
sim/simx/tcu, sw/kernel/include/vx_tensor.h]
app: sgemm_tcu_wg
args: -m 128 -n 128 -k 32
configs: -DVX_CFG_NUM_THREADS=32 -DVX_CFG_NUM_WARPS=8 -DVX_CFG_ISSUE_WIDTH=4 -DVX_CFG_EXT_TCU_ENABLE -DVX_CFG_TCU_WGMMA_ENABLE
-DITYPE=fp16 -DOTYPE=fp32 -DWGMMA_NRC=8 -DSMEM_PAD_BYTES=64 -DVX_CFG_NUM_ALU_BLOCKS=1 -DVX_CFG_NUM_LSU_BLOCKS=1 -DVX_CFG_NUM_FPU_BLOCKS=1

- id: sgemm_tcu_wg_sp-1
via: blackbox
Expand Down
15 changes: 14 additions & 1 deletion docs/designs/cta_dispatch_architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ envelope plus a `kind`-discriminated union:
| `PC`, `entry` | program image PC and kernel entry PC |
| `param` | kargs pointer |
| `ctx_id` | 8-bit launch identity, bumped every `start` |
| `aligned_lmem_size` | per-CTA LMEM footprint, rounded to `MEM_BLOCK_SIZE` |
| `aligned_lmem_size` | per-CTA LMEM footprint, rounded to `LMEM_SLOT_ALIGN` |

`args.compute` ([`:735-744`](../../hw/rtl/VX_gpu_pkg.sv#L735)) carries
`grid_dim[3]`, `block_idx[3]`, `block_dim[3]`, `block_size`, `warp_step[3]`,
Expand Down Expand Up @@ -376,6 +376,19 @@ receiver-side translation. The stride must be `MEM_BLOCK_SIZE`-aligned (the
KMU rounds it) because LMEM is block-addressed; a ragged stride would target
the wrong block.

When the core has an LMEM DMA agent (`VX_CFG_EXT_DXA_ENABLE` or
`VX_CFG_TCU_WGMMA_ENABLE`) the KMU rounds the stride to a whole LMEM bank row
(`LMEM_NUM_BANKS × XLEN/8` bytes) instead, when that is larger: the agents
move whole bank rows, so DXA multicast steps receivers by whole rows and a
block-major WGMMA operand at `__local_mem()` must start one
([tensor_core_wgmma_engine.md §6.1](tensor_core_wgmma_engine.md)). This is
`LMEM_SLOT_ALIGN` in `VX_gpu_pkg`, mirrored for SimX and the runtime by
[`lmem_slot.h`](../../sw/common/lmem_slot.h). The bank row exceeds
`MEM_BLOCK_SIZE` only when `LMEM_NUM_BANKS` x XLEN/8 does: with the default
64-byte block, more than 16 LMEM banks at XLEN 32 or more than 8 at XLEN 64
(one bank per thread by default), so narrower cores keep the `MEM_BLOCK_SIZE`
pitch.

`get_cluster_rank()` is derived, never sent:
`CTA_ID % CTA_CLUSTER_SIZE` ([`vx_spawn2.h:133`](../../sw/kernel/include/vx_spawn2.h#L133)).

Expand Down
4 changes: 3 additions & 1 deletion docs/designs/dxa_async_copy_multicast.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,9 @@ There is no software→global path; DXA is GMEM-read / LMEM-write only.
([`VX_dxa_pkg.sv:54`](../../hw/rtl/dxa/VX_dxa_pkg.sv#L54)): `base_addr` +
`meta` + packed tile sizes (`tile01/tile23/tile4`) + `cfill` +
`size0..4` (OOB bounds) + `stride0..3` (byte row strides) +
`smem_stride` (per-CTA LMEM stride for multicast). Programmed by the
`smem_stride` (per-CTA LMEM stride for multicast: the host writes the
kernel's per-CTA LMEM size, and the setup stage rounds it up to the CTA
slot pitch, `LMEM_SLOT_ALIGN`, as the KMU rounds the slots). Programmed by the
host over a DCR block `[dcr_dxa]` at 0x100, 16 descriptors
([`VX_types.toml:87-123`](../../VX_types.toml#L87)).
- **META subfields** `[dxa_desc_meta]`: `DIM` (3b), `ELEMSZ` (2b),
Expand Down
20 changes: 19 additions & 1 deletion docs/designs/tensor_core_wgmma_engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,24 @@ stride folds into the packing, so building a descriptor costs one OR.
Row-major is the layout of a DXA-loaded slab. The buffers' output muxes are
layout-agnostic; only the fetch differs.

**Alignment.** Each LMEM access returns one bank row
(`NUM_THREADS × XLEN/8` bytes), and the buffers take a row's words from that
one access only, so a descriptor must keep every fetch inside a bank row:

| layout | rule |
|---|---|
| block-major | the offset is a multiple of a bank row |
| row-major | the offset (in bytes) and the stride (in words) are multiples of the operand's **K-slice**: `fedpK` words, or for sparse `tcK` words of A and `2·tcK` of B |

The K-slice is the part of a row one micro-op consumes. It divides a bank
row in every configuration, so an aligned slice never straddles two. One
tile row (`tileK` elements, `tileK/2` for sparse A) is a multiple of the
K-slice everywhere, so a descriptor stepped along K by `tileK` stays legal.
`smem_desc_check` in [`tensor_cfg.h`](../../sw/common/tensor_cfg.h) states the
rule once: SimX aborts on a descriptor that breaks it rather than read the
exact address the hardware cannot, and `wgmma_sync` asserts it in debug
kernel builds.

### 6.2 A buffer — `VX_tcu_abuf`

One per block. It holds one **k-stripe**: the `M_STEPS` A blocks of the
Expand Down Expand Up @@ -525,7 +543,7 @@ constant on both sides.
| API | purpose |
|---|---|
| `wmma_context`, `wgmma_context` | compile-time tile geometry and the instruction emitters for a given pair of formats |
| `vx_make_smem_desc(ptr, leading_bytes)` | build a shared-memory descriptor; a zero stride selects block-major |
| `vx_make_smem_desc(ptr, leading_bytes)` | build a shared-memory descriptor; a zero stride selects block-major; `ptr` and `leading_bytes` follow the alignment rule of §6.1 |
| `load_sp_metadata`, `load_mx_metadata` | emit the `TCU_LD` instructions for a fragment |
| `a_blockmajor_idx`, `b_blockmajor_idx`, `a_sp_blockmajor_idx` | where element `(r, c)` lives in a block-major slab |

Expand Down
6 changes: 6 additions & 0 deletions hw/rtl/VX_gpu_pkg.sv
Original file line number Diff line number Diff line change
Expand Up @@ -2013,6 +2013,12 @@ package VX_gpu_pkg;
localparam LMEM_DMA_IN_TAG_MAX = `MAX(DXA_LMEM_OUT_TAG_W, TCU_LMEM_TAG_W);
localparam LMEM_DMA_TAG_WIDTH = LMEM_DMA_IN_TAG_MAX + `ARB_SEL_BITS(LMEM_DMA_INPUTS, 1);

// Alignment of every CTA's LMEM slot, and so of its LMEM base. A DMA agent
// moves whole bank rows: a block-major WGMMA operand must start one, and
// DXA multicast steps receivers by whole rows.
localparam LMEM_SLOT_ALIGN = LMEM_DMA_EN ? `MAX(`VX_CFG_MEM_BLOCK_SIZE, LMEM_DMA_DATA_SIZE)
: `VX_CFG_MEM_BLOCK_SIZE;

////////////////////////// Icache Parameters //////////////////////////////

// Word size in bytes
Expand Down
7 changes: 4 additions & 3 deletions hw/rtl/VX_kmu.sv
Original file line number Diff line number Diff line change
Expand Up @@ -84,14 +84,15 @@ module VX_kmu import VX_gpu_pkg::*; import VX_trace_pkg::*; #(
// cluster_size : K = cluster_dim product, bounded by NUM_WARPS (cluster
// members are co-resident on one core), so NW_WIDTH+1 bits
// suffice; each dim is sliced to that width.
// aligned_lmem_size : per-CTA LMEM footprint rounded up to MEM_BLOCK_SIZE.
// aligned_lmem_size : per-CTA LMEM footprint rounded up to LMEM_SLOT_ALIGN.
// lmem_slots : the CTA slots that footprint fits (lmem_slots_fit).
// Registered so the broadcast path is flop-driven. All three settle one
// cycle after their DCR write, the same cycle, and DCRs are written before
// `start`, so all are settled before the first CTA is dispatched.
`STATIC_ASSERT(`IS_POW2(LMEM_SLOT_ALIGN), ("LMEM_SLOT_ALIGN must be a power of two"))
wire [`VX_CFG_LMEM_LOG_SIZE:0] aligned_lmem_size_w =
((`VX_CFG_LMEM_LOG_SIZE+1)'(dcr_lmem_size) + (`VX_CFG_LMEM_LOG_SIZE+1)'(`VX_CFG_MEM_BLOCK_SIZE - 1))
& ~((`VX_CFG_LMEM_LOG_SIZE+1)'(`VX_CFG_MEM_BLOCK_SIZE - 1));
((`VX_CFG_LMEM_LOG_SIZE+1)'(dcr_lmem_size) + (`VX_CFG_LMEM_LOG_SIZE+1)'(LMEM_SLOT_ALIGN - 1))
& ~((`VX_CFG_LMEM_LOG_SIZE+1)'(LMEM_SLOT_ALIGN - 1));
reg [NW_WIDTH:0] cluster_size_r;
reg [`VX_CFG_LMEM_LOG_SIZE:0] aligned_lmem_size_r;
reg [`VX_CFG_NUM_WARPS-1:0] lmem_slots_r;
Expand Down
7 changes: 4 additions & 3 deletions hw/rtl/core/VX_cta_dispatch.sv
Original file line number Diff line number Diff line change
Expand Up @@ -476,9 +476,10 @@ module VX_cta_dispatch import VX_gpu_pkg::*; #(

localparam LMEM_LOG = `VX_CFG_LMEM_LOG_SIZE;

// Per-CTA LMEM footprint, block-aligned to MEM_BLOCK_SIZE by the KMU (DXA
// multicast resolves receiver addresses as issuer_addr + r*smem_stride;
// a non-aligned stride would target the wrong block).
// Per-CTA LMEM footprint, aligned to LMEM_SLOT_ALIGN by the KMU, so every
// slot base is too (DXA multicast resolves receiver addresses as
// issuer_addr + r*smem_stride; a non-aligned stride would target the wrong
// block).
wire [LMEM_LOG:0] stride = kmu_req.aligned_lmem_size;
wire is_first_of_cluster = eff_is_first;

Expand Down
7 changes: 6 additions & 1 deletion hw/rtl/dxa/VX_dxa_setup.sv
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,11 @@ module VX_dxa_setup import VX_gpu_pkg::*, VX_dxa_pkg::*; (
wire [31:0] dec_tile3 = rank_ge4 ? 32'(desc_data.tile23[31:16]) : 32'd1;
wire [31:0] dec_tile4 = rank_ge5 ? desc_data.tile4 : 32'd1;

// The host programs the multicast stride as the per-CTA LMEM size; the
// receivers sit one slot pitch apart, so round it as the KMU rounds the pitch.
wire [31:0] desc_smem_pitch = (desc_data.smem_stride + 32'(LMEM_SLOT_ALIGN - 1))
& ~32'(LMEM_SLOT_ALIGN - 1);

wire [31:0] dec_stride0 = rank_ge2 ? desc_data.stride0 : 32'd0;
wire [31:0] dec_stride1 = rank_ge3 ? desc_data.stride1 : 32'd0;
wire [31:0] dec_stride2 = rank_ge4 ? desc_data.stride2 : 32'd0;
Expand Down Expand Up @@ -465,7 +470,7 @@ module VX_dxa_setup import VX_gpu_pkg::*, VX_dxa_pkg::*; (
`endif
s_is_multicast <= ($countones(req_data.cta_mask) > 1);
s_cta_mask <= req_data.cta_mask;
s_smem_stride <= desc_data.smem_stride;
s_smem_stride <= desc_smem_pitch;
s_initial_smem_base <= req_data.smem_addr;
s_cfill <= desc_data.cfill;
s_dest_kmajor <= dec_dest_kmajor;
Expand Down
8 changes: 5 additions & 3 deletions hw/rtl/dxa/VX_dxa_smem_wr.sv
Original file line number Diff line number Diff line change
Expand Up @@ -699,9 +699,11 @@ module VX_dxa_smem_wr import VX_gpu_pkg::*, VX_dxa_pkg::*; #(

// Base word addr per beat: km_word_addr in K-major (per-elem scatter),
// fb_word_addr_r in row-major (streamed). Replay adds the per-receiver
// offset accumulated in beat_offset_r (the dispatcher guarantees smem_stride
// is word-aligned — a multiple of MEM_BLOCK_SIZE ≥ SMEM_WORD_SIZE — so the
// byte→word shift in stride_words is lossless).
// offset accumulated in beat_offset_r (VX_dxa_setup rounds smem_stride to
// the CTA slot pitch, a multiple of LMEM_SLOT_ALIGN, itself a multiple of
// SMEM_WORD_SIZE, so the byte→word shift in stride_words is lossless).
`STATIC_ASSERT((LMEM_SLOT_ALIGN % SMEM_WORD_SIZE) == 0,
("the CTA slot pitch must be a whole number of SMEM words"))
wire [SMEM_ADDR_WIDTH-1:0] base_word_addr = scatter ? km_word_addr : fb_word_addr_r;
wire [SMEM_ADDR_WIDTH-1:0] replay_addr = base_word_addr + beat_offset_r;
// Only the low SMEM_ADDR_WIDTH+SMEM_OFF_W bits of smem_stride are used.
Expand Down
7 changes: 7 additions & 0 deletions hw/rtl/tcu/VX_tcu_abuf.sv
Original file line number Diff line number Diff line change
Expand Up @@ -502,6 +502,13 @@ module VX_tcu_abuf import VX_gpu_pkg::*, VX_tcu_pkg::*; #(
end
`endif

// A block-major A operand must start an LMEM bank row: the fetch drops the
// offset inside the row.
`RUNTIME_ASSERT (~(req_valid && req_a_is_smem && is_first_compute_uop && (desc_a_ldm_words == '0)
&& (req_desc_a[BANK_ROW_WORDS_LOG2+1:0] != '0)),
("%t: *** %s: block-major A base 0x%0h is not bank-row aligned", $time, INSTANCE_ID,
req_desc_a[TCU_SMEM_DESC_OFF_BITS-1:0]))

endmodule

`endif // VX_CFG_TCU_WGMMA_ENABLE
7 changes: 7 additions & 0 deletions hw/rtl/tcu/VX_tcu_bbuf.sv
Original file line number Diff line number Diff line change
Expand Up @@ -707,6 +707,13 @@ module VX_tcu_bbuf import VX_gpu_pkg::*, VX_tcu_pkg::*; #(
end
`endif

// A block-major B operand must start an LMEM bank row: the fetch drops the
// offset inside the row.
`RUNTIME_ASSERT (~(req_valid && ~req_wants_kmajor
&& (req_desc_b[BANK_ROW_WORDS_LOG2+1:0] != '0)),
("%t: *** %s: block-major B base 0x%0h is not bank-row aligned", $time, INSTANCE_ID,
req_desc_b[TCU_SMEM_DESC_OFF_BITS-1:0]))

endmodule

`endif // VX_CFG_TCU_WGMMA_ENABLE
16 changes: 8 additions & 8 deletions sim/simx/cta_dispatcher.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@

#include <VX_types.h>
#include "cta_dispatcher.h"
#include <lmem_slot.h>
#include "core.h"
#include <VX_config.h>
#include <cassert>
Expand Down Expand Up @@ -142,14 +143,13 @@ uint32_t CtaDispatcher::step(const WarpMask& active_warps, uint32_t* wids, cta_w
return 0;
const kmu_req_t& next = bus_in.peek();

// Fixed-stride admission. Round the per-CTA LMEM size up to a
// MEM_BLOCK_SIZE multiple — this uniform stride is the slot pitch.
// Multicast resolves receiver destinations as `issuer_base + r * stride`
// and the LMEM model is block-addressed (byteen-masked), so a non-aligned
// stride would truncate the destination; the descriptor handler rounds
// its stride the same way to stay consistent.
uint32_t stride = (next.lmem_size + VX_CFG_MEM_BLOCK_SIZE - 1u)
& ~uint32_t(VX_CFG_MEM_BLOCK_SIZE - 1u);
// Fixed-stride admission. Round the per-CTA LMEM size up to the slot
// alignment — this uniform stride is the slot pitch. Multicast resolves
// receiver destinations as `issuer_base + r * stride` and the LMEM model
// is block-addressed (byteen-masked), so a non-aligned stride would
// truncate the destination; the descriptor handler rounds its stride the
// same way to stay consistent.
uint32_t stride = lmem_slot_pitch(next.lmem_size);
uint32_t max_slots = usable_slots(stride);

uint32_t k = next.cluster_dim[0]
Expand Down
2 changes: 1 addition & 1 deletion sim/simx/cta_dispatcher.h
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ class CtaDispatcher : public SimObject<CtaDispatcher> {
mutable bool lmem_overrun_reported_;

// Fixed-stride slot allocation. Every resident CTA gets LMEM base
// slot × stride, where stride = align(lmem_size, MEM_BLOCK_SIZE) is uniform
// slot × stride, where stride = lmem_slot_pitch(lmem_size) is uniform
// within a kernel. The occupancy bound floor(capacity/stride) (capped at
// NUM_WARPS) replaces the byte ring's free_size accounting, so no wrap or
// padding arithmetic is needed. A CTA alone in its cluster takes the lowest
Expand Down
5 changes: 3 additions & 2 deletions sim/simx/dxa/dxa_core.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@

#include "dxa_core.h"
#include "dxa_meta.h"
#include "lmem_slot.h"
#include <algorithm>
#include <array>
#include <cstring>
Expand Down Expand Up @@ -202,14 +203,14 @@ class DxaCore::Impl {
case VX_DCR_DXA_DESC_TILESIZE4_OFF: d.tile_sizes[4] = uint16_t(value & 0xffff); break;
case VX_DCR_DXA_DESC_CFILL_OFF: d.cfill = value; break;
case VX_DCR_DXA_DESC_SMEM_STRIDE_OFF:
// Round up to the LMEM word size so multicast destination addresses
// Round up to the CTA slot pitch so multicast destination addresses
// `issuer_addr + r * smem_stride` stay block-aligned. The LMEM model
// treats requests as block-aligned with a byteen mask; a non-aligned
// stride would truncate the address and land writes in the wrong
// block. The dispatcher applies the same rounding to per-CTA LMEM
// allocations so the host-visible "same offset in receiver LMEM"
// contract holds end-to-end.
d.smem_stride = (value + kLmemWordSize - 1u) & ~uint32_t(kLmemWordSize - 1u);
d.smem_stride = lmem_slot_pitch(value);
break;
default: break;
}
Expand Down
44 changes: 36 additions & 8 deletions sim/simx/tcu/tcu_unit.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,12 @@ static constexpr uint64_t kLineMask = ~uint64_t(VX_CFG_MEM_BLOCK_SIZE - 1);
// WGMMA tile-buffer geometry in 32-bit words.
static constexpr uint32_t kLmemBanks = VX_CFG_LMEM_NUM_BANKS;
static constexpr uint32_t kBankRowWords = kLmemBanks * (VX_CFG_XLEN / 32);
static constexpr uint32_t kBBlockWords = kFedpWords * cfg::tcN;
#ifdef VX_CFG_TCU_WGMMA_ENABLE
// A slice-aligned row-major K-slice then never straddles two bank rows.
static_assert((kBankRowWords % kFedpWords) == 0 && (kBankRowWords % (2 * cfg::tcK)) == 0,
"a WGMMA K-slice must divide an LMEM bank row");
#endif
static constexpr uint32_t kBBlockWords = kFedpWords * cfg::tcN;
static constexpr uint32_t kBBlockWordsSp = cfg::tcK * cfg::tcN * 2;
// One bank row holds kBSubBlocks consecutive dense B blocks.
static constexpr uint32_t kBSubBlocks =
Expand Down Expand Up @@ -856,7 +861,7 @@ class TcuUnit::Impl {
wgmma_desc_[wid][0] = trace->src_data.at(0).at(0).u32;
}
wgmma_desc_[wid][1] = trace->src_data.at(1).at(0).u32;
this->decode_descs(wid, tpuArgs.fmt_s, tpuArgs.is_a_smem);
this->decode_descs(wid, tpuArgs.fmt_s, tpuArgs.is_a_smem, is_sparse);
}

auto& req = reqs.at(b);
Expand Down Expand Up @@ -970,14 +975,41 @@ class TcuUnit::Impl {
return ready_mask;
}

// The tile buffers cannot fetch a descriptor that breaks the alignment rule,
// so SimX must not read it from its exact address either.
static void check_desc(uint32_t desc, bool is_b, bool is_sparse) {
uint32_t slice_words = vt::wgmma_slice_words(is_b, is_sparse, cfg::tcK, kFedpWords);
auto fault = vt::smem_desc_check(desc, slice_words, kBankRowWords * 4);
if (fault == vt::smem_desc_fault::none) {
return;
}
std::cerr << "*** WGMMA " << (is_b ? "B" : "A") << " descriptor 0x" << std::hex << desc << std::dec
<< " (offset " << vt::smem_desc_offset(desc) << " B, stride " << vt::smem_desc_stride(desc) << " B): ";
switch (fault) {
case vt::smem_desc_fault::block_base:
std::cerr << "a block-major base must be aligned to an LMEM bank row (" << (kBankRowWords * 4) << " B)";
break;
case vt::smem_desc_fault::row_base:
std::cerr << "a row-major base must be a multiple of the " << (slice_words * 4) << "-byte K-slice";
break;
default:
std::cerr << "a row-major stride must be a multiple of the " << (slice_words * 4) << "-byte K-slice";
break;
}
std::cerr << std::endl;
std::abort();
}

// Decode a warp's WGMMA descriptors into LMEM base and row stride.
void decode_descs(uint32_t wid, uint32_t fmt_s, bool is_a_smem) {
void decode_descs(uint32_t wid, uint32_t fmt_s, bool is_a_smem, bool is_sparse) {
uint32_t e_bits = elem_bits(fmt_s);
uint32_t a_desc = wgmma_desc_[wid][0];
uint32_t b_desc = wgmma_desc_[wid][1];
if (is_a_smem) {
check_desc(a_desc, false, is_sparse);
lmem_desc_[wid][0] = {uint64_t(VX_MEM_LMEM_BASE_ADDR) + vt::smem_desc_offset(a_desc), vt::smem_desc_stride(a_desc) * 8 / e_bits, false};
}
check_desc(b_desc, true, is_sparse);
lmem_desc_[wid][1] = {uint64_t(VX_MEM_LMEM_BASE_ADDR) + vt::smem_desc_offset(b_desc), vt::smem_desc_stride(b_desc) * 8 / e_bits, false};
}

Expand Down Expand Up @@ -1199,16 +1231,12 @@ class TcuUnit::Impl {

uint32_t ratio = elem_ratio(fmt_s);
uint32_t k_words = is_sparse ? cfg::tcK : kFedpWords;
uint32_t e_bits = elem_bits(fmt_s);

// Decode smem descriptors (B always from smem, A optionally).
if (is_setup_uop) {
wgmma_desc_[wid][0] = a_desc;
wgmma_desc_[wid][1] = b_desc;
if (is_a_smem) {
lmem_desc_[wid][0] = {uint64_t(VX_MEM_LMEM_BASE_ADDR) + vt::smem_desc_offset(a_desc), vt::smem_desc_stride(a_desc) * 8 / e_bits, false};
}
lmem_desc_[wid][1] = {uint64_t(VX_MEM_LMEM_BASE_ADDR) + vt::smem_desc_offset(b_desc), vt::smem_desc_stride(b_desc) * 8 / e_bits, false};
this->decode_descs(wid, fmt_s, is_a_smem, is_sparse);
uint32_t nrc = (cd_nregs == 0) ? 8 : (cd_nregs == 1) ? 16 : 32;
cur_xtile_n_ = (nrc * VX_CFG_NUM_THREADS) / wg_cfg::xtileM;
return;
Expand Down
Loading
Loading