Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,12 @@ follows the version pins recorded in [VERSION](VERSION) (`VORTEX_VERSION`,

### Changed

- **V80 image reuse across runs.** A design the card already holds is no longer written again on every device open. SLASH now does
what XRT does for xclbins: `slashkit` stamps a UUID into the vbin, `vrtd` remembers the UUID it wrote and skips a write of the same
one, `VRT_FORCE_PROGRAM=1` overrides, `v80-smi forget-design` reports a design loaded over JTAG (`jtag_load_vortex.sh` now runs
it), and a design another client uses cannot be replaced. The `aved` driver is unchanged in this
respect and still opens with `program=true`. Needs a SLASH with design UUIDs (`vortex_3.x` from 6cb47f6e); rebuild runtimes linked against an earlier SLASH, whose client-library ABI differs; a vbin linked earlier gets one with
`slashkit stamp-uuid`. See [docs/designs/aved_driver_architecture.md](docs/designs/aved_driver_architecture.md#31-opening-the-device).
- **Toolchain pinned to v3.0.2** (`TOOLCHAIN_REV`): LLVM f44a6bb2af4c, with PoCL rebuilt on it. Wide loads/stores and 64-bit
atomics stay behind `+xvortexldstwide` and `+xvortexamod`, so code built with plain `+xvortex` uses no new instructions.
- **Dcache read-merge pairs realign.** A pair of same-word merge ports whose parts a bank served a cycle apart waits for its partner,
Expand Down
92 changes: 66 additions & 26 deletions docs/designs/aved_driver_architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,30 +96,70 @@ a specific failure this platform can produce.

### 3.1 Opening the device

`vrt::Device(bdf, vbin_path, program)` opens the board and optionally
reprograms the PL; `vrt::Kernel` then parses the vbin's `system_map.xml`.

The driver always programs (`program=true`), as the XRT driver's
`load_xclbin()` does. There is no opt-out switch.

There used to be one — `VORTEX_AVED_NO_PROGRAM=1` constructed with
`program=false` — because each open that programs the PL goes through `vrtd`'s
design writer, which runs its reset sequence only when the requested shell
differs from the current one. On a board whose static shell was never written
to OSPI flash, that comparison never settles: no reset runs, the design write
fails with `Input/output error`, and the AMC can be left in `NO_AMC`, costing a
recovery.

That is a property of an **unflashed board**, not of the driver. Write the
static shell to flash first (§5 of `xilinx_slash_setup.md`, *both* boot
partitions) and programming on open is well-defined. Verified on silicon:
consecutive programs with `v80-smi list` reporting a valid shell, sgemm passing
each time, no AMC loss.

The switch and the `/tmp/v80_resident_afu.path` fallback that let a run reuse
whatever bitstream happened to be resident were both removed. Running against a
resident image is what silently produces results attributed to the wrong
bitstream when `FPGA_BIN_DIR` changes.
`vrt::Device(bdf, vbin_path, program)` opens the board and, with
`program=true`, hands the vbin's design to `vrtd` to be written to the PL;
`vrt::Kernel` then parses the vbin's `system_map.xml`.

The driver always passes `program=true`, as the XRT driver always calls
`load_xclbin()`, and it has no logic of its own for what happens next. As with
`load_xclbin()`, that call does not mean a design write on every open: the
platform skips the write when the card already holds that design. A sequence of
runs on one vbin costs one design write.

SLASH does this the way XRT does:

| | XRT | SLASH |
|---|---|---|
| identity of an image | UUID in the xclbin header, stamped by `xclbinutil` | `<UUID>` in the vbin's `system_map.xml`, stamped by `slashkit` at link |
| who remembers what is loaded | the `xocl` driver (`icap_bitstream_uuid`) | `vrtd`, per card |
| same UUID requested | no download | no design write |
| override | `force_program_xclbin=true` in `xrt.ini` | `VRT_FORCE_PROGRAM=1` in the environment |
| image in use by another client | replacing it fails with `EBUSY` | replacing it fails with `VRTD_RET_BUSY` |
| where to see it | sysfs `xclbinuuid` | `v80-smi list` |

`vrtd` forgets the loaded UUID when a write starts, and records it again only
when the write has completed, so a failed write is never mistaken for a loaded
design. It also forgets on every reset, hot-plug and shell switch it performs,
and when the daemon restarts. Each client that loaded or reused the design
holds it until its connection closes; `vrtd` parks the user clock when the last
one leaves.

What follows for this driver:

- **Nothing in `init()` depends on whether the write was skipped.** The clock
is set, the transport is gated, the CP is parked and the device is reset on
every open (§3.2 to §3.4). Those steps already assume a device that a
previous process left running, which is now the common case rather than the
exception.
- **A reused image whose kernel does not answer is not written over.** The open
fails at the transport gate or the reset, and the next open skips the write
again. Writing a design over a hung kernel is what has reset the host, so
that write has to be asked for: recover the card, then
`v80-smi forget-design -d <BDF>` (or run once with `VRT_FORCE_PROGRAM=1`).
- **A vbin without a UUID is written on every open.** That is every vbin linked
before `slashkit` stamped them. `slashkit stamp-uuid <vbin>` gives one a UUID
in place, without relinking.
- **A design loaded outside `vrtd` is invisible to it** until it is told.
`jtag_load_vortex.sh` and `v80-smi write-static-shell --jtag` run
`v80-smi forget-design` themselves; after any other JTAG load, run it by
hand. Otherwise a run of the vbin `vrtd` wrote last would be skipped while
the fabric holds another design. XRT has the same blind spot for a
bitstream loaded behind its driver.
- **A skipped write keeps whatever state the design was left in**, as with
XRT. That is why the CP park and device reset above run on every open.

There is still no never-program switch in the driver. An earlier one
(`VORTEX_AVED_NO_PROGRAM=1`, with a `/tmp/v80_resident_afu.path` fallback) ran
against whatever bitstream happened to be resident, which silently attributes
results to the wrong image when `FPGA_BIN_DIR` changes. The skip is decided by
UUID, in the one place every design write goes through, so a different vbin is
always written.

Writing a design is only well-defined once the static shell is in OSPI flash
(§5 of `xilinx_slash_setup.md`, *both* boot partitions). On a board whose flash
was never written, `vrtd`'s requested-versus-current shell comparison never
settles: no reset runs, the design write fails with `Input/output error`, and
the AMC can be left in `NO_AMC`, costing a recovery.

### 3.2 The transport gate

Expand Down Expand Up @@ -189,8 +229,8 @@ build was wrong.

For the record, `xrt` and `opae` never hit the demux defect because XRT issues
`CTL_AP_RESET` before any CP-window access and re-asserts `ap_rst_n` on every
`load_xclbin`; the aved backend opens with `program = false`, so the routing
latch survived from the previous process.
`load_xclbin`; on the aved backend a design that `vrtd` does not write again
(§3.1) keeps the routing latch of the previous process.

### 3.5 Resuming the CP's position

Expand Down
10 changes: 8 additions & 2 deletions docs/xilinx_slash_setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,8 +287,14 @@ Four things that are easy to get wrong:
`Rescanning PCIe`) and the board returns on its own. Confirm with `v80-smi list`
— all PFs `OK` and a shell reported rather than `unknown`.

Once both partitions are written, the runtime programs the PL on device open
like XRT's `load_xclbin()`; there is no never-program switch.
Once both partitions are written, the runtime hands the vbin to `vrtd` on
device open like XRT's `load_xclbin()`, and like it the design is not written
again when the card already holds it: `vrtd` remembers the UUID of the vbin it
wrote. `v80-smi list` shows the loaded design, `VRT_FORCE_PROGRAM=1` writes it
regardless, and `slashkit stamp-uuid <vbin>` gives a vbin linked before UUIDs
were stamped one without relinking. The client libraries' ABI changed with
this, so rebuild any runtime linked against an earlier SLASH. See
[designs/aved_driver_architecture.md §3.1](designs/aved_driver_architecture.md#31-opening-the-device).

> **Flashing writes the card's boot memory.** A bad write can leave the board
> unbootable, recoverable only over JTAG. Confirm the image and the BDF before
Expand Down
103 changes: 0 additions & 103 deletions hw/scripts/check_axil_blocks.py

This file was deleted.

12 changes: 12 additions & 0 deletions hw/syn/xilinx/aved/tools/jtag_load_vortex.sh
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,18 @@ if PDI_PATH="$PDI" timeout 1800 xsdb "$WORK/load.tcl" 2>&1 | tee /dev/stderr | g
# also exactly when the card reloads from OSPI and loses the AFU.
realpath "$VBIN" > /tmp/v80_resident_afu.path
echo " recorded resident vbin: $VBIN"

# vrtd does not write a design the card already holds, and it knows only
# what it wrote itself: tell it the fabric changed, or the next run of the
# vbin it wrote last would be skipped and run on this one.
BDF=${VRT_DEVICE_BDF:-$(lspci -D -d 10ee:50b4 2>/dev/null | head -1 | cut -d' ' -f1)}
BDF=${BDF%.*}
if [ -z "$BDF" ] || ! v80-smi forget-design -d "$BDF"; then
echo
echo " vrtd could not be told that the design changed (card: ${BDF:-not found})."
echo " Before the next run: v80-smi forget-design -d <BDF>"
exit 1
fi
else
rm -f /tmp/v80_resident_afu.path
echo
Expand Down
7 changes: 4 additions & 3 deletions hw/syn/xilinx/common.mk
Original file line number Diff line number Diff line change
Expand Up @@ -95,14 +95,15 @@ COMPILE_CFLAGS += $(RTL_INCLUDE)
# ast dump flags
XML_CFLAGS = $(filter-out -DSYNTHESIS -DVIVADO, $(CFLAGS)) $(RTL_PKGS) -I$(DPI_DIR) -DSV_DPI

# Fail before synthesis if any 16-byte block in the CP AXI-Lite regfile is only
# Fail before synthesis if any block in the CP AXI-Lite regfile is only
# partially decoded, on cards whose shell resolves AXI-Lite in blocks (the
# descriptor's axil_block_bytes). Measured on V80 silicon: such a block DECERRs
# on EVERY word, including the words that ARE implemented. No simulator
# reproduces it; the symptom is a register reading 0xFFFFFFFF on hardware for no
# reason visible in the RTL.
# reason visible in the RTL. The regfile's unit test sweeps the elaborated
# decoder, so the check follows the RTL whatever shape its decoder takes.
check-axil-blocks:
$(if $(filter-out 0,$(PLATFORM_AXIL_BLOCK_BYTES)),@python3 $(SCRIPT_DIR)/check_axil_blocks.py $(RTL_DIR)/cp/VX_cp_axil_regfile.sv)
$(if $(filter-out 0,$(PLATFORM_AXIL_BLOCK_BYTES)),$(MAKE) -C $(ROOT_DIR)/hw/unittest/cp_axil_regfile run OPTS=--axil-block-bytes=$(PLATFORM_AXIL_BLOCK_BYTES))

# Vivado reads the preprocessed sources (gen_sources.sh -P), so its own
# inference sees the RTL as written; the Xilinx floating-point cores bind to
Expand Down
45 changes: 44 additions & 1 deletion hw/unittest/cp_axil_regfile/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,11 @@
// - Q_RING_BASE_LO/HI assemble into q_state.ring_base.
// - Out-of-range address returns DECERR; rdata is the 0xDEADBEEF
// sentinel for read-side, B has 2'b11 on the write side.
// - No aligned block of --axil-block-bytes (default 16) is decoded only in
// part, for reads or writes, anywhere in the 4 KB CP window. A shell that
// resolves AXI-Lite in such blocks (the V80's) DECERRs every word of a
// partly decoded block, the implemented ones included, and no simulator
// shows it; the V80 image flows run this scenario before synthesis.
// ============================================================================

#include "vl_simulator.h"
Expand Down Expand Up @@ -179,6 +184,13 @@ static constexpr uint16_t Q_ERROR = 0x2C;

int main(int argc, char** argv) {
Verilated::commandArgs(argc, argv);
unsigned block_bytes = 16;
for (int i = 1; i < argc; ++i) {
if (std::strncmp(argv[i], "--axil-block-bytes=", 19) == 0)
block_bytes = (unsigned)std::strtoul(argv[i] + 19, nullptr, 0);
}
EXPECT(block_bytes >= 4 && (block_bytes & (block_bytes - 1)) == 0,
"--axil-block-bytes must be a power of two of at least 4");
vl_simulator<VVX_cp_axil_regfile_top> sim;
uint64_t tick = 0;

Expand Down Expand Up @@ -364,6 +376,37 @@ int main(int argc, char** argv) {
"T11: GPU_DEV_CAPS_LO is read-only");
}

std::printf("PASSED — 11 scenarios\n");
// ----- Test 12: every aligned block is decoded whole or not at all -----
// Writes come last in the sweep: they land on live registers, and nothing
// after this scenario reads them back.
{
constexpr unsigned kWindowBytes = 0x1000;
const unsigned words = block_bytes / 4;
unsigned decoded_blocks = 0;
for (int pass = 0; pass < 2; ++pass) {
const bool writes = (pass == 1);
for (unsigned base = 0; base < kWindowBytes; base += block_bytes) {
unsigned ok = 0;
for (unsigned w = 0; w < words; ++w) {
const uint16_t addr = (uint16_t)(base + 4 * w);
const unsigned resp = writes
? axil_write(sim, tick, addr, 0)
: (unsigned)(axil_read(sim, tick, addr) >> 32);
EXPECT(resp == 0b00 || resp == 0b11, "T12: response is neither OKAY nor DECERR");
ok += (resp == 0b00);
}
if (ok != 0 && ok != words) {
std::fprintf(stderr, "FAIL T12: %s block 0x%03X: %u of %u words decoded\n",
writes ? "write" : "read", base, ok, words);
std::exit(1);
}
decoded_blocks += (!writes && ok == words);
}
}
// The global window and queue 0 are decoded; an empty map passes nothing.
EXPECT(decoded_blocks >= (0x40 + 0x40) / block_bytes, "T12: decoded window smaller than expected");
}

std::printf("PASSED — 12 scenarios\n");
return 0;
}
9 changes: 6 additions & 3 deletions sw/runtime/aved/vortex.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -220,9 +220,11 @@ class vx_device {

#ifdef CPP_API

// Program the PL on open, exactly as the XRT driver's load_xclbin() does.
// Hand the design to the platform on open, exactly as the XRT driver's
// load_xclbin() does. As there, the platform decides whether that is a
// write: vrtd does not write a design the card already holds.
//
// This is only correct because the static shell lives in OSPI flash. vrtd
// A write is only correct because the static shell lives in OSPI flash. vrtd
// runs its reset sequence when the requested shell differs from the
// current one, and a design write only succeeds on a freshly reset device.
// A board whose flash was never written reports Shell: unknown, so the
Expand Down Expand Up @@ -839,7 +841,8 @@ class vx_device {
// image's timing so it cannot disturb the shell before the next load.
void park_clock() noexcept {
constexpr uint64_t kSafeHz = 50000000;
if (sim_mode_)
// No handle when the open itself failed: nothing to park.
if (sim_mode_ || !vrtDevice_.getHandle())
return;
try {
vrtDevice_.setFrequency(kSafeHz);
Expand Down
Loading