Skip to content

SimX: speed up the simulation harness (1.5x serial, ~10x functional MT) - #484

Merged
tinebp merged 1 commit into
masterfrom
perf/fast-simx
Nov 19, 2026
Merged

tinebp merged 1 commit into
masterfrom
perf/fast-simx

Conversation

@tinebp

@tinebp tinebp commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR speeds up SimX by changing only its simulation harness:

  • the SimObject kernels (channels, events, execution domains, the multi-threaded executor);
  • the support utilities in sw/common;
  • the per-lane containers, which are now sized at compile time from VX_CFG_*;
  • the build flags.

No model behavior changes. Every timed run gives the same instructions and cycles as master,
and multi-threaded runs stay bit-identical to serial ones.

On raycast at 64×64:

  • one core runs 1.5× faster;
  • 16 cores, timed, on 16 host threads run 4.3× faster than master serial, with identical cycles;
  • 16 cores, functional, on 16 threads with the new SIMX_QUANTUM=100 run about 10× faster than master serial.

Motivation

Graphics workloads on SimX take days to render one scene, which blocks graphics development and
conformance runs. Profiling raycast showed the cost was not in the ISA:

  • Harness cost: functional mode spent about 50 K host instructions per warp instruction, of
    which only about 1.25 K was ISA execution. The rest was harness and memory-system bookkeeping:
    • per-tick heap allocation;
    • thread-local storage lookups through __tls_get_addr;
    • virtual bind-chain walks;
    • non-inlined assert paths.
  • No multi-thread scaling: the existing multi-threaded executor (SIMX_MT) gave 1.06× on 8
    threads, and timed MT was slower than serial. Every packet send and pop did a locked
    read-modify-write on one process-wide atomic counter, which all workers fought over.

A faster separate "fast core" interpreter was considered and rejected: a second execution engine
would double the work for every new feature. This PR keeps one engine and makes it cheaper.

What changed

Dynamic allocation

Change Where Effect
One per-thread release list returns every pool at thread exit, instead of a thread_local holder per pool type mempool.h cuts the TLS segment from 354 B to 170 B (282 B with every extension) (see static TLS below)
Object pools (events, traces, Instr) grow by slabs; before, they held 64–128 blocks and fell back to operator new after that sw/common/mempool.h no heap traffic once warmed up
Pool lookup through a trivial initial-exec TLS pointer, instead of a thread_local holder with a non-trivial constructor mempool.h ~0.4 %
Functional channel storage: a growable power-of-two RingQueue (new grow()) in place of std::deque simobject_functional.h, ringqueue.h slots reused
Cross-domain sends carry packets up to 128 B inline in the call object; larger packets are boxed both kernels one allocation fewer per crossing
CachePortArb: four per-tick std::vectors became reused members, and the arbiter sleeps after two idle ticks sim/simx/mem/cache.cpp −17 % host instructions; it made 29 of the 50 allocations per instruction

Compile-time sizing (VX_CFG_*)

  • New FixedVector<T, N> (sim/common/fixed_vector.h): inline storage, the std::vector subset
    SimX uses, and bounds asserts.
  • RegDataVec = FixedVector<reg_data_t, VX_CFG_NUM_THREADS> is used for
    instr_trace_t::src_data, dst_data and src_regs. That removes 7 heap allocations per trace.
  • The LsuReq / LsuRsp lanes and the coalescer's per-tick temporaries are sized by
    LSU_MAX_LANES = max(LSU_CHANNELS, DCACHE_CHANNELS).
  • OpcUnit register files are std::array<std::array<Word, VX_CFG_NUM_THREADS>, MAX_NUM_REGS>:
    • a constant trip count for lane loops;
    • no vector<vector> indirection;
    • no .at() checks.
  • Combined: about −7 % host instructions.
  • XLEN, NUM_THREADS and NUM_WARPS were already compile-time; it was the containers that
    ignored them.

Communication overhead

  • In-flight count per domain: the process-wide std::atomic in-flight counter is replaced by a
    plain counter per execution domain, each on its own cache line. Only the owning domain's context
    reserves toward its endpoints or pops from them. idle() sums the domains. This is the main
    multi-thread fix.
  • Lock-free cross-domain mailboxes: each (target, source) pair has its own lane, written by one
    source and drained by one target. Lanes are triple-buffered by epoch. This replaces a
    mutex-guarded shared inbox, a global pending atomic and a sort every cycle. An entry written at
    cycle t is due at t+1, so draining the lanes in source order reproduces the old canonical
    (due, src, seq) order exactly.
  • Direct endpoint queries: full, size, capacity and reserve go straight to the cached
    endpoint, instead of a virtual recursion through each forwarding channel.
  • Less per-object TLS work: tick_domain no longer writes thread-local state for each object.
    The current object is tracked only when the boundary audit is enabled.
  • Barrier: it spins 4096 times and then yields, so descheduled peers are not starved on a
    loaded host.

Thread-local storage and static compilation

Change Effect
Platform thread-locals (tl_exec_domain_, tl_cycles_, …) use the initial-exec TLS model removes __tls_get_addr calls from libsimx.so, which were 8 % of time
__assert failure path moved into a cold, non-inlined lambda (sw/common/util.h) each call site becomes a compare plus a call, so small accessors (peek, pop, full, instance) inline again
SIMX_OPT ?= -O3 -fno-semantic-interposition -fno-plt (new release default in sim/simx/Makefile) −5.6 % host instructions vs -O2; without -fno-semantic-interposition, calls inside libsimx.so cannot inline
LTO measured: no gain over -O3, so not enabled

Static TLS budget. With initial-exec, glibc places libsimx.so's whole TLS segment in its
static block, even though the library is loaded with dlopen. The spare room there is a few hundred
bytes, shared by every dlopen'ed library that does the same. The per-type pool holders took 320 B of
it, so they were folded into one list, and the segment is now 170 B (282 B with every extension) (master: 354 B, dynamic).

Quantum synchronization (functional kernel only)

  • -DSIMX_QUANTUM=<Q> follows the SIMX_MT pattern: types.h resolves it and hands it to
    SimPlatform::set_quantum().
  • Domains run Q cycles between barriers.
  • A cross-domain call runs at the first quantum boundary at or after its due cycle.
  • Results are bit-identical for every thread count at a given Q, but Q changes the
    (non-physical) functional cycle count.
  • The timed kernel keeps per-cycle lockstep and asserts Q = 1.

Performance

Wall clock, raycast 64×64. Master 7300bc849 and this branch were run alternately on a shared
64-core host (load 12–28), best of 2.

Configuration Master This PR Speedup Instructions / cycles
1 core, timed 45.51 s 30.02 s 1.52× 5,664,693 / 18,460,024, identical
1 core, functional 40.58 s 26.93 s 1.51× 5,664,693 / 16,056,752, identical
16 cores, timed, serial 55.44 s 28.88 s 1.92× 5,664,933 / 1,424,691, identical
16 cores, timed, SIMX_MT=16 26.70 s 12.97 s 2.06× (4.3× vs master serial) identical
16 cores, functional, serial 40.24 s 27.87 s 1.44× 5,664,933 / 1,259,471, identical
16 cores, functional, SIMX_MT=16 22.63 s 11.99 s 1.89× identical
16 cores, functional, serial, SIMX_QUANTUM=100 46.10 s (master serial, Q=1) 26.56 s 1.74× 5,664,933 / 1,354,947 (Q changes functional cycles)
16 cores, functional, SIMX_MT=16 SIMX_QUANTUM=100 19.41 s (master MT16) 4.39 s 4.4× vs master MT16; ~10× vs master serial (40–46 s) 5,664,933 / 1,354,947

Simulation throughput, warp instructions per second:

  • 1 core: 124 K → 189 K;
  • 16 cores, timed, 16 threads: 212 K → 437 K;
  • 16 cores, functional, 16 threads, Q=100: about 1.29 M, against 102–141 K for master serial.

Host instructions, single thread (callgrind Ir, raycast 16×16; deterministic, unlike wall time
on a loaded host):

Kernel Master This PR Ratio
timed 23.84 G 13.91 G 1.71×
functional 21.08 G 12.21 G 1.73×

Per-change contribution to host instructions, measured one at a time on the timed kernel:

Change Ir
CachePortArb scratch reuse and sleep −17 %
initial-exec TLS −8 %
compile-time lane containers −7 %
-O3 -fno-semantic-interposition -fno-plt −5.6 %
direct endpoint queries −1 to −2 %
pool lookup through initial-exec TLS −0.4 %
cold assert path not measured separately; it lets the accessors above inline

The other changes target multi-thread scaling, which Ir does not capture.

Constraints and challenges

  • Cycle exactness. Timed SimX is the RTL's timing model and is checked by model_parity, so no
    change could move a cycle. Every harness change was validated as a pure refactor: identical
    instructions, cycles and PERF: counters on every case run.
  • Determinism across thread counts. The executor guarantees bit-identical results for any
    SIMX_MT. The mailboxes had to reproduce the old inbox's canonical order exactly. This was checked
    at Q=1 and at Q=100: 517,448 cycles at 32×32 with the old inbox and the new one.
  • Harness-only scope. No second core engine, no model restructuring. New features must not cost
    duplicate work.
  • Measurement. perf is unavailable (perf_event_paranoid=4), and wall time varies ±30 % on the
    shared host. Single-thread A/B used callgrind instruction counts; wall time used alternating runs,
    best of 2.
  • Compile-time sizing. FixedVector capacities come from VX_CFG_*. Every container whose
    length can exceed NUM_THREADS (LSU lanes vs D-cache channels) is sized by the maximum of the
    two, and overflow asserts.

Why further gains are hard

  1. The uncore domain is a serial point. Every core's L1 misses go through domain 0 (L2, memory).
    With 16 sockets, 16 threads are barely faster than 8: the workers wait on the uncore. This is the
    limiting term for multi-threading. Removing it changes the model's topology (see future work).
  2. Timed multi-threading must stay in per-cycle lockstep. Two barriers per cycle are needed for
    cycle-accurate cross-domain timing, so timed MT cannot use quanta. Measured on 16 cores, timed
    MT16 is 2.2× its own serial time (28.88 s → 12.97 s), against 6.3× for functional MT16 at Q=100
    (26.56 s → 4.39 s).
  3. Most of the remaining single-thread cost is in the models. About 30 % is harness: the tick
    scan over active objects and the per-packet event objects. About 70 % is the models' own on_tick
    work (pipeline stages, caches, coalescer). A further large single-thread gain means touching the
    models, or a second engine, which this PR rules out.
  4. A larger quantum has diminishing returns. Q=100 already removes most barrier cost. A larger Q
    delays every L1 miss crossing into the uncore by up to Q cycles, which inflates functional cycles
    and stalls cores without speeding the host.

Future work

  • Split the uncore domain per memory channel or L2 bank (a model topology change, needing
    RegSlice boundaries on the new edges). This is the largest remaining multi-thread lever for both
    kernels.
  • Event-driven tick scheduling: skip objects with no input and no pending work, instead of
    scanning every active object each cycle; extend CachePortArb's sleep to other units.
  • Packet and event allocation: move per-packet event objects onto per-channel intrusive slots.
  • PGO: profile-guided layout of libsimx.so. Not measured here, because instruction counts
    cannot show layout gains and wall time was too noisy.
  • Adaptive quantum in functional mode, shrinking around bursts of cross-domain traffic.
  • More compile-time specialization: carry other VX_CFG_* sizes (warps, LSU queue depths) into
    fixed containers where profiles show heap or indirection cost.

Documentation

  • docs/designs/simobject.md: per-domain in-flight accounting, the mailbox design, SIMX_QUANTUM.
  • docs/install_vortex.md, docs/debugging.md: the SIMX_QUANTUM knob, an example, and a note
    that multi-threading needs several sockets.
  • docs/designs/simx_simulator_architecture.md: a pointer to quantum synchronization.

Tests

New smoke case debug:simx_functional-3: sgemm on 4 cores, functional, SIMX_MT=4,
SIMX_QUANTUM=100, at XLEN 32 and 64. It is the first CI coverage of the quantum path.

Impact and evidence

See docs/review/code-review-policy.md.

Impact class: R2 (ci/change_impact.py classify: simx-model, build-config, sw-stack). Evidence
warning simx-rtl-lockstep ("SimX model changed with no RTL change"): the change is
behavior-preserving, shown by the SimX-only model_parity comparison below.

Contracts touched: none. No RTL, interface, ISA, VX_types.toml, ABI struct or testcase change.

evidence status detail
functional tests (simx) run 1 core: vecadd, sgemm, diverge, dogfood (22 sub-tests), fence, printf, sort, bfs, multikernel, module_reload. 16 cores, serial and SIMX_MT=16: gbar_phase, sgemm, bfs, multikernel, raycast. All PASS with cycles identical to master.
model_parity run, SimX leg only all 87 cases with a SimX leg (core, regression, tensor, tensor_mx, tensor_wg, tensor_sp, dxa, vm, graphics, raytracing, zfhmin, tfu): same exit code (0) and every PERF: line identical to master. The rtlsim leg is unchanged code, so the parity verdict is master's.
perf_gate n/a runs rtlsim cycles only; no RTL change
Beacon logic_gate n/a no RTL change
synthesis n/a no RTL change
XLEN 64 build run vecadd, sgemm, dogfood (22 sub-tests), diverge, sort, bfs at XLEN 64: PASS, cycles identical to master; debug:simx_functional-3 passes at XLEN 32 and 64
all-extensions build run TCU, TEX, RASTER, OM, RTU, DXA, VM, functional, MT: compiles under -Werror; ci/check_simx_mt_boundary.sh passes
independent review run independent review agent (simx-review checklist): no F3. Fixed: an F2 (missing braces) and F1s (a mailbox-lane reference held across a call that can grow mail_ in serial runs, stale comments and docs, the static TLS budget, no CI case for SIMX_QUANTUM). Re-validated after the fixes: raycast 1-core and 16-core timed and functional cycle-identical, 8 apps cycle-identical to master, serial and MT4 identical at Q=100.
  • No test silenced.
  • No golden data changed.
  • No RTL change; SimX cycles shown unchanged by the parity comparison.
  • No toolchain pin moved.

🤖 Generated with Claude Code

Graphics scenes take days on SimX. On raycast, functional mode spent
~50K host instructions per warp instruction, only ~1.25K of them ISA work,
and lockstep SIMX_MT=8 gave 1.06x. This change keeps the single engine and
the models' behavior, and removes the harness overhead:

- Communication: replace the process-wide atomic in-flight counter (two
  locked RMWs per packet, contended by every worker) with a per-domain
  counter; replace the locked shared cross-domain inbox with lock-free
  per-(target, source) mailbox lanes, triple-buffered by epoch, whose
  drain order reproduces the canonical (due, src, seq) order; bind-chain
  queries go straight to the cached endpoint; the barrier spins then
  yields.
- Allocation: object pools grow by slabs instead of falling back to the
  heap; functional channels use a growable RingQueue instead of
  std::deque; small cross-domain packets travel inline; CachePortArb
  reuses its per-tick scratch and sleeps when idle (-17% Ir).
- Compile-time sizing: FixedVector<T, N> with VX_CFG_* capacities for
  per-lane trace data, LSU requests/responses and the OpcUnit register
  file.
- TLS and code generation: initial-exec TLS for the platform's per-thread
  context (__tls_get_addr was 8% of time), with one per-thread pool
  release list so the static TLS segment shrinks to 170 B (master 354 B);
  cold out-of-line assert path; -O3 -fno-semantic-interposition -fno-plt
  as the release default.
- Functional kernel: -DSIMX_QUANTUM=<Q> runs domains Q cycles between
  barriers; results are bit-identical across thread counts for a given Q.
  The timed kernel keeps per-cycle lockstep.

Raycast 64x64: 1 core 45.5 s -> 30.0 s; 16 cores timed SIMX_MT=16
26.7 s -> 13.0 s (4.3x vs master serial 55.4 s), cycles identical;
16 cores functional SIMX_MT=16 SIMX_QUANTUM=100 4.4 s (~10x vs master
serial). Callgrind Ir: timed 23.84G -> 13.91G, functional 21.08G -> 12.21G.

Validation: the SimX leg of all 87 model_parity cases gives the same exit
code and PERF lines as master 7300bc8; 1-core and 16-core (serial and
MT16) regression apps are cycle-identical; XLEN 64 vecadd, sgemm, dogfood,
diverge, sort, bfs identical; all-extensions build compiles under -Werror.
New smoke case debug:simx_functional-3 covers SIMX_QUANTUM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tinebp
tinebp merged commit d8fa003 into master Nov 19, 2026
198 of 202 checks passed
@tinebp
tinebp deleted the perf/fast-simx branch November 19, 2026 04:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant