Repository navigation
SimX: speed up the simulation harness (1.5x serial, ~10x functional MT) - #484
Merged
Merged
Conversation
Graphics scenes take days on SimX. On raycast, functional mode spent ~50K host instructions per warp instruction, only ~1.25K of them ISA work, and lockstep SIMX_MT=8 gave 1.06x. This change keeps the single engine and the models' behavior, and removes the harness overhead: - Communication: replace the process-wide atomic in-flight counter (two locked RMWs per packet, contended by every worker) with a per-domain counter; replace the locked shared cross-domain inbox with lock-free per-(target, source) mailbox lanes, triple-buffered by epoch, whose drain order reproduces the canonical (due, src, seq) order; bind-chain queries go straight to the cached endpoint; the barrier spins then yields. - Allocation: object pools grow by slabs instead of falling back to the heap; functional channels use a growable RingQueue instead of std::deque; small cross-domain packets travel inline; CachePortArb reuses its per-tick scratch and sleeps when idle (-17% Ir). - Compile-time sizing: FixedVector<T, N> with VX_CFG_* capacities for per-lane trace data, LSU requests/responses and the OpcUnit register file. - TLS and code generation: initial-exec TLS for the platform's per-thread context (__tls_get_addr was 8% of time), with one per-thread pool release list so the static TLS segment shrinks to 170 B (master 354 B); cold out-of-line assert path; -O3 -fno-semantic-interposition -fno-plt as the release default. - Functional kernel: -DSIMX_QUANTUM=<Q> runs domains Q cycles between barriers; results are bit-identical across thread counts for a given Q. The timed kernel keeps per-cycle lockstep. Raycast 64x64: 1 core 45.5 s -> 30.0 s; 16 cores timed SIMX_MT=16 26.7 s -> 13.0 s (4.3x vs master serial 55.4 s), cycles identical; 16 cores functional SIMX_MT=16 SIMX_QUANTUM=100 4.4 s (~10x vs master serial). Callgrind Ir: timed 23.84G -> 13.91G, functional 21.08G -> 12.21G. Validation: the SimX leg of all 87 model_parity cases gives the same exit code and PERF lines as master 7300bc8; 1-core and 16-core (serial and MT16) regression apps are cycle-identical; XLEN 64 vecadd, sgemm, dogfood, diverge, sort, bfs identical; all-extensions build compiles under -Werror. New smoke case debug:simx_functional-3 covers SIMX_QUANTUM. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR speeds up SimX by changing only its simulation harness:
sw/common;VX_CFG_*;No model behavior changes. Every timed run gives the same instructions and cycles as master,
and multi-threaded runs stay bit-identical to serial ones.
On raycast at 64×64:
SIMX_QUANTUM=100run about 10× faster than master serial.Motivation
Graphics workloads on SimX take days to render one scene, which blocks graphics development and
conformance runs. Profiling raycast showed the cost was not in the ISA:
which only about 1.25 K was ISA execution. The rest was harness and memory-system bookkeeping:
__tls_get_addr;SIMX_MT) gave 1.06× on 8threads, and timed MT was slower than serial. Every packet send and pop did a locked
read-modify-write on one process-wide atomic counter, which all workers fought over.
A faster separate "fast core" interpreter was considered and rejected: a second execution engine
would double the work for every new feature. This PR keeps one engine and makes it cheaper.
What changed
Dynamic allocation
thread_localholder per pool typemempool.hInstr) grow by slabs; before, they held 64–128 blocks and fell back tooperator newafter thatsw/common/mempool.hthread_localholder with a non-trivial constructormempool.hRingQueue(newgrow()) in place ofstd::dequesimobject_functional.h,ringqueue.hCachePortArb: four per-tickstd::vectors became reused members, and the arbiter sleeps after two idle tickssim/simx/mem/cache.cppCompile-time sizing (
VX_CFG_*)FixedVector<T, N>(sim/common/fixed_vector.h): inline storage, thestd::vectorsubsetSimX uses, and bounds asserts.
RegDataVec = FixedVector<reg_data_t, VX_CFG_NUM_THREADS>is used forinstr_trace_t::src_data,dst_dataandsrc_regs. That removes 7 heap allocations per trace.LsuReq/LsuRsplanes and the coalescer's per-tick temporaries are sized byLSU_MAX_LANES = max(LSU_CHANNELS, DCACHE_CHANNELS).OpcUnitregister files arestd::array<std::array<Word, VX_CFG_NUM_THREADS>, MAX_NUM_REGS>:vector<vector>indirection;.at()checks.XLEN,NUM_THREADSandNUM_WARPSwere already compile-time; it was the containers thatignored them.
Communication overhead
std::atomicin-flight counter is replaced by aplain counter per execution domain, each on its own cache line. Only the owning domain's context
reserves toward its endpoints or pops from them.
idle()sums the domains. This is the mainmulti-thread fix.
source and drained by one target. Lanes are triple-buffered by epoch. This replaces a
mutex-guarded shared inbox, a global pending atomic and a sort every cycle. An entry written at
cycle
tis due att+1, so draining the lanes in source order reproduces the old canonical(due, src, seq)order exactly.full,size,capacityandreservego straight to the cachedendpoint, instead of a virtual recursion through each forwarding channel.
tick_domainno longer writes thread-local state for each object.The current object is tracked only when the boundary audit is enabled.
loaded host.
Thread-local storage and static compilation
tl_exec_domain_,tl_cycles_, …) use the initial-exec TLS model__tls_get_addrcalls fromlibsimx.so, which were 8 % of time__assertfailure path moved into a cold, non-inlined lambda (sw/common/util.h)peek,pop,full,instance) inline againSIMX_OPT ?= -O3 -fno-semantic-interposition -fno-plt(new release default insim/simx/Makefile)-O2; without-fno-semantic-interposition, calls insidelibsimx.socannot inline-O3, so not enabledStatic TLS budget. With initial-exec, glibc places
libsimx.so's whole TLS segment in itsstatic block, even though the library is loaded with dlopen. The spare room there is a few hundred
bytes, shared by every dlopen'ed library that does the same. The per-type pool holders took 320 B of
it, so they were folded into one list, and the segment is now 170 B (282 B with every extension) (master: 354 B, dynamic).
Quantum synchronization (functional kernel only)
-DSIMX_QUANTUM=<Q>follows theSIMX_MTpattern:types.hresolves it and hands it toSimPlatform::set_quantum().Qcycles between barriers.Q, butQchanges the(non-physical) functional cycle count.
Q = 1.Performance
Wall clock, raycast 64×64. Master
7300bc849and this branch were run alternately on a shared64-core host (load 12–28), best of 2.
SIMX_MT=16SIMX_MT=16SIMX_QUANTUM=100SIMX_MT=16 SIMX_QUANTUM=100Simulation throughput, warp instructions per second:
Host instructions, single thread (callgrind Ir, raycast 16×16; deterministic, unlike wall time
on a loaded host):
Per-change contribution to host instructions, measured one at a time on the timed kernel:
CachePortArbscratch reuse and sleep-O3 -fno-semantic-interposition -fno-pltThe other changes target multi-thread scaling, which Ir does not capture.
Constraints and challenges
model_parity, so nochange could move a cycle. Every harness change was validated as a pure refactor: identical
instructions, cycles and
PERF:counters on every case run.SIMX_MT. The mailboxes had to reproduce the old inbox's canonical order exactly. This was checkedat Q=1 and at Q=100: 517,448 cycles at 32×32 with the old inbox and the new one.
duplicate work.
perfis unavailable (perf_event_paranoid=4), and wall time varies ±30 % on theshared host. Single-thread A/B used callgrind instruction counts; wall time used alternating runs,
best of 2.
FixedVectorcapacities come fromVX_CFG_*. Every container whoselength can exceed
NUM_THREADS(LSU lanes vs D-cache channels) is sized by the maximum of thetwo, and overflow asserts.
Why further gains are hard
With 16 sockets, 16 threads are barely faster than 8: the workers wait on the uncore. This is the
limiting term for multi-threading. Removing it changes the model's topology (see future work).
cycle-accurate cross-domain timing, so timed MT cannot use quanta. Measured on 16 cores, timed
MT16 is 2.2× its own serial time (28.88 s → 12.97 s), against 6.3× for functional MT16 at Q=100
(26.56 s → 4.39 s).
scan over active objects and the per-packet event objects. About 70 % is the models' own
on_tickwork (pipeline stages, caches, coalescer). A further large single-thread gain means touching the
models, or a second engine, which this PR rules out.
delays every L1 miss crossing into the uncore by up to Q cycles, which inflates functional cycles
and stalls cores without speeding the host.
Future work
RegSliceboundaries on the new edges). This is the largest remaining multi-thread lever for bothkernels.
scanning every active object each cycle; extend
CachePortArb's sleep to other units.libsimx.so. Not measured here, because instruction countscannot show layout gains and wall time was too noisy.
VX_CFG_*sizes (warps, LSU queue depths) intofixed containers where profiles show heap or indirection cost.
Documentation
docs/designs/simobject.md: per-domain in-flight accounting, the mailbox design,SIMX_QUANTUM.docs/install_vortex.md,docs/debugging.md: theSIMX_QUANTUMknob, an example, and a notethat multi-threading needs several sockets.
docs/designs/simx_simulator_architecture.md: a pointer to quantum synchronization.Tests
New smoke case
debug:simx_functional-3: sgemm on 4 cores, functional,SIMX_MT=4,SIMX_QUANTUM=100, at XLEN 32 and 64. It is the first CI coverage of the quantum path.Impact and evidence
See docs/review/code-review-policy.md.
Impact class: R2 (
ci/change_impact.py classify: simx-model, build-config, sw-stack). Evidencewarning
simx-rtl-lockstep("SimX model changed with no RTL change"): the change isbehavior-preserving, shown by the SimX-only
model_paritycomparison below.Contracts touched: none. No RTL, interface, ISA,
VX_types.toml, ABI struct or testcase change.SIMX_MT=16: gbar_phase, sgemm, bfs, multikernel, raycast. All PASS with cycles identical to master.model_parityPERF:line identical to master. The rtlsim leg is unchanged code, so the parity verdict is master's.perf_gatelogic_gatedebug:simx_functional-3passes at XLEN 32 and 64-Werror;ci/check_simx_mt_boundary.shpassessimx-reviewchecklist): no F3. Fixed: an F2 (missing braces) and F1s (a mailbox-lane reference held across a call that can growmail_in serial runs, stale comments and docs, the static TLS budget, no CI case forSIMX_QUANTUM). Re-validated after the fixes: raycast 1-core and 16-core timed and functional cycle-identical, 8 apps cycle-identical to master, serial and MT4 identical at Q=100.🤖 Generated with Claude Code