[superme1one]

Summary

Specialize the default Iluvatar GEMM for B1/B2 Gate/Up (K4096/N32768) and Down (K16384/N4096). Reuse each weight load across two rows with four FP32 accumulators per lane. Exact dtype/layout/scalar/device/alignment checks retain BLAS fallback for every other call.

Motivation

Small decode matrices spend disproportionate time in the general BLAS path. The measured private Gate/Up implementation reduced two-task application time by 9.22–9.28%; adding Down reduced time by another 6.58–6.75% against its own MLP baseline. These gains are not additive.

Type of Change

• perf — narrowly scoped kernel optimization; floating-point/quality details below.
• Breaking public API change.

Platforms Affected

• Iluvatar (WITH_ILUVATAR).
• Shared CUDA template source (other backends retain original selection).
• Python/public API or build-system changes.

Smoke Test Result

Current submission: host eligibility/fallback test compiled and executed with MSVC C++17 /W4 /WX (exit0), Python syntax/Ruff, clang-format and git diff --check passed.

c++ -std=c++17 -Isrc tests/host/iluvatar_gemv_policy_test.cpp -o /tmp/policy-test
/tmp/policy-test
python -m pytest tests/test_iluvatar_mlp_gemv.py -q

The first command pair is the portable equivalent of the executed host check. The device pytest command is supplied for the pending current-revision hardware check, not claimed as executed. Draft: current-base CoreX build/smoke and same-two-task integration regression are pending. This submission pass performed local CPU validation; it did not resume the paused GPU experiment.

Test Results on Supported Platforms

Platform	Affected	Current-revision build/smoke	Notes
Iluvatar	Yes	Pending	Historical private-kernel/native/application evidence documented; hardware-owner validation needed for rebased integration
NVIDIA / MetaX / Moore / other CUDA-family	No	N/A - backend-local headers only	Provider selection unchanged
CPU / Ascend / Cambricon	No	N/A	No provider changes

Benchmark / Performance Impact

Historical four-optimization TG-V200/CoreX result: 100 fixed new GSM8K targets, max_tokens=512, greedy EOS=[2,73440], unchanged strict last-#### numeric scoring. Official accuracy 82/100 → 84/100; all requests completed/parsed, zero errors and truncations. 34/100 token-different, including one correct→wrong (1097: 480→448, reference480) and three wrong→correct (672,1259,1203).

Sum of per-version same-card formal paired-wave makespans 749.203818 → 505.326616 s (-32.5515%); completed request throughput +48.2613%. Published output tokens 13894→14213, so token throughput +51.6653% is reported separately. Each target's two versions used the same physical card; first20 serial, remaining80 sharded over two cards. Load/warmup/final shutdown excluded; queue/text processing included. These are combined gains, not individual-PR gains or the parallel run's wall time. Eight demonstrations came from a disjoint test-corpus subset (internal paired evaluation).

Measured baseline: InfiniLM 6e7f9537173cd33bb30539ebf8d92ab6d869660d. Main later advanced to cc6d52ae04f4c9ff17229bc7fc13a9ad105aca6d with auto-tuning initialization. Current submissions use that InfiniLM main and InfiniOps 1a68b261d19f0ed6f76bbabeab7a25f479f05e07. Historical GPU evidence applies to the recorded private builds, not this final rebased revision.

See docs/performance/iluvatar-mlp-gemv.md for shapes, numerical contract, measurement denominator and reproduction commands.

Notes for Reviewers

Both GEMV optimizations share a single dispatch point, so they are kept in one PR. Kernel arithmetic is factored into one shape-specialized template; experimental counters are removed. The new alignment rejection is conservative. FP32 accumulation order differs from BLAS; the combined quality evidence includes one task regression.

No model weights, prompts, private server addresses or credentials are included. No existing R5 PR is modified. Review scope is kernel integration and fallback correctness; readiness requires the affected-platform smoke checks above.

Related optimization PRs

• perf(iluvatar): specialize BF16 Gate-Up and Down GEMV
• perf(iluvatar): fuse causal Softmax with FP32 accumulation
• perf(iluvatar): batch Decode Attention QK reductions
• perf(attention): add opt-in Iluvatar fused causal Prefill