[PR 2/2] LTX2: SVG config and pipeline

[jitendra-jalwaniya]

Overview

This PR extends Sparse VideoGen (SVG) spatiotemporal attention support to LTX-2 (LTX2) video generation models on Cloud TPUs, building on the custom Ulysses/ring SVG kernel infrastructure introduced for Wan (PR #480).

Self-attention in LTX-2 transformer blocks dynamically profiles query tokens to choose between spatial and temporal attention patterns per head, skipping unneeded query–key interactions while executing through hardware-aligned local-band kernels on TPU. Sparse attention is opt-in (use_svg_attention: True), disabled by default, and configurable across denoising steps, layers, and sparsity densities. Audio self-attention and cross-modal attention remain dense to preserve temporal and semantic grounding.

This is PR 2/2 (config and pipeline side). It depends on #497, which adds the SVG attention path to the LTX2 model.

Changes in this PR:

• ltx2_pipeline.py: builds the SVG attention_config from the pyconfig and passes it to the transformer; passes the denoising step index (svg_step_index) through both the scanned and unscanned diffusion loops.
• ltx2_video.yml, ltx2_3_video.yml: add SVG options (disabled by default) and svg_flash_block_sizes for the sparse kernel tiling.
• generate_ltx2.py: add SVG options to the AOT cache metadata keys so compiled artifacts are not reused across SVG settings.
• docs/svg.md: new SVG-on-TPU documentation covering Wan and LTX-2.
• tests/ltx2/test_svg_config_propagation_ltx2.py: new config-propagation tests.

---

VABench Evaluation: SVG vs. Dense Attention

The end-to-end results below require both #497 and this PR.

We evaluated SVG against dense attention on the Full VABench Benchmark suite (778 prompts across all 24 Easy/Hard bundles and 7 content categories) for LTX-2 synchronized text-to-audio-video (T2AV) generation at long sequence length (768 × 1280 × 241 frames, $N = 29,760$ video tokens, 10.04s @ 24 fps video + 24 kHz PCM audio) on TPU v6e-8 (8 chips), followed by a 15-dimension VABench evaluation across 8× NVIDIA A100-80GB GPUs. Each prompt was generated once per configuration:

• Dense Baseline: use_svg_attention=False, attention=ulysses_custom
• SVG Ulysses: use_svg_attention=True, svg_spatial_density=0.25, attention=ulysses_custom, svg_active_* left at defaults (SVG active on all steps and layers)

1. TPU v6e-8 Generation Performance (778 Videos @ 768 × 1280 × 241)

Metric	Dense Baseline	SVG Ulysses	Delta (SVG vs. Dense)
Denoising Time / Video (40 steps)	72.00 s	61.62 s	-10.38 s (-14.42% / 1.17× speedup)
Per-Step Denoising Latency	1.800 s / step	1.541 s / step	-0.259 s / step
Total Inference / Video (end-to-end)	102.94 s	91.23 s	-11.71 s (-11.38%)
Benchmark Wall Time (778 Videos)	~22.24 hours	~19.72 hours	-2.52 hours saved

2. 15-Dimension VABench Quality Highlights

Enabling SVG yields faster generation with comparable overall quality: most metrics are on par or slightly higher, with a small drop in judged visual realism (-1.78%):

• Audio-Video Synchronization & Lip-Sync:
  • Synchformer Temporal Desynchronization (second_desync $\downarrow$): Reduced from 0.6743 s $\rightarrow$ 0.6404 s (-5.03% better sync overall), with strong gains on Animals (-18.25%), Music (-12.24%), Virtual Worlds (-10.97%), and Synchronous Physical Sounds (-6.12%).
  • LatentSync Lip-Sync Error (second_lsa $\downarrow$): Reduced by -31.60% overall (0.9813 $\rightarrow$ 0.6712), and by -39.15% on Human Sounds (1.3270 $\rightarrow$ 0.8075). We observed this with video heads routed through SVG while audio and cross-modal attention stay dense; since each prompt was generated once, we have not yet measured seed-to-seed variance for this metric.
• Cross-Modal Alignment: Higher alignment across all three embedding models: ImageBind-Huge (+1.91%), ViCLIP-L (+1.82%), and LAION-CLAP (+1.79%).
• Audio Aesthetic Quality: AudioBox Aesthetics improved by +1.67% (3.5645 $\rightarrow$ 3.6241), with DNSMOS (+0.55%) and NISQA (+0.39%) on par.
• Multimodal Judge & Fine-Grained QA (Qwen2.5-Omni-7B):
  • Parity on most multimodal judge criteria (alignment: 4.47 vs. 4.45 [-0.60%], audio realism: 3.94 vs. 3.91 [-0.82%], expressiveness: 4.25 vs. 4.23 [-0.45%]); visual realism is slightly lower (4.55 vs. 4.47 [-1.78%]).
  • Notable accuracy gains on multi-turn question answering: Audio QA (+4.90%) and Visual QA (+4.64%).

Full 15-Dimension VABench Comparison Table (778 Prompts)

Module	Dimension	Metric / Criterion	Direction	Dense Baseline	SVG Ulysses	Absolute Delta	Relative Change
M1: Audio Quality & Aesthetics	first_dnsmos	Microsoft DNSMOS (sig_bak_ovr + p808)	$\uparrow$	1.6538	1.6629	+0.0091	+0.55%
	first_nisqa	NISQA v2 Speech/Audio Naturalness MOS	$\uparrow$	1.5264	1.5324	+0.0060	+0.39%
	first_audiobox	Meta AudioBox Aesthetics	$\uparrow$	3.5645	3.6241	+0.0596	+1.67%
M2: Cross-Modal Sync & Alignment	second_viclip	ViCLIP-L Text-Video Similarity	$\uparrow$	0.1918	0.1953	+0.0035	+1.82%
	second_clap	LAION-CLAP Text-Audio Similarity	$\uparrow$	0.3835	0.3904	+0.0069	+1.79%
	second_imagebind	Meta ImageBind-Huge AV Alignment	$\uparrow$	0.2144	0.2185	+0.0041	+1.91%
	second_desync	Synchformer Temporal Desync Offset (s)	$\downarrow$	0.6743 s	0.6404 s	-0.0339 s	-5.03% (Better Sync)
	second_lsa	LatentSync Lip-Sync Distance	$\downarrow$	0.9813	0.6712	-0.3101	-31.60% (Better Lip-Sync)
M3: Multimodal Judge (Qwen2.5-Omni-7B)	third_alignment	AV Semantic & Temporal Alignment (1–5)	$\uparrow$	4.4743	4.4473	-0.0270	-0.60% (Parity)
	third_audio_reality	Acoustic Realism & Fidelity (1–5)	$\uparrow$	3.9383	3.9062	-0.0321	-0.82% (Parity)
	third_visual_reality	Visual Realism & Coherence (1–5)	$\uparrow$	4.5476	4.4666	-0.0810	-1.78%
	third_expressiveness	Emotional & Dynamic Expressiveness (1–5)	$\uparrow$	4.2468	4.2275	-0.0193	-0.45% (Parity)
	third_artistry	Audiovisual Aesthetic Quality (1–5)	$\uparrow$	3.6272	3.6478	+0.0206	+0.57%
	Module 3 Mean	Mean Multimodal Judge Score (1–5)	$\uparrow$	4.1668	4.1391	-0.0278	-0.67% (Parity)
M4: Multi-Turn Question Answering	fourth_qa_audio	Audio QA Accuracy (0–1)	$\uparrow$	0.6438	0.6754	+0.0315	+4.90%
	fourth_qa_vision	Visual QA Accuracy (0–1)	$\uparrow$	0.6088	0.6370	+0.0283	+4.64%
	Module 4 Mean	Mean Multi-Modal QA Accuracy (0–1)	$\uparrow$	0.6263	0.6562	+0.0299	+4.77%

---

Configuration Example

To enable SVG on LTX-2, add the following to ltx2_video.yml or override via CLI flags. The VABench run above used attention: ulysses_custom and svg_spatial_density: 0.25; the example below shows a restricted step/layer window:

attention: ulysses_ring_custom_fixed_m
use_svg_attention: True
svg_spatial_density: 0.20
svg_active_start_step: 10
svg_active_end_step: 35
svg_active_start_layer: 1
svg_active_end_layer: 28
svg_profile_query_count: 64
svg_sample_max_row: 10000
svg_profile_seed: 0
svg_include_first_frame: True

SVG requires one of the custom Ulysses/ring attention backends (the LTX2 default attention: flash is not supported).

---

Testing

Run the LTX-2 SVG unit tests from the repository root:

python -m pytest -q \
  src/maxdiffusion/tests/ltx2/test_svg_config_propagation_ltx2.py \
  src/maxdiffusion/tests/ltx2/test_svg_attention_ltx2.py

All existing Wan and LTX-2 unit tests continue to pass.

[github-actions]

e2e testgrid: https://8bcf50593faf4ea38060e236169827e5-dot-us-central1.composer.googleusercontent.com/dags/maxdiffusion_tpu_e2e/grid

[gemini-code-assist]

Code Review

This pull request integrates Sparse VideoGen (SVG) attention support into the LTX2 pipeline in MaxDiffusion. Key changes include adding SVG configuration parameters to the LTX2 video config files, propagating these settings through the pipeline and model construction, updating the diffusion loop to pass step indices, validating incompatibility with CFG cache and MagCache, and adding comprehensive unit tests for configuration propagation and validation. I have no feedback to provide as there are no review comments.

[Perseus14]

Nice work wiring SVG attention into the LTX-2 pipeline, configs, and docs! The attention_config construction and svg_step_index propagation through both the lax.scan and Python diffusion loops look clean.

I left a few inline comments to address before merging:

1. Default layer count (28 -> 48): LTX-2 and LTX-2.3 have 48 layers, so let's update the svg_num_layers fallback in ltx2_pipeline.py (and confirm whether svg_active_end_layer: 28 in docs/svg.md and the PR description should also be 48).
2. Docs config snippet (ulysses_shards): attention: ulysses_ring_custom_fixed_m in docs/svg.md (and the PR description) requires ulysses_shards > 0, otherwise pyconfig.py raises a ValueError since ltx2_video.yml defaults to ulysses_shards: -1.
3. YAML config keys (ltx2_video.yml / ltx2_3_video.yml): Add the remaining svg_* keys read by ltx2_pipeline.py so they can be overridden via CLI, and clean up the unused Wan 2.2 dual-expert keys (svg_high_noise_density / svg_low_noise_density).
4. AOT metadata & tests: Avoid invalidating dense AOT cache hashes when use_svg_attention=False, and add a couple of unit tests for svg_spatial_density=1.0 and svg_step_index forwarding.

[Perseus14]

Small tip worth adding here: in ltx2_video.yml, profiler_steps defaults to 5 (steps 0–4), while the LTX-2 example above starts SVG at step 10 (svg_active_start_step: 10). Could you add a short note reminding users to set profiler_steps higher than svg_active_start_step (or lower svg_active_start_step when profiling) so the profile actually captures the sparse SVG steps?

[Perseus14]

Up on line 348, we disable SVG when svg_spatial_density >= 1.0:

use_svg = bool(getattr(config, "use_svg_attention", False)) and (expert_density < 1.0)

Here in __call__, though, we only check getattr(self.config, "use_svg_attention", False) without checking if svg_spatial_density < 1.0. To keep the two checks consistent, could we also check that float(getattr(self.config, "svg_spatial_density", 0.25)) < 1.0 before raising the error?

[Perseus14]

If use_svg_attention is False, changing any of the svg_* config values won't change the compiled model, so we ideally shouldn't change the AOT cache hash when SVG is turned off.

In aot_cache.py, there is already a helper extract_svg_meta(config, pipeline) that returns {"use_svg_attention": False} when SVG is disabled and only hashes all the svg_* parameters when SVG is enabled. Could we reuse that helper (or only include the svg_* keys when use_svg_attention is True)?