Latent Storage Benchmark (lsbench)

How far can VAE latents be compressed before the decoded images or videos degrade?

Modern generative pipelines store VAE latents (for training caches, datasets, archives), usually as fp32/bf16 tensors. A 1024x1024 FLUX.2 latent is 2 MiB in fp32, larger than the PNG it came from. This benchmark measures reconstruction quality vs. storage size and I/O time for many latent storage schemes, from plain casts to block-wise quantization with learned codebooks, across several image and video VAEs.

It can also be used as a plain VAE benchmark (fp32 storage = no compression).

Key findings

• Storage format is a free lunch down to ~5 bits per value. On FLUX.2, 8-bit and fp16/bf16 storage are indistinguishable from fp32 on every metric, and 5.5 bits per value is still within noise (FID 2.15 vs 2.08, PSNR -0.07 dB).
• The best scheme found is block-wise min/max quantization with a Lloyd-Max codebook (lloyd_minmax_square). At 4.5 bits per value it is 7.1x smaller than fp32 (3.5x smaller than fp16) for -0.25 dB PSNR and +0.3 FID.
• Ranking of schemes is almost VAE-independent (Spearman rank correlation >= 0.97 on PSNR across five image VAEs, >= 0.98 across three video VAEs). Absolute sensitivity differs: FLUX.2 is the most robust, SDXL (4 channels) the least.
• What makes it work (ablation on FLUX.2): block-wise min/max ranges give most of the gain over global min/max, and Lloyd-Max level tables help mainly at 2-3 bits.

FLUX.2 VAE, 1000 images at 1024x1024. PSNR (solid, left axis) and FID (dashed, right axis); lines are Pareto fronts of the three feature sets (global min/max only, + block-wise min/max, + Lloyd-Max tables); white markers are naive formats. Bits per value is bytes * 8 / (32 * 128 * 128) and includes all metadata.

Storage (FLUX.2)	Bits / value	KiB / image	vs fp32	PSNR (dB)	SSIM	LPIPS	FID
fp32 (reference)	32.0	2048	1.0x	31.749	0.8711	0.0260	2.08
fp16 / bf16	16.0	1024	2.0x	31.749 / 31.748	0.8711	0.0260	2.08
fp8_e4m3	8.0	512	4.0x	31.672	0.8706	0.0263	2.14
int8_tensor	8.0	512	4.0x	31.733	0.8710	0.0260	2.10
Lloyd-Max, 6 bit, block 4	6.5	417	4.9x	31.732	0.8710	0.0260	2.10
Lloyd-Max, 5 bit, block 4	5.5	353	5.8x	31.684	0.8705	0.0263	2.15
Lloyd-Max, 4 bit, block 4	4.5	289	7.1x	31.501	0.8687	0.0272	2.39
Lloyd-Max, 3 bit, block 4	3.5	225	9.1x	30.836	0.8609	0.0312	3.31
Lloyd-Max, 2 bit, block 4	2.5	161	12.8x	28.753	0.8313	0.0496	7.84

Lloyd-Max rows are lloyd_minmax_square with block: 4, range_bits: 4. FID is computed between the originals and the reconstructions, so even fp32 has a non-zero floor caused by the VAE itself; compare against the fp32 row.

Serialization of the Lloyd-Max scheme costs roughly 1.5-3.5 ms per 1024x1024 image on a GPU (about 0.3 ms for uniform quantizers) and deserialization about 0.2 ms.

Full tables and plots for every phase are in results/.

What is benchmarked

Pipeline for every VAE:

image/video -> vae.encode -> storage.serialize -> bytes -> storage.deserialize -> vae.decode -> metrics(original, reconstruction)

The size of bytes is the reported storage size, so every storage must put everything needed for decoding (shapes, scales, codebooks) into the byte string. All no_vae rows apply the same storage schemes directly to raw pixels and serve as a sanity baseline.

VAEs (5 image + 3 video, plus no_vae identity baselines)

Name	Domain	Latent (per 1024² image / 75x256² video)	Source
flux_2	image	32 ch, 8x spatial	FLUX.2-klein-4B
qwen_image_2_1	image	64 ch, 16x spatial	texture-fix-vae-for-qwen-image-2.1
flux_1	image	16 ch, 8x spatial	FLUX.1-schnell
sd_3_5	image	16 ch, 8x spatial	Stable Diffusion 3.5 Large
sdxl	image	4 ch, 8x spatial	sdxl-vae
wan	video	16 ch, 8x spatial, 4x temporal	Wan2.1
ltx_2_5	video	128 ch, 32x spatial, 8x temporal	LTX-2.5
minimax_h3	video	24 ch	MiniMax-H3

All wrappers return normalized latents (the VAE's own scale/shift/BN statistics are applied inside the wrapper), which is what a storage would see in a real training pipeline.

Latent storages (34 image + 14 video implementations, 200+ configurations)

Family	Config names	Idea
Floating point	fp32, fp16, bf16, fp8_e4m3, fp8_e5m2, fp8_e4m3_scaled, mxfp8	casts; per-sample or per-32-block (OCP MX) scaling
INT8	int8_tensor, int8_channel, int8_group, int8_asym, int8_mse_clip, int8_outlier, int8_hadamard, int8_mulaw, int8_minmax_square, nf8	symmetric/asymmetric scales at different granularity, clipping, outlier side-channel, Hadamard rotation, mu-law companding
INT4	int4_group, int4_asym_group, int4_asym_square, int4_minmax_group, int4_minmax_square, int4_mse_clip, int4_hadamard, int4_outlier	the same ideas at 4 bits (GPTQ/AWQ, QuaRot, SqueezeLLM style)
4-bit float / codebook	nvfp4, mxfp4, nf4, nf4_dq, kmeans4	FP4 E2M1 with block scales, NormalFloat (QLoRA), k-means codebook
Other bit widths	int2_minmax_square, int6_minmax_square	block-wise min/max at 2 and 6 bits
Block-wise min/max + codebook	lloyd_minmax_square, normal_cdf_minmax_square, video: lloyd_minmax_cube	see below

Naming: group = consecutive values in memory order, square = block x block spatial patch of one channel, cube = block³ spatio-temporal patch (video).

lloyd_minmax_square (the best scheme found) splits every channel into block x block patches. For each patch it stores the min and max as range_bits-bit indices into the channel's global [min, max] range, and quantizes the values inside the patch to bits-bit codes. The 2^bits reconstruction levels are a per-sample Lloyd-Max (1D k-means) table stored in the bytes (or uniform levels with lloyd: false; no blocks with block: null). Result names encode the hyperparameters, e.g. lloyd_minmax_square_c4_b4_r4 = 4 bit codes, 4x4 blocks, 4 bit ranges; minmax_square_* is the same without Lloyd tables.

Metrics

	Image	Video
Pixel	MSE, PSNR	MSE, PSNR (over the whole video)
Structural	SSIM	SSIM (per frame)
Perceptual	LPIPS (AlexNet)	LPIPS (per frame)
Distribution	FID (Inception, 2048-d)	FVD (I3D, StyleGAN-V weights)
Efficiency	bytes per image, serialize / deserialize ms per image	bytes per video, serialize / deserialize ms per video

PSNR, SSIM and LPIPS are averaged per sample. FID/FVD compare the set of originals with the set of reconstructions.

Datasets

Name	Domain	Used samples	Preparation
axis-v1_1k	image	1000 images in 8 categories (animals, architecture, cinematic, humans, nature, night/neon, robotics, vehicles)	center crop + resize to 1024x1024
movirec	video	500 clips	first 75 frames, center crop + resize to 256x256

Benchmark phases

Running every storage on every VAE is expensive (and unnecessary), so the benchmark is staged, each stage narrowing the question of the previous one. Each stage has its own config in configs/ and result folder in results/.

Phase	Config	Question	VAEs	Storage configs	Metrics
1. Broad search	lsbench_1.0.yaml	Which families of algorithms are worth keeping?	FLUX.2 (+ no_vae)	73 (all families)	MSE, FID
2. Generalization	lsbench_1.1.yaml	Do the best algorithms rank the same on other VAEs and on video?	5 image + 3 video (+ no_vae)	20 image / 24 video	MSE, PSNR, SSIM, LPIPS, FID / FVD
3. Ablation	lsbench_1.2.yaml	Which features of the best algorithm matter?	FLUX.2	96	MSE, PSNR, SSIM, LPIPS, FID

Not every (VAE, storage, metric) combination is computed on purpose:

1. Phase 1 compares many algorithms from different families (floating point, per-tensor/channel/group integer, Hadamard rotation, outlier side-channels, NormalFloat/FP4, codebooks, block-wise min/max with Lloyd-Max tables) on a single image VAE, only to filter out the worst performers. Early findings: all 8-bit schemes are equivalent; at 4 bits, rotation (Hadamard) hurts, per-channel Lloyd tables give nothing over per-sample tables but are 10x slower to fit, and block-wise min/max + Lloyd-Max tables dominates the quality/size front.

2. Phase 2 applies a hand-picked subset of the best algorithm of each family to different VAEs, image and video. The goal is to show that performance of storage algorithms is highly correlated and mostly VAE-independent, so conclusions of phase 1/3 transfer. Spearman rank correlation of the storage ranking between VAE pairs:

   Metric	Min over VAE pairs	Notes
   image PSNR	0.974	5 VAEs
   image FID	0.928	5 VAEs
   video PSNR	0.986	3 VAEs
   video LPIPS	0.982	3 VAEs
   video FVD	0.630	noisy: 500 clips, LTX-2.5 deviates most

   What does differ is sensitivity, e.g. 4-bit Lloyd-Max (c4_b4_r5) costs 0.2 dB PSNR on FLUX.2 but 0.6-1.1 dB on the other image VAEs.

3. Phase 3 takes the final best algorithm and benchmarks it with many settings (bits 2-8, block 2/4/8/global, range bits 4/6, Lloyd tables on/off) to isolate which feature produces which gain. Again for a single VAE, as phase 2 showed that the result transfers.

Phase 1 reports only MSE and FID; PSNR there is derived from the mean MSE and differs from the per-sample-mean PSNR used in phases 2 and 3, so do not compare PSNR values across phases.

Setup

Requires Linux, Python >= 3.12, and a CUDA GPU (CPU works but is very slow for large VAEs).

uv venv --python=3.12
source .venv/bin/activate
# adjust the index URL to your CUDA version
uv pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu134
uv pip install -r requirements.txt
uv pip install -e .

Datasets

cd datasets
git clone https://huggingface.co/datasets/refine-axis/axis-v1_1k   # images
git clone https://huggingface.co/datasets/madebyollin/movirec      # videos
cd ..

# one-time: crops/resizes into prepared_datasets/, the only folder the loaders read
python scripts/prepare_datasets.py configs/lsbench_1.1.yaml

Models

Only the VAE weights are needed (the vae/ subfolder, or for the standalone VAE repos config.json + diffusion_pytorch_model.safetensors at the repo root). Some repositories are gated: accept the license on the model page and run hf auth login first (SD 3.5, LTX-2.5, MiniMax-H3).

cd models

# image VAEs
hf download black-forest-labs/FLUX.2-klein-4B --include "vae/*" --local-dir FLUX.2-klein-4B
hf download madebyollin/texture-fix-vae-for-qwen-image-2.1 --include "config.json" "diffusion_pytorch_model.safetensors" --local-dir texture-fix-vae-for-qwen-image-2.1
hf download stabilityai/sdxl-vae --include "config.json" "diffusion_pytorch_model.safetensors" --local-dir sdxl-vae
hf download black-forest-labs/FLUX.1-schnell --include "vae/*" --local-dir FLUX.1-schnell
hf download stabilityai/stable-diffusion-3.5-large --include "vae/*" --local-dir stable-diffusion-3.5-large

# video VAEs
hf download Wan-AI/Wan2.1-T2V-1.3B-Diffusers --include "vae/*" --local-dir Wan2.1-T2V-1.3B-Diffusers
hf download Lightricks/LTX-2.5-Diffusers --include "vae/*" --local-dir LTX-2.5-Diffusers
hf download MiniMaxAI/MiniMax-H3 --include "vae/*" --local-dir MiniMax-H3

# I3D detector for the FVD metric
mkdir -p i3d && curl -L -o i3d/i3d_torchscript.pt "https://www.dropbox.com/s/ge9e5ujwgetktms/i3d_torchscript.pt?dl=1"

Usage

# run a benchmark (resumable: finished rows are skipped, new config entries are computed)
python scripts/run_benchmark.py configs/lsbench_1.2.yaml

# quick smoke test on a few samples (results/lsbench_fast/, not tracked by git)
python scripts/run_benchmark.py configs/lsbench_fast.yaml --overwrite

# re-make plots from a results.parquet
python scripts/make_report.py results/lsbench_1.1

# ablation plots of phase 3
python scripts/make_ablation_plots.py results/lsbench_1.2

Each run writes to results/<benchmark_name>/:

• results.parquet: long-format table (domain, vae, storage, metric, value), the source of truth
• tables/<domain>_<metric>.md: VAE x storage pivots
• plots/<vae>/ and plots/all_vaes/: metric vs size plots with Pareto fronts, plus a weighted summary score
• config.yaml: the config used

Results are keyed by (VAE, storage, metric) only. After changing datasets, num_samples or other data settings, use --overwrite or a new benchmark_name, otherwise old and new rows get mixed.

Notes and limitations

• Single run per configuration, no confidence intervals; timings are from one GPU and indicative only.
• Only reconstruction quality is measured, not the effect of quantized latents on downstream training or generation.
• Quality metrics are measured against the original image, so VAE error is included in all numbers.

License

See LICENSE. Datasets and model weights are subject to their own licenses.