Research line · dominant
VAE-latent diffusion
Compress images with a reconstruction-trained autoencoder, then run diffusion or flow matching in that latent space.
What defines membership
The generative latent should be a compact code learned by an autoencoder trained to reconstruct pixels.
How the line developed
Read top to bottom: what came before, the idea itself, the evidence for it, what improved, and where it breaks.
The idea
LDM / Stable Diffusion · 2021-12landmark
Establishes that diffusion in a pretrained autoencoder's latent is orders of magnitude cheaper than in pixels, at negligible quality cost.
Evidence
SiT · 2024-01core
Isolates objective, coupling, and sampler as independent choices on an unchanged DiT backbone.
Improvement
DiT · 2022-12landmark
Swaps the UNet for a plain transformer and shows diffusion obeys transformer scaling laws.
DC-AE · 2024-10strong-followup
Pushes spatial compression to 128x, trading latent size against reconstruction difficulty.
HiDream-I1 · 2025-05strong-followup
Sparse rather than dense scaling for the same quality tier.
Lumina-Image 2.0 · 2025-03strong-followup
Efficiency-focused variant with a Gemma-2 text encoder.
At scale
SD3 / MMDiT · 2024-03landmark
Flow matching plus dual-stream MM-DiT at production scale; the recipe most open systems still follow.
Qwen-Image · 2025-08core
Vision-language text encoder and a data pipeline built for text rendering; the open baseline of 2025-26.
Seedream 4.0 · 2025-09core
Generation, editing, and multi-image composition in one industrial system at 4K.
Qwen-Image 2.0 · 2026-05core
The 2026 successor generation of the leading open family.
What it gets right
- Enormous ecosystem: weights, LoRAs, tooling, and community fine-tunes all assume it
- Well-understood training dynamics and failure modes
- Cheap inference relative to pixel-space at high resolution
Where it is weak
- The latent is semantically impoverished, so the generator spends capacity rediscovering structure a foundation model already has
- Reconstruction-optimized latents are demonstrably not the easiest latents to generate in (see the reconstruction-generation dilemma)
Reported numbers
As published, with the guidance method, budget, and model size that produced them. Seethe comparison page for why these cannot be ranked naively.
| Paper | Benchmark | Value | Guidance | Budget | Params |
|---|---|---|---|---|---|
| Lumina-Image 2.0 | DPG-Bench | 87.2 | — | — | 2.6B |
| Qwen-Image | DPG-Bench | 88.32 | — | — | 20B |
| Lumina-Image 2.0 | GenEval | 0.73 | — | — | 2.6B |
| Qwen-Image | GenEval | 0.91 | — | — | 20B |
| Qwen-Image | GenEval | 0.87 | — | — | 20B MMDiT + Qwen2.5-VL 7B encoder |
| SD3 / MMDiT | GenEval | 0.74 | — | — | 8B + 4.7B T5-XXL |
| DiT | ImageNet 256x256 gFID | 9.62 | none | 7M steps | 675M |
| DiT | ImageNet 256x256 gFID | 2.27 | cfg=1.5 | 7M steps | 675M |
| SiT | ImageNet 256x256 gFID | 8.3 | none | 7M steps | 675M |
| SiT | ImageNet 256x256 gFID | 2.06 | cfg=1.5 | 7M steps | 675M |
| DC-AE | ImageNet 256x256 rFID | 0.69 | — | — | — |
| LDM / Stable Diffusion | ImageNet 256x256 rFID | 0.62 | — | — | — |
Competing answers
Open problems it has not solved
- Standard VAE latents (e.g. SD-VAE) are optimized purely for pixel reconstruction, so they carry little semantic structure — giving a diffusion model a low-information target to learn.
- A single latent space optimized for pixel-reconstruction fidelity is often a poor space for a generative prior to learn in, and pushing on one side tends to hurt the other.
Also in this line
Chimera, LLaDA-Image