Research line · ascendant
Representation-aligned latents
Keep the autoencoder, but apply pressure from a vision foundation model so its latent (or the generator's hidden states) become semantically structured.
What defines membership
The VAE is worth keeping; what needs fixing is the absence of semantic structure, and an alignment loss can supply it.
How the line developed
Read top to bottom: what came before, the idea itself, the evidence for it, what improved, and where it breaks.
The idea
REPA · 2024-10landmark
Aligning a diffusion transformer's hidden states to frozen DINOv2 features speeds training 17.5x with no change to the latent or architecture.
Evidence
MAETok · 2025-02core
Reframes the target as latent geometry: fewer Gaussian-mixture modes and more discriminative features, no variational form required.
Improvement
VA-VAE / LightningDiT · 2025-01core
Moves the alignment pressure into VAE training itself, and names the reconstruction-generation dilemma explicitly.
REPA-E · 2025-04core
Makes VAE and diffusion transformer trainable end to end through the alignment loss, which naive joint training destroys.
REG · 2025-07core
Entangles a semantic token into the denoising target rather than aligning hidden states as a side loss.
What it gets right
- Large speedups with minimal disruption to an existing, well-understood stack
- Directly measurable: convergence-rate gains are large and reproducible
- Compatible with the entire VAE-latent ecosystem
Where it is weak
- Treats the symptom rather than the cause — the latent is still a reconstruction code
- Adds a dependency on a frozen encoder anyway, which invites the question RAE asks
Reported numbers
As published, with the guidance method, budget, and model size that produced them. Seethe comparison page for why these cannot be ranked naively.
| Paper | Benchmark | Value | Guidance | Budget | Params |
|---|---|---|---|---|---|
| MAETok | ImageNet 256x256 gFID | 2.31 | none | 4M steps | 675M |
| MAETok | ImageNet 256x256 gFID | 1.67 | cfg=2.0, 250 steps | 4M steps | 675M generator + 176M tokenizer |
| REPA | ImageNet 256x256 gFID | 5.9 | none | 4M iters | 675M |
| REPA | ImageNet 256x256 gFID | 1.42 | guidance-interval | 800 epochs (~4M iters) | 675M |
| REPA-E | ImageNet 256x256 gFID | 1.69 | none | 800 epochs | 675M |
| REPA-E | ImageNet 256x256 gFID | 1.12 | CFG (scale not stated) | 800 epochs | 675M |
| VA-VAE / LightningDiT | ImageNet 256x256 gFID | 2.17 | none | 800 epochs | 675M |
| VA-VAE / LightningDiT | ImageNet 256x256 gFID | 1.35 | cfg-interval + timestep shift (scale not stated) | 800 epochs | 675M |
| MAETok | ImageNet 256x256 rFID | 0.48 | — | — | — |
| VA-VAE / LightningDiT | ImageNet 256x256 rFID | 0.28 | — | — | — |
Competing answers
Open problems it has not solved
- A single latent space optimized for pixel-reconstruction fidelity is often a poor space for a generative prior to learn in, and pushing on one side tends to hurt the other.
- Standard VAE latents (e.g. SD-VAE) are optimized purely for pixel reconstruction, so they carry little semantic structure — giving a diffusion model a low-information target to learn.
- Two independent groups agree DINOv2 is the best encoder to build a generative model on, and disagree about why. Nobody knows which property actually causes the gain.
Also in this line
REPA spatial-structure study