landmark

Diffusion Transformers with Representation Autoencoders

Boyang Zheng, Nanye Ma, Shengbang Tong, Saining Xie · NYU · 2025-10 · arXiv:2510.11690 · code

Why it matters

Replaces the VAE with a Representation Autoencoder: a frozen pretrained encoder (DINO, SigLIP, or MAE) plus a lightweight trained ViT decoder. The resulting latent is high-dimensional (width must be >= token dimension) and semantically rich rather than reconstruction-only; needs a dimension-dependent noise schedule shift and noise-augmented decoding to train stably, and a DiT^DH diffusion head to decode well. Strong reconstruction *and* generation simultaneously, closing the dilemma VA-VAE/REPA-E only partially closed.

What this paper does

read: full text

Before this

Every diffusion transformer since latent diffusion inherited Stable Diffusion's autoencoder more or less unchanged, while research attention went to the denoiser.

The problem

That autoencoder compresses a 256-pixel image to a 32 by 32 grid of 4 channels using a reconstruction-only objective on a legacy convolutional design. The result captures local appearance but carries little global semantic structure, and costs roughly six times more encoding FLOPs than the alternative.

The idea

Stop training an encoder. Take a pretrained representation model, freeze it, treat its features as the generative latent, and train only a decoder back to pixels.

How it works

DINOv2-B, SigLIP2-B or MAE-B serve as the frozen encoder; a ViT-XL decoder is trained with L1, LPIPS and adversarial losses. Unlike a VAE this preserves token count (around 256) while keeping high per-token dimensionality (768 channels). Three obstacles had to be cleared. First, a transformer whose width is below the token dimension cannot fit the data, which the paper proves as a loss lower bound, so width must meet or exceed 768. Second, noise schedules inherited from pixel and VAE inputs ignore token dimensionality; a dimension-dependent shift takes gFID from 23.08 to 4.81. Third, the decoder is trained on a discrete set of clean latents but must decode noisy diffusion output, so training it with added Gaussian noise takes gFID from 4.81 to 4.28. To meet the width requirement without quadratic cost, DiT^DH pairs a standard backbone with a shallow two-layer head of width 2048.

Evidence

gFID 1.51 without guidance and 1.13 with autoguidance at 800 epochs, against DiT-XL's 9.62 and 2.27 at 1400 epochs. At 80 epochs it already reaches 2.16 unguided. Reconstruction FID is 0.49 against SD-VAE's 0.62.

Limitations

Two findings bound the claim. Raw pixels at matched dimensionality reach only FID 51.09, so the gain comes from semantic structure and not from dimensionality alone. And DiT^DH gives no benefit on low-dimensional VAE latents, scoring 11.70 against a standard DiT's 7.13, so the architecture is specific to this regime rather than generally better.

Why it matters

It reverses the relationship between reconstruction and generation. In the VAE picture the latent exists to be decodable and generation consumes it; here the latent exists because it is meaningful, and decodability becomes the decoder's problem. The specificity of the fixes is itself the evidence that high-dimensional semantic latents are a different regime, not a drop-in swap.

Abstract, in the authors' own words

Latent generative modeling, where a pretrained autoencoder maps pixels into a latent space for the diffusion process, has become the standard strategy for Diffusion Transformers (DiT); however, the autoencoder component has barely evolved. Most DiTs continue to rely on the original VAE encoder, which introduces several limitations: outdated backbones that compromise architectural simplicity, low-dimensional latent spaces that restrict information capacity, and weak representations that result from purely reconstruction-based training and ultimately limit generative quality. In this work, we explore replacing the VAE with pretrained representation encoders (e.g., DINO, SigLIP, MAE) paired with trained decoders, forming what we term Representation Autoencoders (RAEs). These models provide both high-quality reconstructions and semantically rich latent spaces, while allowing for a scalable transformer-based architecture. Since these latent spaces are typically high-dimensional, a key challenge is enabling diffusion transformers to operate effectively within them. We analyze the sources of this difficulty, propose theoretically motivated solutions, and validate them empirically. Our approach achieves faster convergence without auxiliary representation alignment losses. Using a DiT variant equipped with a lightweight, wide DDT head, we achieve strong image generation results on ImageNet: 1.51 FID at 256x256 (no guidance) and 1.13 at both 256x256 and 512x512 (with guidance). RAE offers clear advantages and should be the new default for diffusion transformer training.

Research line

Representation-space generationcontested

Reported results

Benchmark	Value	Guidance	Budget	Source
ImageNet 256x256 gFID	1.51	none	800 epochs	Table 8
ImageNet 256x256 gFID	1.13	AutoGuidance	800 epochs	Table 8
ImageNet 256x256 rFID	0.49	—	—	Table 1
ImageNet 256x256 rFID	0.16	—	—	Table 1

AutoGuidance is a stronger guidance method than the plain CFG used by the VAE baselines in most cross-paper comparisons, and contributes a large share of the gap.

MAE-B gives RAE's best reconstruction but not its best generation — a compact illustration that rFID does not predict gFID.

Design-axis choices

Representation

Semantic / foundation latent

Frozen encoder + trained decoder

Objective

Flow matching / rectified flow

Architecture

Diffusion Transformer (DiT)

DiT with diffusion head (DiT^DH)

Problem

Standard VAE latents (e.g. SD-VAE) are optimized purely for pixel reconstruction, so they carry little semantic structure — giving a diffusion model a low-information target to learn.
A single latent space optimized for pixel-reconstruction fidelity is often a poor space for a generative prior to learn in, and pushing on one side tends to hurt the other.
Diffusion transformers were built around low-dimensional latents (4-16 channels); RAE-style frozen encoders produce latents with hundreds to thousands of channels. Does standard diffusion training still work well at that dimensionality, and what has to change?

Method note — the shared flow-matching interpolation

Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:

x_t = (1-t)\,x_0 + t\,x_1, \qquad v_\theta(x_t, t) \approx x_1 - x_0

Builds on

builds_on Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think — Sec. 1: REPA showed alignment helps; RAE asks why not generate in that space.
builds_on Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models — Both attack the reconstruction-generation dilemma by aligning the encoder to a foundation model; RAE goes further and freezes the encoder entirely instead of just aligning it.
uses_representation_from DINOv2: Learning Robust Visual Features without Supervision — DINOv2 is one of the three frozen encoder families RAE evaluates as the generative latent.
uses_architecture_from Scalable Diffusion Models with Transformers — RAE's diffusion transformer is a DiT variant (with an added diffusion-head decoder stage).

Built on by

Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders builds_on this — Scale-RAE takes RAE's ImageNet-scale recipe and scales the decoder to open-vocabulary text-to-image data.
Improved Baselines with Representation Autoencoders improves this — RAEv2 fixes RAE's classifier-free-guidance instability and roughly 10x's convergence speed via multilayer representation aggregation and a re-parameterized DiT output.
Distilling Drifting Transformers with Representation Autoencoders builds_on this — Distilling Drifting Transformers targets RAE's specific latent space for a distillation method built around its geometry.

Challenges / competes with

challenges VAE latent — RAE replaces the VAE outright with a frozen pretrained encoder plus a lightweight trained decoder.
challenges Back to Basics: Let Denoising Generative Models Denoise — JiT's pixel-space return directly challenges the representation-space generation line's premise that a learned (semantic or otherwise) latent is needed.
competes_with Latent Diffusion Model without Variational Autoencoder — SVG and RAE independently propose the same core move (frozen foundation encoder as the generative latent) within days of each other, differing mainly in encoder choice and detail recovery mechanism.
competes_with TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation — TokenFlow's dual-codebook (semantic + pixel) design and RAE's single frozen-encoder-plus-trained-decoder design are alternative answers to the same underlying question: how much of reconstruction detail must live alongside semantic content in one representation.