core
Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders
Shengbang Tong, Boyang Zheng, Ziteng Wang, Bingda Tang, Nanye Ma, Ellis Brown, Jihan Yang, Rob Fergus, Yann LeCun, Saining Xie · NYU, Meta AI (FAIR) · 2026-01 · arXiv:2601.16208 · code
Why it matters
Tests whether the RAE recipe survives the jump from ImageNet to open-vocabulary text-to-image: scales a SigLIP-2-frozen RAE decoder on web, synthetic, and text-rendering data. Scale improves general fidelity but targeted data composition is still required for specific domains like rendered text; RAE-based models stay stable through 256 fine-tuning epochs where VAE-based models catastrophically overfit by epoch 64.
What this paper does
read: full textBefore this
Representation autoencoders had been demonstrated on class-conditional ImageNet, a controlled setting with a thousand labels and one domain.
The problem
Whether the recipe survives open-vocabulary text-to-image, where prompts are freeform and the decoder must handle far more than a thousand categories.
The idea
Scale the frozen-encoder recipe to text-to-image and find out what actually limits it: raw data volume, or what the data contains.
How it works
SigLIP-2 So400M at patch 14 is the frozen encoder, giving 1152-dimensional tokens, chosen because language-supervised features are semantically structured enough to serve both understanding and generation. Text conditioning follows the MetaQuery framework directly: 256 learnable query tokens are prepended to the text prompt, a pretrained language model processes text and queries jointly, and a two-layer MLP connector projects into the diffusion transformer's hidden space. The decoder is scaled on web, synthetic and text-rendering data.
Evidence
The finding worth carrying is about data composition rather than scale. Generic web and synthetic data gave only marginal gains on ImageNet and moderate gains on more diverse images, while text rendering required targeted supervision — without text-specific data the decoder fails to reproduce glyph detail. Separately, under fine-tuning on high-quality data, VAE-based models degrade sharply after 64 epochs while RAE models stay stable through 256 with only mild decline.
Limitations
The visual-understanding results are explicitly described as preliminary exploration rather than competitive, and the paper notes that reaching that level would need any-resolution inputs, multimodal continual pretraining and much higher quality data. Test-time scaling gains are modest and framed as a first step.
Why it matters
It answers the scaling question for the representation-latent line and supplies its most distinctive evidence: stability under long fine-tuning, which is a different kind of claim from a better score and harder to attribute to tuning. It also settles how this line does conditioning, by adopting MetaQuery wholesale rather than designing a mechanism for the semantic latent.
Abstract, in the authors' own words
Representation Autoencoders (RAEs) have shown distinct advantages in diffusion modeling on ImageNet by training in high-dimensional semantic latent spaces. In this work, we investigate whether this framework can scale to large-scale, freeform text-to-image (T2I) generation. We first scale RAE decoders on the frozen representation encoder (SigLIP-2) beyond ImageNet by training on web, synthetic, and text-rendering data, finding that while scale improves general fidelity, targeted data composition is essential for specific domains like text. We then rigorously stress-test the RAE design choices originally proposed for ImageNet. Our analysis reveals that scaling simplifies the framework: while dimension-dependent noise scheduling remains critical, architectural complexities such as wide diffusion heads and noise-augmented decoding offer negligible benefits at scale Building on this simplified framework, we conduct a controlled comparison of RAE against the state-of-the-art FLUX VAE across diffusion transformer scales from 0.5B to 9.8B parameters. RAEs consistently outperform VAEs during pretraining across all model scales. Further, during finetuning on high-quality datasets, VAE-based models catastrophically overfit after 64 epochs, while RAE models remain stable through 256 epochs and achieve consistently better performance. Across all experiments, RAE-based diffusion models demonstrate faster convergence and better generation quality, establishing RAEs as a simpler and stronger foundation than VAEs for large-scale T2I generation. Additionally, because both visual understanding and generation can operate in a shared representation space, the multimodal model can directly reason over generated latents, opening new possibilities for unified models.
Research line
Reported results
| Benchmark | Value | Guidance | Budget | Source |
|---|---|---|---|---|
| DPG-Bench | 76.9 | — | ~30K iterations | Table 4 |
| GenEval | 0.495 | — | ~30K iterations | Table 3 |
Same budget-limited ablation caveat as the GenEval row.
Scale-RAE is a controlled scaling study, not a system release. Reading this as a state-of-the-art claim is a category error. The legitimate comparison is its within-paper ablation at matched budget: RAE 49.5 versus a VAE baseline at 39.6.
Design-axis choices
Objective
Conditioning & control
Method note — the shared flow-matching interpolation
Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:
Builds on
- uses_architecture_from Transfer between Modalities with MetaQueries — Scale-RAE's paper states it explicitly: "We adopt the MetaQuery architecture for text-to-image (T2I) generation." It prepends 256 learnable query tokens to the prompt, processes them jointly with a Qwen-2.5 1.5B LLM, and projects the result through a 2-layer MLP connector into the DiT backbone — the same query-bridge mechanism MetaQuery introduced, applied on top of RAE's frozen-encoder latent.
- builds_on Diffusion Transformers with Representation Autoencoders — Scale-RAE takes RAE's ImageNet-scale recipe and scales the decoder to open-vocabulary text-to-image data.
- uses_representation_from SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features — Scale-RAE's frozen encoder is SigLIP-2, chosen for its language-aligned features at text-to-image scale.
Built on by
Nothing recorded yet.
Challenges / competes with
- competes_with SVG-T2I: Scaling Up Text-to-Image Latent Diffusion Model Without Variational Autoencoder — SVG-T2I and Scale-RAE are the direct competing answers to 'does the frozen-encoder recipe scale to text-to-image', published about a month apart.