strong-followup
Back to Basics: Let Denoising Generative Models Denoise
Tianhong Li, Kaiming He · MIT · 2025-11 · arXiv:2511.13720 · code
Why it matters
Argues pixel-space diffusion was never unstable because of pixels — it was unstable because of epsilon-prediction. Switching to x-prediction on large patches lets a plain ViT ('Just image Transformers') match tokenizer-based DiT/PixArt-alpha at 256-1024px with no VAE, no pretraining, and no auxiliary losses at all.
What this paper does
read: full textBefore this
Pixel-space diffusion was widely believed to fail at high resolution, and the field's answer was to compress into a latent first. Attempts to fix pixel space reached for dense convolutions and architectural workarounds.
The problem
The authors argue that diagnosis was simply wrong. The instability was never about pixels; it was about what the network was asked to predict.
The idea
Natural images lie on a low-dimensional manifold, but a noised quantity is spread across the full high-dimensional space. Predicting clean data only requires representing the manifold; predicting noise requires representing everything. A limited-capacity network can do the first and fails at the second.
How it works
Just image Transformers is a plain Vision Transformer on raw non-overlapping patches, 16 by 16 or 32 by 32 pixels, giving tokens of 768 or 3072 dimensions, with standard blocks and adaLN-Zero conditioning. It predicts the clean patch directly rather than noise or velocity. No tokenizer, no pretraining, no auxiliary losses.
Evidence
The decisive result is the contrast, not the absolute number: at 200 epochs on ImageNet 256, x-prediction reaches FID 4.37 while epsilon- and v-prediction both exceed 96, a catastrophic failure under an otherwise identical setup. Scaling to 600 epochs gives 2.36 at Large, 1.86 at Huge and 1.82 at Giant.
Limitations
The authors position the work as foundational rather than state of the art. They deliberately forgo gains available from self-supervised pretraining or auxiliary losses, introduce no new architectural components, evaluate only on ImageNet, and do not discuss wall-clock or memory cost in depth.
Why it matters
If a plain transformer on raw pixels is competitive once the prediction target is fixed, then the entire argument about which latent is best was optimizing something that may be optional. This is the sharpest live challenge to the premise every latent-based line shares.
Abstract, in the authors' own words
Today's denoising diffusion models do not "denoise" in the classical sense, i.e., they do not directly predict clean images. Rather, the neural networks predict noise or a noised quantity. In this paper, we suggest that predicting clean data and predicting noised quantities are fundamentally different. According to the manifold assumption, natural data should lie on a low-dimensional manifold, whereas noised quantities do not. With this assumption, we advocate for models that directly predict clean data, which allows apparently under-capacity networks to operate effectively in very high-dimensional spaces. We show that simple, large-patch Transformers on pixels can be strong generative models: using no tokenizer, no pre-training, and no extra loss. Our approach is conceptually nothing more than "Just image Transformers", or JiT, as we call it. We report competitive results using JiT with large patch sizes of 16 and 32 on ImageNet at resolutions of 256 and 512, where predicting high-dimensional noised quantities can fail catastrophically. With our networks mapping back to the basics of the manifold, our research goes back to basics and pursues a self-contained paradigm for Transformer-based diffusion on raw natural data.
Research line
Reported results
| Benchmark | Value | Guidance | Budget | Source |
|---|---|---|---|---|
| ImageNet 256x256 gFID | 1.82 | CFG-interval | 600 epochs | Table 6 |
Uses roughly 2B parameters against the 675-839M of the latent-space rows, and a guidance-interval technique. No unguided number is reported, so this cannot be compared against the unguided column. Any claim that pixel-space now beats latent-space is confounded by scale and guidance simultaneously.
Design-axis choices
Representation
Objective
Problem
- Is a learned latent space necessary for diffusion transformers at all, or can a correctly reparameterized pixel-space model (JiT) match latent-space performance with no tokenizer?
Builds on
Nothing recorded yet.
Built on by
Nothing recorded yet.
Challenges / competes with
- challenges VAE latent — JiT trains competitive diffusion transformers directly on raw pixel patches, challenging the assumption that a VAE latent space is necessary at all.
- challenges Diffusion Transformers with Representation Autoencoders — JiT's pixel-space return directly challenges the representation-space generation line's premise that a learned (semantic or otherwise) latent is needed.
This is a strong-followup entry — a short-form summary. Full paper-page explanations (before/problem/ core idea/representation/architecture/objective/conditioning/training/inference/results/ ablations/limitations) are written for landmark and core papers first; see PROJECT_STATE.md for the schedule.