strong-followup
A Frozen Pixel-Space Diffusion Model Can Guide Itself with Its Own Samples
· Nanyang Technological University · 2026-07 · arXiv:2607.29122
Why it matters
Adds a lightweight self-guidance head on a frozen pixel-space backbone, improving JiT from 1.86 to 1.67 and PixelREPA from 1.81 to 1.59 for under one percent extra compute. An instance of the atlas's recurring finding that inference-time technique moves these numbers as much as architecture does.
What this paper does
read: full textBefore this
Pixel-space diffusion models, which generate directly over raw pixels instead of a VAE latent, had been improving through new prediction targets, training objectives, and architectures such as JiT and PixelREPA, but each such gain required training a new model from scratch. Existing guidance methods such as classifier-free guidance improve quality at sampling time but need a separately trained (or jointly trained) unconditional counterpart and add a second forward pass, without making use of anything specific to a pixel-space model's internal structure.
The problem
Find a cheap way to improve the samples of an already-trained, frozen pixel diffusion model without retraining its backbone, exploiting the model's own internal representations rather than external supervision.
The idea
Intermediate layers of a pretrained pixel diffusion transformer already decode into coarse, low-frequency structure, while later layers progressively refine high-frequency detail. Attaching a small trainable head to an intermediate layer of a frozen backbone and using the gap between that head's coarse prediction and the backbone's final prediction as a self-guidance direction improves samples for a small fraction of the original training cost, and the head trains better on the model's own generated samples than on real images.
How it works
The backbone is entirely frozen throughout; the only trained component is a lightweight prediction head attached to one intermediate transformer layer, trained on model-generated (synthetic) samples using under 1% of the backbone's full training compute, roughly 50 epochs over 1 million synthetic samples compared to about 600 epochs for the base model. At sampling time the head produces an intermediate prediction x_inter and the backbone produces a final prediction x_final; these are combined as x_guided = x_inter + w * (x_final - x_inter), where w=1 recovers the unguided output and w>1 extrapolates further. This self-guidance is applied only within a restricted diffusion-time interval t in [0.1, 1.0], using 50 Heun sampling steps, with typical scales of 1.1 to 1.4, and it composes with standard classifier-free guidance applied over the same interval.
Evidence
On ImageNet 256x256 with classifier-free guidance, JiT-H/16 improves from FID 1.86 to 1.67, PixelREPA-H/16 from FID 1.81 to 1.59, and DeCo-XL/16 from FID 1.69 to 1.63. Without classifier-free guidance, the gains are much larger: JiT-H/16 goes from FID 7.15 to 2.26, JiT-L/16 from 13.85 to 4.47, and JiT-B/16 from 25.42 to 9.47. On ImageNet 512x512 with classifier-free guidance, JiT-H/32 improves from FID 1.94 to 1.84. All results use 50 Heun sampling steps.
Limitations
The authors state their experiments are mainly class-conditional ImageNet generation with pixel-space diffusion models, and it remains unclear whether the intermediate-to-final refinement they observe also generalizes to larger text-conditioned models or to domains with different image distributions.
Why it matters
It shows an inference-time technique built from a frozen model's own internal structure can move FID as much as, or more than, a new architecture or training objective, for a small fraction of the compute of retraining, and that a model's own generated samples can be better training data than real images for fitting the high-frequency detail pixel diffusion tends to underfit.
Abstract, in the authors' own words
Pixel-space diffusion models aim to learn an end-to-end generator directly over raw pixels. This is challenging because a single model must capture both global structure and local texture in the same high-dimensional space. While recent work improves pixel diffusion through alternative prediction targets, training objectives, and architectures, these advances typically require training a new model from scratch. We show there is a cheaper, complementary strategy: \textbf{a frozen, pretrained pixel diffusion model can guide itself}. Our key observation is that intermediate layers of a pretrained pixel diffusion transformer can be decoded into coarse predictions that capture the main low-frequency structure, while the final layers progressively refine local, high-frequency details. We therefore attach a lightweight prediction head to an intermediate layer, keep the backbone frozen, and use the discrepancy between the intermediate and final predictions as a self-guidance direction during sampling. To train this head, we further find that real images are not necessary. Instead, model-generated samples suffice and even outperform real images for training the head, especially in enhancing the high-frequency components that pixel diffusion tends to underfit. Across multiple pixel diffusion models on ImageNet, our \textbf{Synthetic Self-Guidance (SSG)} consistently improves generation while adapter training requires less than 1$\%$ of full-model training compute: it reduces FID by over 50$\%$ across the evaluated JiT variants without classifier-free guidance (CFG) and further improves strong baselines with CFG, e.g., JiT-H/16 from 1.86 to 1.67 and PixelREPA-H/16 from 1.81 to 1.59. Our code is available at https://github.com/zfu006/SSG.
Research lines
Builds on
Nothing recorded yet.
Built on by
Nothing recorded yet.
This is a strong-followup entry — a short-form summary. Full paper-page explanations (before/problem/ core idea/representation/architecture/objective/conditioning/training/inference/results/ ablations/limitations) are written for landmark and core papers first; see PROJECT_STATE.md for the schedule.