Method · Diffusion-based synthesis
DAP
Diffusion Models as Dataset Distillation Priors
Duo Su, Huyu Wu, Huanran Chen, Yiming Shi, Yuzhu Wang, Xi Ye, Jun Zhu
ICLR 2026 · first public 2025-10-20 · arXiv 2510.17421
In one paragraph
Formalizes the representativeness already implicit in a pretrained diffusion model by quantifying feature-space similarity between synthetic and real data with a Mercer kernel, then uses it as training-free guidance for the reverse diffusion process instead of adding external constraints; reports state-of-the-art fidelity and cross-architecture generalization on ImageNet-1K and its subsets.
Explained
What came before
Minimax and MGD3 both add hand-designed representativeness/diversity objectives on top of a pretrained diffusion model -- Minimax by fine-tuning with minimax criteria, MGD3 by mode-discovery guidance -- treating representativeness as something that must be imposed from outside the generator.
The problem
The paper argues these external constraints are unnecessary complexity: a well-trained diffusion model already has usable diversity and generalization priors (shown via low, matched train/test negative log-likelihood, Table 1) simply by virtue of its score-matching training objective and architecture- agnostic feature extraction, but no prior work exploits the model's own internal features as a representativeness signal, so they resort to bolt-on losses or clustering instead.
The idea
Decompose the desired conditional score $\nabla_x\log p(x|\mathcal{R})=\nabla_x\log p(x)+\nabla_x\log p(\mathcal{R}|x)$: the first (unconditional) term already gives diversity+generalization for free from the pretrained model; the second (representativeness) term can be built, with no retraining, from a Mercer-kernel-induced distance between the diffusion backbone's own intermediate features on synthetic versus real samples, then injected as classifier-style energy guidance during sampling.
How it works
Treat the diffusion U-Net or DiT backbone as a feature map $\phi:\mathcal{X}\to\mathbb{R}^n$; define representativeness as a Boltzmann distribution over the average Mercer-kernel-induced distance $\mathcal{D}_\mathcal{K}$ between a synthetic sample's features and features of real training samples of the same class (linear kernel used in practice for tractability), proven (Theorems 3.1-3.2) to be a valid RKHS-norm distance so gradients are well-behaved. During VP-SDE reverse sampling, at each step the noisy synthetic latent and a noised real training sample of the target class are both passed through the same frozen backbone layer (the paper ablates which layer: U-Net "Mid" block for SD, early transformer blocks 4-12 for DiT); the gradient of the feature distance is added to the standard score/ noise prediction, scaled by guidance strength $\gamma$ (Algorithm 1). No feature extractor, generator, or classifier is trained; the pretrained SD-v1.5 (general text-to-image) or DiT-XL/2-256 (ImageNet- pretrained, as used by Minimax) backbone is reused as both generator and its own representativeness critic.
Evidence
ImageNet-1K, DiT backbone, ResNet-18 soft-label protocol (Table 2), IPC10/50: 49.1±1.2 / 62.7±1.5, beating IGD 45.5±0.5/59.8±0.3, MGD3 45.6±0.8/60.2±0.1, D3HR 44.3±0.3/59.4±0.1, VLCP 46.7±0.4/60.5±0.2 and Minimax 44.3±0.5/58.6±0.3 under one shared protocol. ImageNette/ImageWoof, hard-label protocol (Table 3), IPC10 ResNet-18: Nette 66.4±0.5 (IGD 67.7±0.3 slightly ahead, attributed by the authors to IGD's architecture-specific influence guidance baked in for ResNet-18); Woof IPC50 ResNetAP-10: 63.3±0.5 vs IGD 62.7±1.2, Minimax 59.8±0.8, MGD3 56.5±1.9. Isolating ablation (Fig. 5): feature-layer choice matters (mid U-Net layer / early DiT transformer blocks are best; final layers, which favor distribution alignment over representativeness, are worse) and increasing guidance scale $\gamma$ raises representativeness (visualized as larger similarity "sector area" in Fig. 2 and higher accuracy) up to a point, after which it distorts the diversity/generalization gradient field and accuracy drops -- directly isolating the representativeness-guidance mechanism from the base model's inherent priors. t-SNE (Fig. 4) shows synthetic features matching real train/test manifolds with intra-class diversity preserved. Cost: training-free (no fine-tuning), so its added cost is only extra backbone forward/ backward passes for guidance at each sampling step; exact wall-clock not extracted from the main text.
Limitations
Stated: future work needed to extend to newer generators (FLUX, SD3.5) and beyond vision. Observed: when applied to DiT, the backbone (DiT-XL/2-256) is itself pretrained on the full ImageNet target dataset, so DAP's strongest ImageNet-1K numbers still depend on a target-trained generator -- exactly the dependency CoDA later criticizes across the family; the guidance scale $\gamma$ requires per- backbone/per-layer tuning (Fig. 5) and is reported to destabilize the other priors if set too high, so the training-free claim trades fine-tuning cost for a sampling-time hyperparameter search; the representativeness distance is computed per-class against real training samples at every guidance step, meaning (unlike D3HR/CoDA style one-shot statistics) it needs repeated access to real data features throughout sampling, not just at initialization.
Written by the atlas from the paper's full text. Check the paper for exact numbers.
Where it sits
- Diffusion-based synthesis (Generative priors)
- Setting: Image classification
Design choices
| Prior / networks used | diffusion, text-to-image-diffusion |
| Optimization regime | training-free |
| What is stored | pixels |
| Largest scale evaluated | imagenet-subsets, imagenet-1k |
Builds on
Abstract (verbatim from arXiv)
Dataset distillation aims to synthesize compact yet informative datasets from large ones. A significant challenge in this field is achieving a trifecta of diversity, generalization, and representativeness in a single distilled dataset. Although recent generative dataset distillation methods adopt powerful diffusion models as their foundation models, the inherent representativeness prior in diffusion models is overlooked. Consequently, these approaches often necessitate the integration of external constraints to enhance data quality. To address this, we propose Diffusion As Priors (DAP), which formalizes representativeness by quantifying the similarity between synthetic and real data in feature space using a Mercer kernel. We then introduce this prior as guidance to steer the reverse diffusion process, enhancing the representativeness of distilled samples without any retraining. Extensive experiments on large-scale datasets, such as ImageNet-1K and its subsets, demonstrate that DAP outperforms state-of-the-art methods in generating high-fidelity datasets while achieving superior cross-architecture generalization. Our work not only establishes a theoretical connection between diffusion priors and the objectives of dataset distillation but also provides a practical, training-free framework for improving the quality of the distilled dataset.
BibTeX (generated; prefer the venue's official entry)
@article{su2025diffusion,
title = {Diffusion Models as Dataset Distillation Priors},
author = {Duo Su and Huyu Wu and Huanran Chen and Yiming Shi and Yuzhu Wang and Xi Ye and Jun Zhu},
journal = {ICLR 2026},
year = {2025}
}Nearby in Diffusion-based synthesis
Learnability-guided diffusion — Learnability-Guided Diffusion for Dataset Distillation
Jeffrey A. Chan-Santiago, Mubarak Shah · CVPR 2026notablepaper ↗
ManifoldGD — ManifoldGD: Training-Free Hierarchical Manifold Guidance for Diffusion-Based Dataset Distillation
Ayush Roy, Wei-Yang Alex Lee, Rudrasis Chakraborty et al. · CVPR 2026notablepaper ↗code ↗