Method · Diffusion-based synthesis
Pushforward quantization
Dataset Distillation as Pushforward Optimal Quantization
Hong Ye Tan, Emma Slade
ICLR 2026 · first public 2025-01-13 · arXiv 2501.07681
In one paragraph
Shows that decoupled/disentangled dataset-distillation methods with an encoder-decoder structure are instances of classical optimal quantization and Wasserstein-barycenter problems, and uses this connection to propose clustering in a diffusion model's latent space to select prototype points, reporting better performance and cross-model generalization than D4M on ImageNet-1K, plus state-of-the-art results using the distilled noise initializations with a stronger diffusion transformer.
Explained
What came before
D4M clusters a target dataset's latents into per-class prototypes and decodes them through a pretrained latent diffusion model with no theoretical account of why clustering should approximate the data distribution well; IGD and Minimax (used later on a DiT backbone) add guidance or fine-tuning without a consistency argument either.
The problem
No prior dataset-distillation work proves that a distilled set (produced by clustering-and-decoding) is a provably good approximation of the population risk gradient as the budget grows -- the field lacks a theoretical account of why disentangled, encoder-decoder-based methods like D4M work, or what governs their error.
The idea
Disentangled distillation methods with an encoder-decoder structure are literally instances of optimal quantization / Wasserstein-barycenter problems: choosing K prototype points per class to minimize expected projection distance to the class's latent distribution. Framed this way, (a) a proper optimal-quantization clustering (weighted mini-batch k-means with per-cluster population weights, not unweighted k-means or a Wasserstein-barycenter objective) provably minimizes distortion and gives convergent gradient-risk approximations as more points are added, and (b) diffusing the population forward and clustering in the (higher-fidelity) diffusion latent space at a partial noise level preserves this guarantee via a Wasserstein-continuity bound on diffusion SDEs.
How it works
A four-step pipeline: (1) encode the real per-class training data into a pretrained diffusion model's latent space (LDM/UNet in the main experiments, DiT in a secondary set); (2) run mini-batch k-means (equivalent to a competitive-learning vector-quantization, CLVQ, algorithm) per class in that latent space, recording each cluster's population count $v_k$; (3) decode the resulting cluster centers through the pretrained (frozen, untrained/unfine-tuned) diffusion decoder to get the distilled images; (4) train the student with a population-count-derived per-sample weight $w_k = v_k/\sum_j v_j$ folded into a weighted cross-entropy loss, $\min_\theta \sum_{(x,y,w)} w\cdot\ell(x,y,\theta)$, rather than treating every distilled sample as equally likely. Theorem 1 bounds the difference in expectation of any Lipschitz function between two latent distributions diffused backward to a partial time $\delta$ by their Wasserstein-2 distance at $t=T$, times a Lipschitz constant, justifying clustering at a partial (not full) noise level; Corollary 1 gives an $O(K^{-1/d})$ convergence rate in quantization points $K$ and latent dimension $d$.
Evidence
ImageNet-1K, soft-label (ResNet-18 teacher, ceiling ~69.8), Table 2: IPC10, ResNet-18/50/101: 33.1+/-0.60 / 34.4+/-0.99 / 36.7+/-0.80 vs D4M 27.9 / 33.5 / 34.2 (numbers for D4M taken from the original paper); IPC50: 56.2+/-0.07 / 62.5+/-0.24 / 63.6+/-0.13 vs D4M 55.2 / 62.4 / 63.4; IPC200, ResNet-101: 68.6+/-0.08 vs D4M 68.1, a "30% reduction in error gap" at IPC200/ResNet-101. On a stronger DiT backbone (Table 4): ImageNet-1K IPC10/50: 53.0+/-0.2 / 62.7+/-0.1 vs IGD 45.5+/-0.5 / 59.8+/-0.3 and Minimax-IGD 46.2+/-0.6 / 60.3+/-0.4; ImageWoof IPC10/50: 48.8+/-2.0 / 65.4+/-0.7 vs IGD 41.0+/-0.8 / 62.7+/-1.2, Minimax-IGD 43.3+/-0.3 / 65.0+/-0.8. No MGD3, D3HR or CaO2 comparison anywhere. Wasserstein validation (Table 1): the population-weighted clustering reduces latent W2 distance to the target vs D4M's unweighted clustering by 15.7% at IPC10 and 16.1% at IPC50 (class 0 example: 49.67 to 42.97 at IPC10). Cross-architecture generalization (Table 3, ResNet-18 teacher): uniformly beats D4M on ResNet-18/MobileNet-V2/EfficientNet-B0 students, slightly underperforms D4M with a Swin-T student. Cost: "trivial additional computation" over D4M (Appendix G); memory constant in IPC, matching D4M's advantage over bi-level methods.
Limitations
Stated: k-means/Lloyd's-algorithm clustering can converge to a stationary but not globally optimal grid in higher dimensions; the consistency proof covers this specific quantization-based construction only, explicitly leaving "consistency or convergence of other dataset distillation frameworks such as bi-level methods and diffusion guidance" as open questions the paper does not resolve; the DiT-backbone gains are empirical, and the paper notes that sharper bounds exploiting sub-Gaussianity remain future work. Observed: no MGD3, D3HR or CaO2 comparison, so it is untested against the family's mode-guidance and inversion-latent-statistics branches; slightly underperforms D4M for transformer-architecture students (Swin-T), suggesting the quantization-optimality argument does not uniformly transfer across evaluator architectures; baseline D4M numbers on the UNet backbone are quoted from the original paper rather than re-run under the paper's own protocol.
Written by the atlas from the paper's full text. Check the paper for exact numbers.
Where it sits
- Diffusion-based synthesis (Generative priors)
- Setting: Image classification
Design choices
| Prior / networks used | diffusion |
| Optimization regime | training-free |
| What is stored | pixels |
| Largest scale evaluated | imagenet-subsets, imagenet-1k |
Builds on
Abstract (verbatim from arXiv)
Dataset distillation aims to find a synthetic training set such that training on the synthetic data achieves similar performance to training on real data, with orders of magnitude less computational requirements. Existing methods can be broadly categorized as either bi-level optimization problems that have neural network training heuristics as the lower level problem, or disentangled methods that bypass the bi-level optimization by matching distributions of data. The latter method has the major advantages of speed and scalability in terms of size of both training and distilled datasets. We demonstrate that when equipped with an encoder-decoder structure, the empirically successful disentangled methods can be reformulated as an optimal quantization problem, where a finite set of points is found to approximate the underlying probability measure by minimizing the expected projection distance. In particular, we link existing disentangled dataset distillation methods to the classical optimal quantization and Wasserstein barycenter problems, demonstrating consistency of distilled datasets for diffusion-based generative priors. We propose Dataset Distillation by Optimal Quantization, based on clustering in a latent space. Compared to the previous SOTA method D\textsuperscript{4}M, we achieve better performance and inter-model generalization on the ImageNet-1K dataset with trivial additional computation, and SOTA performance in higher image-per-class settings. Using the distilled noise initializations in a stronger diffusion transformer model, we obtain SOTA distillation performance on ImageNet-1K and its subsets, outperforming diffusion guidance methods.
BibTeX (generated; prefer the venue's official entry)
@article{tan2025dataset,
title = {Dataset Distillation as Pushforward Optimal Quantization},
author = {Hong Ye Tan and Emma Slade},
journal = {ICLR 2026},
year = {2025}
}Nearby in Diffusion-based synthesis
Learnability-guided diffusion — Learnability-Guided Diffusion for Dataset Distillation
Jeffrey A. Chan-Santiago, Mubarak Shah · CVPR 2026notablepaper ↗
ManifoldGD — ManifoldGD: Training-Free Hierarchical Manifold Guidance for Diffusion-Based Dataset Distillation
Ayush Roy, Wei-Yang Alex Lee, Rudrasis Chakraborty et al. · CVPR 2026notablepaper ↗code ↗