Method · Distribution and feature matching
DM
Dataset Condensation with Distribution Matching
Bo Zhao, Hakan Bilen
WACV 2023 · first public 2021-10-08 · arXiv 2110.04181
In one paragraph
Introduces distribution matching: instead of matching gradients or unrolling training, synthesize images so their class-wise feature means match those of real images across many randomly sampled embedding networks, removing the bi-level optimization entirely and making condensation cheap enough to scale to larger, more realistic datasets.
Explained
What came before
Meta-learning dataset distillation (dd-2018) and gradient matching (dc-2021, dsa-2021) both require expensive bi-level optimization; the paper reports DSA needs about 15 GPU-hours to learn 500 CIFAR-10 images -- comparable to training six deep networks -- and needs several hyperparameters retuned per setting.
The problem
Bi-level optimization, even without full unrolling, is too costly to scale condensation to larger, more realistic datasets and architectures.
The idea
If the synthetic set's per-class feature distribution matches the real data's in the embedding space of many different (mostly randomly initialized) networks, a model trained on it should behave like one trained on the real data -- and this can be enforced directly with no inner-loop training at all.
How it works
Optimizes synthetic pixel images (initialized from randomly sampled real images) by minimizing, for each of many randomly sampled network architectures/initializations, the squared distance between real and synthetic class-wise mean embeddings (an empirical MMD with a linear kernel) computed after Differentiable Siamese Augmentation. No network is ever trained; only the synthetic images receive gradients. Effectively a single hyperparameter (the image learning rate) needs tuning across settings.
Evidence
CIFAR-10, 50 IPC, ConvNet: DM 63.0% vs. DSA 60.6% (Table 1). CIFAR-100, 50 IPC: 43.6% vs. DSA 42.8%. TinyImageNet, 50 IPC: 24.1% vs. Herding 16.7%. ImageNet-1K, 50 IPC: 11.4% (vs. random selection 7.54%). Cross-architecture (CIFAR-10, 50 IPC, trained on ConvNet, evaluated on AlexNet/VGG/ResNet, Table 3): 61.3%/59.9%/57.0% vs. DSA 53.3%/51.0%/47.3%. Continual learning (CIFAR-100, 5-step): 34.4% vs. DSA 31.7%. Cost: under 20 minutes for CIFAR-10 IPC=50 vs. DSA's 15 hours (over 45x speedup, "under 2.2% of the training cost"); TinyImageNet took 27 hours on one V100, ImageNet-1K 28 hours on ten GTX 1080s. Ablation (Table 4): nine different distributions of sampling networks (from purely random to well-trained) all give similar accuracy on CIFAR-10, showing the objective is not sensitive to which networks are sampled. Batch norm instead of instance norm gains +2.2pp (CIFAR-10) and +4.4pp (CIFAR-100).
Limitations
At 1 image/class, DSA outperforms DM on some datasets. Only the first moment (class mean) of the feature distribution is matched; higher moments are left free, which idm-2023, m3d-2024 and datadam-2023 later target directly. Because synthetic images are initialized from and stay close to real images, gains diminish at large IPC as the initial distribution gap shrinks. More sophisticated evaluation architectures (ResNet) yield worse condensation quality than simple ConvNets, indicating some architecture-specific bias.
Written by the atlas from the paper's full text. Check the paper for exact numbers.
Where it sits
- Distribution and feature matching (Surrogate matching)
- Setting: Image classification
Design choices
| What is stored | pixels |
| Labels | hard |
| Prior / networks used | random-networks |
| Optimization regime | single-level |
| Largest scale evaluated | mnist-cifar, tiny-imagenet, imagenet-1k |
Abstract (verbatim from arXiv)
Computational cost of training state-of-the-art deep models in many learning problems is rapidly increasing due to more sophisticated models and larger datasets. A recent promising direction for reducing training cost is dataset condensation that aims to replace the original large training set with a significantly smaller learned synthetic set while preserving the original information. While training deep models on the small set of condensed images can be extremely fast, their synthesis remains computationally expensive due to the complex bi-level optimization and second-order derivative computation. In this work, we propose a simple yet effective method that synthesizes condensed images by matching feature distributions of the synthetic and original training images in many sampled embedding spaces. Our method significantly reduces the synthesis cost while achieving comparable or better performance. Thanks to its efficiency, we apply our method to more realistic and larger datasets with sophisticated neural architectures and obtain a significant performance boost. We also show promising practical benefits of our method in continual learning and neural architecture search.
BibTeX (generated; prefer the venue's official entry)
@article{zhao2021dataset,
title = {Dataset Condensation with Distribution Matching},
author = {Bo Zhao and Hakan Bilen},
journal = {WACV 2023},
year = {2021}
}Nearby in Distribution and feature matching
RAHA — Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation
Jongoh Jeong, Sun-Kyung Lee, Kuk-Jin Yoon · ECCV 2026notableVision–languagepaper ↗code ↗
MDM — Multimodal Distribution Matching for Vision-Language Dataset Distillation
Jongoh Jeong, Hoyong Kwon, Minseok Kim et al. · CVPR 2026notableVision–languagepaper ↗code ↗
Harmonic Dataset Distillation for Time Series Forecasting
Seungha Hong, Sanghwan Jang, Wonbin Kweon et al. · AAAI 2026notableTime seriespaper ↗
Algorithmic Guarantees for Distilling Supervised and Offline RL Datasets
Aaryan Gupta, Rishi Saket, Aravindan Raghuveer · ICLR 2026notableOther datapaper ↗