Method · Distribution and feature matching
DataDAM
DataDAM: Efficient Dataset Distillation with Attention Matching
Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z. Liu, Yuri A. Lawryshyn, Konstantinos N. Plataniotis
ICCV 2023 · first public 2023-09-29 · arXiv 2310.00093
In one paragraph
Matches spatial attention maps computed at multiple layers of randomly initialized networks between real and synthetic batches, instead of matching gradients or single-layer features, reporting up to 6.5-point gains on CIFAR-100 and 4.1-point gains on ImageNet-1K over prior methods.
Explained
What came before
dm-2023 matches only the class-wise mean feature (a single statistic) between real and synthetic data; cafe-2022's richer feature alignment can bias images and adds time cost; gradient- and trajectory-matching methods (dc-2021, dsa-2021, mtt-2022) need bi-level optimization or stored expert trajectories and can be memory-hungry.
The problem
Mean-only feature matching under-constrains the spatial structure that actually discriminates classes, while methods that add richer supervision (CAFE) sacrifice the efficiency that makes distribution matching attractive in the first place.
The idea
Match the spatial attention maps produced by multiple layers of randomly initialized networks (which encode where a network is "looking") in addition to a single top-layer MMD term, giving synthetic images multi-scale spatial supervision while remaining as cheap as plain distribution matching.
How it works
Optimizes synthetic pixel images. For several random ConvNets, computes spatial attention at every intermediate layer as the normalized sum of absolute feature-map values across channels; the Spatial Attention Matching (SAM) loss is the MSE between real and synthetic normalized attention maps at each layer, and a complementary MMD loss aligns the final-layer embeddings (loss = SAM + lambda*MMD). No inner-loop training of the encoders (random init only) and no classification loss.
Evidence
CIFAR-10, 50 IPC: 67.0% vs. DM 63.0% (Table 1, +4.0pp). CIFAR-100, 50 IPC: 49.4% vs. DM 42.9% (+6.5pp). TinyImageNet, 50 IPC: 28.7% vs. DM 25.3%. ImageNet-1K (64x64), 50 IPC: 15.5% vs. DM 11.4% (+4.1pp, Table 2). Cost (Table 4, CIFAR-10, per-step): DataDAM 0.09-0.16s and 3.45-3.72GB vs. DC 0.16s/3.3-4.5GB, DSA 0.22s, and MTT 0.36s with OOM at IPC=50 -- almost two orders of magnitude faster than DC/DSA and comparable to DM's own cost. Ablation (Table 5, CIFAR-10, IPC=10): MMD-only 48.9%, SAM-only 49.8%, feature-map transfer without attention 47.2%, SAM+MMD combined 54.2% -- neither loss alone is sufficient.
Limitations
Authors state the formulation is limited to convolutional networks and struggles to generalize to Vision Transformers. Like all fixed-ratio distillation methods, it must be re-optimized whenever the IPC (distillation ratio) changes rather than supporting a single reusable representation across budgets.
Written by the atlas from the paper's full text. Check the paper for exact numbers.
Where it sits
- Distribution and feature matching (Surrogate matching)
- Setting: Image classification
Design choices
| What is stored | pixels |
| Labels | hard |
| Prior / networks used | random-networks |
| Optimization regime | single-level |
| Largest scale evaluated | mnist-cifar, tiny-imagenet, imagenet-1k |
Abstract (verbatim from arXiv)
Researchers have long tried to minimize training costs in deep learning while maintaining strong generalization across diverse datasets. Emerging research on dataset distillation aims to reduce training costs by creating a small synthetic set that contains the information of a larger real dataset and ultimately achieves test accuracy equivalent to a model trained on the whole dataset. Unfortunately, the synthetic data generated by previous methods are not guaranteed to distribute and discriminate as well as the original training data, and they incur significant computational costs. Despite promising results, there still exists a significant performance gap between models trained on condensed synthetic sets and those trained on the whole dataset. In this paper, we address these challenges using efficient Dataset Distillation with Attention Matching (DataDAM), achieving state-of-the-art performance while reducing training costs. Specifically, we learn synthetic images by matching the spatial attention maps of real and synthetic data generated by different layers within a family of randomly initialized neural networks. Our method outperforms the prior methods on several datasets, including CIFAR10/100, TinyImageNet, ImageNet-1K, and subsets of ImageNet-1K across most of the settings, and achieves improvements of up to 6.5% and 4.1% on CIFAR100 and ImageNet-1K, respectively. We also show that our high-quality distilled images have practical benefits for downstream applications, such as continual learning and neural architecture search.
BibTeX (generated; prefer the venue's official entry)
@article{sajedi2023datadam,
title = {DataDAM: Efficient Dataset Distillation with Attention Matching},
author = {Ahmad Sajedi and Samir Khaki and Ehsan Amjadian and Lucy Z. Liu and Yuri A. Lawryshyn and Konstantinos N. Plataniotis},
journal = {ICCV 2023},
year = {2023}
}Nearby in Distribution and feature matching
RAHA — Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation
Jongoh Jeong, Sun-Kyung Lee, Kuk-Jin Yoon · ECCV 2026notableVision–languagepaper ↗code ↗
MDM — Multimodal Distribution Matching for Vision-Language Dataset Distillation
Jongoh Jeong, Hoyong Kwon, Minseok Kim et al. · CVPR 2026notableVision–languagepaper ↗code ↗
Harmonic Dataset Distillation for Time Series Forecasting
Seungha Hong, Sanghwan Jang, Wonbin Kweon et al. · AAAI 2026notableTime seriespaper ↗
Algorithmic Guarantees for Distilling Supervised and Offline RL Datasets
Aaryan Gupta, Rishi Saket, Aravindan Raghuveer · ICLR 2026notableOther datapaper ↗