Method · Gradient matching
DC
Dataset Condensation with Gradient Matching
Bo Zhao, Konda Reddy Mopuri, Hakan Bilen
ICLR 2021 · first public 2020-06-10 · arXiv 2006.05929
In one paragraph
Introduces dataset condensation as gradient matching: at every training step, match the gradients a network computes on a small synthetic set to the gradients it computes on the corresponding class of real data, learning the synthetic images by backpropagating through this per-step matching loss rather than unrolling full training.
Explained
What came before
Responds to dd-2018's bi-level backprop-through-training, which the paper frames as expensive and hard to scale because it must unroll and differentiate through many training steps, and to coreset-selection methods (random, herding, k-center, forgetting), which are cheap but limited to real images and shown to plateau well below synthesized data at the same budget.
The problem
Bi-level dataset distillation (dd-2018) requires backpropagating through an entire (possibly multi-step) inner training procedure to know how changing the synthetic images affects final model performance -- expensive, higher-order, and hard to scale to more steps or larger networks. The paper asks whether a cheaper, local proxy for "trains like the real data" exists.
The idea
Instead of matching final trained performance (which requires unrolling many steps), match the gradient a network computes on the synthetic set to the gradient it computes on the real set, at every step of an ordinary training trajectory -- if the two gradients agree at each step, the parameter trajectories induced by training on either set will stay close, without ever backpropagating through more than one step.
How it works
Optimizes a small per-class synthetic image set (pixels only, fixed hard labels). In the loop: initialize a network from a random $\theta_0\sim P_{\theta_0}$, and at each of T outer steps compute the classification-loss gradient $\nabla_\theta\mathcal{L}^{\mathcal{S}}(\theta_t)$ on a synthetic class-batch and $\nabla_\theta\mathcal{L}^{\mathcal{T}}(\theta_t)$ on the corresponding real class-batch, then minimize their distance $D(\cdot,\cdot)$ (Eq. 9) by updating the synthetic images via one SGD step; periodically update $\theta_t$ itself by training on the synthetic images. The distance is a layer-wise, per-output-node cosine distance (Eq. 10) -- gradients are grouped by output node rather than flattened across the whole network -- which the paper finds is the effective choice (vs. a single flattened Euclidean or Cosine distance over all parameters). The whole procedure is repeated over many random initializations $\theta_0$ so the synthetic set is not tied to one network.
Evidence
Table 1 (ConvNet, hard labels, coreset baselines vs. Ours vs. whole dataset): MNIST IPC1/10/50 = 91.7+/-0.5 / 97.4+/-0.2 / 98.8+/-0.2% (full 99.6+/-0.0%); FashionMNIST IPC1/10/50 = 70.5+/-0.6 / 82.3+/-0.4 / 83.6+/-0.4% (full 93.5+/-0.1%); SVHN IPC1/10/50 = 31.2+/-1.4 / 76.1+/-0.6 / 82.3+/-0.3% (full 95.4+/-0.1%); CIFAR-10 IPC1/10/50 = 28.3+/-0.5 / 44.9+/-0.5 / 53.9+/-0.5% (full 84.8+/-0.1%), beating random/herding/k-center/forgetting coreset selection at every IPC on every dataset. CIFAR-100 (Table T20, appendix): IPC1/10 = 12.8+/-0.3 / 25.2+/-0.3% (full 56.2+/-0.3%); no IPC50 reported. Cost (Table T22): on MNIST/LeNet, DD uses 785MB/160min for 79.5+/-8.1% vs. DC's 653MB/46min for 93.9+/-0.6%; on CIFAR-10/AlexCifarNet, DD uses 3211MB/214min for 36.8+/-1.2% vs. DC's 1445MB/105min for 39.1+/-1.2% -- less memory, less time, and higher accuracy than dd-2018's bi-level BPTT at the same budget. Ablation (Table T18, MNIST IPC1, matched train/test architecture): the layer-wise per-output-node cosine distance ("Ours") reaches 70.5/91.7/85.0/82.7/81.7/89.4% on MLP/ConvNet/LeNet/AlexNet/VGG/ResNet vs. a flattened whole-network Euclidean distance's 69.3/92.7/65.0/66.2/57.1/68.0% and a flattened Cosine distance's 45.2/69.2/61.1/58.3/55.0/68.8% -- the layer-wise grouping (not cosine similarity alone) is what keeps accuracy high on deeper architectures (LeNet/AlexNet/VGG/ResNet), where flattened distances degrade sharply.
Limitations
Does not use data augmentation during matching (identified and fixed by dsa-2021, its direct successor); only reports IPC 1/10/50 on MNIST/FashionMNIST/SVHN/CIFAR-10 and IPC 1/10 (no 50) on CIFAR-100, with no Tiny-ImageNet or ImageNet-scale result. The single-step gradient-matching signal is short-horizon by construction -- it only guarantees local agreement between synthetic- and real-data gradients at sampled points along a trajectory, not that the trajectories stay close over many steps, a gap dcc-2022 shows can make it underperform random selection on fine-grained data and that mtt-2022 addresses directly by matching multi-step trajectories instead.
Written by the atlas from the paper's full text. Check the paper for exact numbers.
Where it sits
- Gradient matching (Surrogate matching)
- Setting: Image classification
Design choices
| What is stored | pixels |
| Labels | hard |
| Prior / networks used | random-networks |
| Optimization regime | single-level |
| Largest scale evaluated | mnist-cifar |
Abstract (verbatim from arXiv)
As the state-of-the-art machine learning methods in many fields rely on larger datasets, storing datasets and training models on them become significantly more expensive. This paper proposes a training set synthesis technique for data-efficient learning, called Dataset Condensation, that learns to condense large dataset into a small set of informative synthetic samples for training deep neural networks from scratch. We formulate this goal as a gradient matching problem between the gradients of deep neural network weights that are trained on the original and our synthetic data. We rigorously evaluate its performance in several computer vision benchmarks and demonstrate that it significantly outperforms the state-of-the-art methods. Finally we explore the use of our method in continual learning and neural architecture search and report promising gains when limited memory and computations are available.
BibTeX (generated; prefer the venue's official entry)
@article{zhao2020dataset,
title = {Dataset Condensation with Gradient Matching},
author = {Bo Zhao and Konda Reddy Mopuri and Hakan Bilen},
journal = {ICLR 2021},
year = {2020}
}Nearby in Gradient matching
Linear Gradient Matching — Dataset Distillation for Pre-Trained Self-Supervised Vision Models
George Cazenavette, Antonio Torralba, Vincent Sitzmann · NeurIPS 2025notablePre-training & transferpaper ↗code ↗
Distilled Datamodel with Reverse Gradient Matching
Jingwen Ye, Ruonan Yu, Songhua Liu et al. · CVPR 2024notablepaper ↗
Static-dynamic video DD — Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglement
Ziyu Wang, Yue Xu, Cewu Lu et al. · CVPR 2024coreVideopaper ↗code ↗