Method · Gradient matching
DSA
Dataset Condensation with Differentiable Siamese Augmentation
Bo Zhao, Hakan Bilen
ICML 2021 · first public 2021-02-16 · arXiv 2102.08259
In one paragraph
Adds differentiable Siamese augmentation — applying the same random augmentation to a real and a synthetic batch before computing the gradient-matching loss — reporting roughly 7-point accuracy gains on CIFAR-10 and CIFAR-100 over the unaugmented gradient-matching baseline.
Explained
What came before
Responds directly to dc-2021 (DC), whose gradient matching is done on raw, unaugmented images even though the resulting condensed set is later trained with standard data augmentation, and to dd-2018 (DD) and LD (label distillation), which DSA also compares against on the same benchmarks.
The problem
Standard data augmentation (crop, flip, color, etc.) usually helps training, but applying it naively to a condensed set is wasted or even harmful: the images were optimized without ever "seeing" augmentation, so an augmented view of a synthetic image may no longer carry the same information the gradient-matching objective packed into the original pixels.
The idea
Make the condensed images "augmentation-ready" by exposing gradient matching itself to augmentation: at every matching step, sample one random augmentation and apply the *same* transformation (parameterized by the same random draw $\omega$) to both the real and the synthetic batch before computing their gradients, so the images are optimized to remain informative under the transformation rather than only in their un-augmented form.
How it works
Keeps dc-2021's per-class synthetic image set (pixels, hard labels) and per-step gradient-matching loss $D(\nabla_\theta\mathcal{L}(\mathcal{S},\theta_t),\nabla_\theta\mathcal{L}(\mathcal{T},\theta_t))$, but replaces the loss's inputs with gradients computed on augmented batches $\mathcal{A}(\mathcal{S},\omega)$ and $\mathcal{A}(\mathcal{T},\omega)$, where $\omega$ (the augmentation's random parameters -- crop location, rotation angle, etc.) is drawn once per step and applied identically ("Siamese") to both batches (Eq. 4-5, summed over a family of augmentation types $\Omega$). Everything else -- repeated random network initializations, one forward-backward pass per step, no unrolling -- is unchanged from dc-2021.
Evidence
Table 1 (ConvNet, hard labels): MNIST IPC1/10/50 = 88.7+/-0.6 / 97.8+/-0.1 / 99.2+/-0.1% (full 99.6+/-0.0%, DC 91.7/97.4/98.8%); FashionMNIST IPC1/10/50 = 70.6+/-0.6 / 84.6+/-0.3 / 88.7+/-0.2% (full 93.5+/-0.1%, DC 70.5/82.3/83.6%); SVHN IPC1/10/50 = 27.5+/-1.4 / 79.2+/-0.5 / 84.4+/-0.4% (full 95.4+/-0.1%, DC 31.2/76.1/82.3% -- DSA is slightly below DC at IPC1 here); CIFAR-10 IPC1/10/50 = 28.8+/-0.7 / 52.1+/-0.5 / 60.6+/-0.5% (full 84.8+/-0.1%, DC 28.3/44.9/53.9% -- the roughly 7-10 point CIFAR-10 gain over DC at IPC10/50 that the paper's abstract headlines). CIFAR-100 IPC1/10 = 13.9+/-0.3 / 32.3+/-0.3% (full 56.2+/-0.3%, DC 12.8/25.2%). States "less than 1% data" (50/class) reaches 99.6, 94.9, 88.5, 71.5% *relative* to full-data accuracy on MNIST/FashionMNIST/SVHN/CIFAR-10 respectively. Cross-architecture (Table 3, condensed on row architecture C, evaluated on column architecture T, CIFAR-10 IPC10): condensing on ConvNet transfers at 75.6-89.3% across MLP/ConvNet/LeNet/AlexNet/ VGG/ResNet, with all six condensation architectures transferring within a similar band -- the paper's evidence that DSA's gain is not architecture-specific. Ablation (Table 5, CIFAR-10 IPC50, evaluated under four different test-time augmentations): the full method (Siamese augmentation at condense time, standard augmentation at test time) reaches 49.1/47.9/46.9/46.8% under Crop/Flip/Scale/Rotation vs. 45.5% flat with no augmentation anywhere (variant A); applying augmentation independently (not Siamese) to the real and synthetic batches during condensing (variant C, 42.8/46.2/44.5/44.5%) or only at test time with no condense-time augmentation (variant B, 46.9/46.1/45.7/45.0%) both underperform the shared/Siamese variant -- isolating that *consistency* between the real and synthetic augmentation during matching, not just more augmented data, drives the gain.
Limitations
Still inherits dc-2021's single-step, short-horizon gradient-matching objective and its dependence on the specific matching architecture; only tested at the same MNIST/FashionMNIST/SVHN/CIFAR-10/100 small-scale benchmarks as dc-2021, no Tiny-ImageNet or ImageNet result. The augmentation family $\Omega$ (crop, flip, scale, rotation, color, cutout) and its parameterization are fixed choices, not learned; DSA slightly underperforms DC on SVHN at IPC1 (27.5% vs. 31.2%), showing the augmentation fix is not a uniform win at the smallest budgets. No GPU-hour or memory figure is reported.
Written by the atlas from the paper's full text. Check the paper for exact numbers.
Where it sits
- Gradient matching (Surrogate matching)
- Setting: Image classification
Design choices
| What is stored | pixels |
| Labels | hard |
| Prior / networks used | random-networks |
| Optimization regime | single-level |
| Largest scale evaluated | mnist-cifar |
Abstract (verbatim from arXiv)
In many machine learning problems, large-scale datasets have become the de-facto standard to train state-of-the-art deep networks at the price of heavy computation load. In this paper, we focus on condensing large training sets into significantly smaller synthetic sets which can be used to train deep neural networks from scratch with minimum drop in performance. Inspired from the recent training set synthesis methods, we propose Differentiable Siamese Augmentation that enables effective use of data augmentation to synthesize more informative synthetic images and thus achieves better performance when training networks with augmentations. Experiments on multiple image classification benchmarks demonstrate that the proposed method obtains substantial gains over the state-of-the-art, 7% improvements on CIFAR10 and CIFAR100 datasets. We show with only less than 1% data that our method achieves 99.6%, 94.9%, 88.5%, 71.5% relative performance on MNIST, FashionMNIST, SVHN, CIFAR10 respectively. We also explore the use of our method in continual learning and neural architecture search, and show promising results.
BibTeX (generated; prefer the venue's official entry)
@article{zhao2021dataset,
title = {Dataset Condensation with Differentiable Siamese Augmentation},
author = {Bo Zhao and Hakan Bilen},
journal = {ICML 2021},
year = {2021}
}Nearby in Gradient matching
Linear Gradient Matching — Dataset Distillation for Pre-Trained Self-Supervised Vision Models
George Cazenavette, Antonio Torralba, Vincent Sitzmann · NeurIPS 2025notablePre-training & transferpaper ↗code ↗
Distilled Datamodel with Reverse Gradient Matching
Jingwen Ye, Ruonan Yu, Songhua Liu et al. · CVPR 2024notablepaper ↗
Static-dynamic video DD — Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglement
Ziyu Wang, Yue Xu, Cewu Lu et al. · CVPR 2024coreVideopaper ↗code ↗