DC-BENCH: Dataset Condensation Benchmark

Justin Cui, Ruochen Wang, Si Si, Cho-Jui Hsieh

NeurIPS 2022 · first public 2022-07-20 · arXiv 2207.09639

paper ↗code ↗project page ↗core✓ full text read

In one paragraph

Introduces DC-BENCH, the first large-scale standardized benchmark for dataset condensation, evaluating methods under controlled data-augmentation and architecture choices so that gains attributable to the condensation method itself can be separated from evaluation-pipeline confounds, and releases evaluators, baselines and generated datasets for reproducible comparison.

Explained

What came before

By 2022, DC (gradient matching), DSA (differentiable Siamese augmentation), DM (distribution matching) and MTT/TM (trajectory matching) each reported their own numbers against their own choice of baselines, augmentation and architecture, and each claimed to beat coreset selection (random, K-Center/herding) by a wide margin. No shared protocol existed to check whether these comparisons were apples-to-apples.

The problem

Condensation methods differ not only in synthesis objective but in the augmentation and architecture used to *evaluate* the resulting dataset, and these evaluation-side choices were not controlled for; the paper's own diagnosis is that condensed-dataset quality is "shadowed" by such factors, so reported accuracy gaps may reflect the evaluation recipe rather than the synthetic data.

The idea

Build one benchmark that fixes the evaluation protocol (five augmentation strategies, five architectures, IPC sweeps up to 1000, and a NAS task) and reruns DC, DSA, DM and TM plus random and K-Center selection through it, so every number is directly comparable; use this to test whether condensation's advantage over selection survives once augmentation, architecture and compression ratio are no longer confounds.

How it works

No new synthesis method: DC-BENCH is an evaluation library (open-sourced) with four protocols — (1) accuracy under 5 augmentation strategies (DSA, AutoAugment, RandAugment, ImagenetAug, none), reported as no-augmentation / best-augmentation / average-across-augmentations; (2) accuracy across a sweep of compression ratios (IPC 1 to 1000 on CIFAR-10); (3) cross-architecture transfer (ConvNet, MLP, ResNet-18, ResNet-152, ViT); (4) a Neural Architecture Search task using NAS-Bench-201, measuring Spearman rank correlation between architecture rankings on condensed vs. full data. All experiments standardized to 1x NVIDIA A100 40GB per run; methods reported as OOM if they exceed memory. Also tests initializing DC/DSA/DM/TM's synthetic images from K-Center-selected images instead of random/Gaussian init.

Evidence

All numbers below use hard labels (no soft-label/relabel regime existed yet for these small-scale methods) and are directly comparable within each table since augmentation/architecture are held fixed per row. Augmentation (Table 1, CIFAR-10 IPC10): DC's accuracy rises from 44.43% (no aug) to 47.11% (avg) to 50.99% (best aug), a 6.56-point swing from augmentation choice alone; DSA rises 9.4 points, TM 10.2 points; under best augmentation, K-Center selection (52.96%) beats DC (50.99%) and is close to DSA (52.96% vs DSA's own number) at IPC10 on CIFAR-10, and at IPC50 K-Center even beats or matches DC on CIFAR-100/TinyImageNet. Compression ratio (Fig. 2, CIFAR-10): condensation's edge over random selection is "only obvious for IPCs less than 200"; above IPC200, "all methods perform similarly to the random selection baseline," and DM's claimed advantage over DSA at IPC50 does not hold at larger IPCs. Cross-architecture transfer (Table 2, IPC10): all synthesis methods drop when transferred off their synthesis architecture, rankings are not preserved (DC beats K-Center on ConvNet but not on ResNet), and K-Center transfers better than "several condensation methods"; no method transfers well to ResNet-152 (a data-hungry large model: 45.59% from 300 random images/class vs. 70.29% at IPC1000). NAS (Table 3, NAS-Bench-201, 100 sampled architectures): Spearman rank correlation between condensed-data rankings and full-dataset rankings is *negative* for DC (-0.19), DSA (-0.37), DM (-0.37) and KIP (-0.50), and only weakly positive for K-Center (0.11) and near-zero for random (-0.06) and TM (-0.09) — none reach the original dataset's 0.7487, contradicting an earlier DC paper's reported 0.79 correlation on a smaller 720-architecture toy search space. K-Center initialization for DC/DSA/DM/TM synthesis (Fig. 4) needs "about 30% of the computation budget" to match random-init's end accuracy, and ends 1.3 points higher on average; DSA synthesis itself takes ~15 hours at IPC50 on CIFAR-10 (cost note, Section 4.6).

Limitations

Stated: the benchmark's scope (architectures, datasets, downstream tasks) will need to expand as the field grows; evaluated only on image classification (CIFAR-10/100, TinyImageNet), with no text/graph/audio tasks, at the time of writing. Observed: TM/MTT could not be scaled beyond IPC50 on CIFAR-10 due to memory/runtime and is excluded from the compression-ratio sweep; the NAS evaluation uses a reduced (3-block) architecture during search because full NAS-Bench-201 networks perform poorly on condensed data, which may itself bias the correlation results; only four synthesis methods (DC, DSA, DM, TM) plus KIP (kernel) are covered, predating decoupled/relabel+KD methods (SRe2L and later) entirely, so its findings about augmentation/architecture sensitivity are not shown to generalize to that later paradigm.

Written by the atlas from the paper's full text. Check the paper for exact numbers.

Where it sits

Setting: Image classification

Design choices

Largest scale evaluated

mnist-cifar, tiny-imagenet

Abstract (verbatim from arXiv)

Dataset Condensation is a newly emerging technique aiming at learning a tiny dataset that captures the rich information encoded in the original dataset. As the size of datasets contemporary machine learning models rely on becomes increasingly large, condensation methods become a prominent direction for accelerating network training and reducing data storage. Despite numerous methods have been proposed in this rapidly growing field, evaluating and comparing different condensation methods is non-trivial and still remains an open issue. The quality of condensed dataset are often shadowed by many critical contributing factors to the end performance, such as data augmentation and model architectures. The lack of a systematic way to evaluate and compare condensation methods not only hinders our understanding of existing techniques, but also discourages practical usage of the synthesized datasets. This work provides the first large-scale standardized benchmark on Dataset Condensation. It consists of a suite of evaluations to comprehensively reflect the generability and effectiveness of condensation methods through the lens of their generated dataset. Leveraging this benchmark, we conduct a large-scale study of current condensation methods, and report many insightful findings that open up new possibilities for future development. The benchmark library, including evaluators, baseline methods, and generated datasets, is open-sourced to facilitate future research and application.

BibTeX (generated; prefer the venue's official entry)

@article{cui2022bench,
  title   = {DC-BENCH: Dataset Condensation Benchmark},
  author  = {Justin Cui and Ruochen Wang and Si Si and Cho-Jui Hsieh},
  journal = {NeurIPS 2022},
  year    = {2022}
}