Dataset Condensation via Efficient Synthetic-Data Parameterization

What came before

dc-2021 and dsa-2021 match network gradients between real and synthetic batches inside a bi-level loop, training the matching networks on the synthetic data itself; dm-2023's feature matching is faster but reported to underperform gradient matching. All prior methods store condensed data pixel-for-pixel at the original resolution, one image per stored unit.

The problem

Under a fixed storage budget, pixel-for-pixel storage does not exploit the spatial regularity of natural data (nearby pixels are correlated), so the number of synthetic samples the budget can hold is unnecessarily small; separately, gradient matching against networks trained on the tiny synthetic set causes those networks to overfit early, so the real/synthetic gradient norms diverge and matching quality (measured with MSE-style objectives) degrades during optimization.

The idea

Store each condensed unit at reduced resolution and deterministically upsample it into several images at matching time (multi-formation), so a fixed pixel budget yields more, regularized training images; combine this with an improved gradient-matching recipe that computes gradients from networks trained on the real data (not the synthetic data), matched by a distance-based (MSE) rather than cosine objective, with stronger network regularization.

How it works

A multi-formation function f partitions each stored image into equal tiles and bilinearly upsamples each tile back to full resolution (factor 2 for CIFAR-10/Speech Commands, factor 3 for ImageNet), turning n stored elements into n' > n training images under the same total pixel budget; f is differentiable, so synthetic pixels are optimized end-to-end through it. Gradients for matching are computed from a network theta_T periodically re-initialized and trained a few epochs on the real data (not the synthetic data), using CutMix and augmentation for stronger regularization, with an MSE (not cosine) distance between per-class real and synthetic gradients. Hard labels throughout; no learned labels or teacher.

Evidence

CIFAR-10, 50 images/class budget (equal pixel storage), ConvNet-3, hard labels: IDC 74.5% vs. DSA 60.6%, DM 65.6%, and IDC-I (same optimization, no multi-formation) 69.5% (Table 1) -- isolating multi-formation's contribution at ~5pp on top of the optimization fix's ~9pp gain over DM. ImageNet-10, 20 images/class, ResNetAP-10: IDC 76.6% vs. DM 59.3%, IDC-I 65.5% (Table 3). Speech Commands, 20 spectrograms/class: IDC 86.6% vs. DM 77.2%, IDC-I 83.0% (Table 4). Storage accounting is total pixels (equal-storage comparison with pixel-space DSA/DM/KIP baselines): e.g. "50x32x32" pixels/class is the same budget whether stored as 50 full images or 200 half-resolution tiles. Ablation (Table 5, CIFAR-10, IPC=50): DSA (Syn-trained networks + cosine) 60.6% -> Real-trained networks + cosine 63.4% -> Real-trained + MSE 67.0% -> + regularization 69.5% (this is IDC-I; multi-formation is not yet added), separating the optimization fix from the parameterization gain. Ablation on multi-formation factor (Table 7): factor 2 is optimal for CIFAR-10 (74.5%) but factor 3-4 is optimal for ImageNet-10 (76.6/77.5%), i.e. the ideal number-of-images- vs-resolution trade-off is dataset- and resolution-dependent. Post-downsampling comparison (Table 6): IDC's end-to-end optimized multi-formation (74.5% at 50x32x32 px) beats naively downsampling then upsampling a 4x larger stored set (68.8% at 200x16x16 px), showing the gain is from co-optimizing storage with the formation function, not just from resolution reduction itself.

Limitations

IDC requires more training steps to converge than baselines because of the formation process, though a fixed-training-step comparison (Table 2) still favors IDC. Optimal multi-formation factor is dataset- and resolution-dependent and must be tuned (2 for CIFAR-10, 3-4 for ImageNet), with no automatic selection rule. No GPU-hour or wall-clock synthesis cost is reported (only per-step evaluation training time in Table 2, which measures downstream training, not condensation cost). The matching objective is gradient matching, not distribution matching; the paper explicitly discusses (Section "Discussion on Dataset Structure") that deviating from same-shape storage toward dictionary codes or generator parameterizations was considered but not pursued because such structures can require more storage or compute than they save.