Dataset Distillation

What came before

Responds to network/model distillation (Hinton et al., 2015), which compresses a trained model rather than a dataset, and to coreset construction, instance selection and active-learning subset methods, which select real "valuable" examples rather than synthesize new ones and therefore need far more than one example per class. Also related to, but going further than, gradient-based hyperparameter optimization (Maclaurin et al. 2015), which backpropagates through training to tune hyperparameters rather than the training data itself.

The problem

Asks how much information a training set actually contains and whether that information can be repackaged as a much smaller synthetic set. Standard training needs tens of thousands to millions of gradient steps over the full data; the paper's diagnosis is that nothing in principle requires the training examples themselves, rather than some other small set of images, to be the vehicle for that information.

The idea

Treat the pixels of a handful of synthetic images (and the learning rate) as the parameters being optimized, and define their loss as the real-data loss of a network after it takes a few gradient steps on them starting from a given initialization. Because that composed function is differentiable end to end, the synthetic images can be learned by backpropagating through the inner gradient steps.

How it works

Optimizes M synthetic images $\tilde{\mathbf{x}}$ and per-step learning rates $\tilde{\eta}$; the network architecture is fixed and untrained. For a fixed initialization $\theta_0$, one or more GD steps on $\tilde{\mathbf{x}}$ produce $\theta_1,\theta_2,\dots$, and the outer objective is the real training-data loss $\ell(\mathbf{x},\theta_T)$, minimized over $\tilde{\mathbf{x}},\tilde{\eta}$ by backpropagating through the T inner steps. For random initializations the objective is the expectation over $\theta_0\sim p(\theta_0)$ (Eq. 4), so the same synthetic set must work for many sampled networks (4-16 samples per outer step). Multi-step, multi-epoch unrolling is made tractable with back-gradient optimization (Hessian-vector products via Pearlmutter's trick) instead of naive backprop-through-time. Class labels are fixed (hard), not optimized. A linear-regression special case (Sec. 3.3) shows that for one GD step and arbitrary initialization, at least M >= D synthetic points are needed, where D is the input dimension.

Evidence

MNIST/LeNet, fixed init, 10 images total (1/class), 10 GD steps x 3 epochs: 12.90% -> 93.76% test accuracy vs. 99% fully trained (Fig. 2a); CIFAR-10, fixed init, 100 images (10/class): 8.82% -> 54.03% vs. 80% fully trained (Fig. 2b, Table 1). Random init, 100 images: MNIST 79.5+/-8.1%, CIFAR-10 36.8+/-1.2% (Table 1), beating k-means, average-real, random-real and optimized-real-image baselines at the same image budget. Cross-domain adaptation (100 images) among MNIST/USPS/SVHN beats a few-shot domain-adaptation baseline (Motiian et al. 2017) on 2/3 transfers (Table 2). Fixed ImageNet-pretrained AlexNet adapted with 1 image/class: PASCAL-VOC 70.75% (vs. 19.41% random-real, 75.57% full fine-tune) and CUB-200 38.76% (vs. 7.11% random-real, 41.21% full fine-tune) (Table 3). Ablation: at matched total image budget, multiple GD steps drastically outperform a single step with a larger batch (Fig. 5), and more epochs help but saturate (Fig. 4) -- isolating that spreading a fixed budget over more optimization steps, not just more pixels, is what makes the images informative. Cost: each training run took 1-4 hours on a single Titan Xp/V100 GPU (fixed init) or 4 GPUs (random init, more initializations sampled per step); no epoch/wall-clock comparison to full-data training is given.

Limitations

States it is sensitive to the distribution of initializations p(theta_0) and only tests MNIST, CIFAR-10 and small transfer-learning benchmarks (PASCAL-VOC, CUB-200) at low resolution -- no results at CIFAR-100/Tiny-ImageNet/ImageNet scale. Labels are hard and tied to a fixed class assignment, not learned. The linear-case analysis shows a hard lower bound (M >= input dimension D) for one-step, arbitrary-initialization distillation, foreshadowing why later work moved to random/expert-trajectory ensembles or closed-form inner solvers. Backpropagating through even a handful of inner steps requires second-order (Hessian-vector) computation, which does not scale to the many steps or large architectures later families needed -- the scaling problem the family's subsequent papers (kernel-closed-form, gradient-matching, trajectory-matching) were created to avoid.