Embarassingly Simple Dataset Distillation

What came before

Responds to dd-2018/BPTT's memory and instability problems, and to the family of inner-loop surrogates that were built to avoid unrolling entirely -- kip-2021 (NTK), rfad-2022 (NNGP/random features), frepo-2022 (empirical feature kernel), rcig-2023 (convexified implicit gradients) -- as well as to outer-loop surrogate objectives dc-2021/dsa-2021 (gradient matching) and mtt-2022/ftd-2023 (trajectory matching), arguing all of these either restrict the loss to MSE and favor wide networks, or use proxy metrics whose alignment with true test accuracy is unclear.

The problem

Diagnoses exactly why plain BPTT underperforms: (1) large memory/compute from storing the whole unrolled trajectory, (2) unstable meta-gradients from long products of Hessian matrices over a highly non-convex inner problem (training a network from scratch), and (3) that naive truncated BPTT (T-BPTT), while stabilizing gradients, discards the early-training information that matters most (T-BPTT converges faster initially but plateaus below full BPTT).

The idea

Combine truncation (only backpropagate through a short window of M steps, for stability and memory) with randomization (place that window at a uniformly random position along a longer N-step unroll, resampled every outer step) so the resulting gradient is an unbiased-in-position subsample of the full BPTT gradient that still covers the entire trajectory over many outer steps, rather than only ever seeing the same (early or late) window.

How it works

Optimizes the distilled set $\mathcal{U}$ directly by gradient descent on pixels (and, for larger IPC/many-class settings, learned labels), with no kernel, surrogate loss, or precomputed expert trajectories. Each outer step: sample an unrolling length $N\in[M,T]$, initialize $\theta_0\sim p(\theta)$, take $N$ inner SGD/Adam steps on $\mathcal{U}$, but only accumulate gradients for the last $M$ steps (i.e., start backpropagating once $n=N-M$), then update $\mathcal{U}$ from the resulting meta-gradient $\mathcal{G}_{RaT\text{-}BPTT}$ -- a random M-length subsample of the terms in the full BPTT sum (Eq. 4). No model pool, no wide-network bias, no MSE restriction; works for any architecture and any differentiable loss, using the Higher library for efficient meta-gradient computation. A separate Boost-DD algorithm (Algorithm 2) builds a nested, budget-flexible dataset by distilling successive IPC-b blocks with a reduced ("stale") learning rate on previously distilled blocks.

Evidence

Standard benchmarks (Table 1, narrow ConvNet): CIFAR-10 IPC 1/10/50 = 53.2/69.4/75.3%; CIFAR-100 IPC 1/10/50 = 35.3/47.5/50.6%; CUB-200 IPC 1/10 = 13.8/17.7%; Tiny-ImageNet IPC 1/10 = 20.1/24.4% -- state of the art on CIFAR-10, CIFAR-100 and CUB-200 at IPC 10/50, and statistically matching rcig-2023 (a kernel-closed-form method) across IPC 1/10/50 overall, without any inner-loop approximation. Combined with linear-basis parameterization (Deng and Russakovsky 2022): CIFAR-10 IPC1/10 improves from 53.2/69.4% to 68.2/72.8% (+1.6 points over LinBa alone). Wide-vs-narrow-network transfer (Sec. 4.1): RaT-BPTT trained on a narrow network transfers to wide-network evaluation nearly losslessly (e.g. CIFAR-100 IPC1: 35.3% narrow vs. 36.5% wide-transfer), whereas kernel-based methods evaluated on the narrow networks they were not tuned for drop sharply (RCIG CIFAR-100 IPC1: 39.3% wide-evaluated vs. 35.5% when re-evaluated narrow; FRePo Tiny-ImageNet IPC10: 25.4% wide vs. 22.4% narrow) -- the paper's central claim that RaT-BPTT "generalizes gracefully" in both directions while surrogate methods are architecture-width-dependent. Ablation on the truncation window (Sec 4.3, Fig. 6): random-uniform window placement beats both a fixed T-BPTT window and forward/backward-moving windows throughout training, isolating randomized window position (not just truncation) as the mechanism; a hardness-weighted (non-uniform) sampling can add a further 0.4 points on CIFAR-10 IPC10 but is dropped to keep the method simple. Intercorrelation finding: subsampling 5 images/class from a jointly distilled IPC50 set performs worse than 5 random real images per class for both KIP- and RaT-BPTT-distilled data, though RaT-BPTT degrades less than KIP; Boost-DD's strongly-boosted variant nearly matches jointly-distilled accuracy while yielding genuinely nested, budget-flexible subsets. No GPU-hour/wall-clock headline number is given, but the paper states RaT-BPTT's memory use "often exceed[s] that of directly training the model" because it still must store all intermediate parameters across the unrolled window.

Limitations

States explicitly (Appendix A) that GPU memory consumption is substantial -- often exceeding direct training -- because RaT-BPTT still unrolls and backpropagates over multiple steps, and that larger models would need activation checkpointing to manage this, unlike the kernel-closed-form family's closed-form (unrolling-free) inner solve. Design choices (window size, unrolling length, uniform vs. hardness-weighted window sampling) are guided by empirical tuning rather than theory. A hardness-score analysis (Appendix D) shows that increasing IPC does not proportionally distill harder ("forgettable") examples -- easy, unforgettable examples are captured well at all IPC, but hard examples remain under-distilled regardless of budget, identifying a specific bottleneck to further scaling. The boosting fix for intercorrelation is only partial: boosted blocks still show inter-block correlation (an early block's subsampled accuracy differs from a later block's), and only the (accuracy-costly) strongly-boosted variant fully removes cross-budget degradation. Evaluated only on image-classification benchmarks up to Tiny-ImageNet scale; no ImageNet-1K result and no soft-label/teacher-relabeling scheme.