Rethinking Dataset Distillation: Hard Truths about Soft Labels

What came before

label-worth-2024 (Qin et al.) showed that random real images with teacher soft labels match SRe2L's reported ImageNet-1K accuracy, undercutting the claim that synthesized images are doing the work. Meanwhile the coreset literature (EL2N/GraNd, Dyn-Unc, forgetting, k-center, GraphCut, Cal) consistently shows high-quality subsets beat random under hard labels, and SelMatch's sliding-window search finds the IPC-optimal difficulty band at real training cost. Small-scale trajectory-matching methods (TM, DATM) report large gains over coresets on TinyImageNet, and DATM further jointly learns soft labels with images. No prior work had asked whether the "coresets beat random" finding and the "random beats distillation" finding could both be true at once, under different label regimes.

The problem

Coreset selection and dataset distillation are almost never evaluated under the same label regime (coresets: hard labels; large-scale DD: cutmix-augmented per-crop teacher relabeling every epoch, termed SL+KD; some small-scale DD: one frozen soft label per image, termed SL), so it is unknown whether data quality itself stops mattering as label supervision grows, or whether DD methods are simply worse at building coresets than the coreset literature's own methods.

The idea

Split label supervision into three regimes — hard labels (HL), one frozen soft label per image (SL), and per-crop teacher relabeling every epoch (SL+KD) — and separately scale subset quality, subset size (IPC) and training compute within each. Data quality drives performance in HL exactly as the coreset literature reports; it plays only a minor role in SL beyond a small minimum IPC, because per-sample training signal homogenizes; and in SL+KD it stops mattering almost entirely, with performance instead set by compute and saturating near full-dataset accuracy regardless of which subset (of any size or quality) is used to start.

How it works

Constructs subsets by retaining a fraction $f\in[20\%,100\%]$ of ImageNet-1K by EL2N difficulty score at IPC 10-700 (class-balance capped at 700), and trains ResNet-18 students under each of HL, SL and SL+KD at four compute budgets (2-50 full-dataset-equivalent epochs), comparing to a full-dataset-trained reference at matched compute. Derives EL2N-SL, a soft-label generalization of the EL2N/GraNd score ($\mathrm{EL2N\text{-}SL}(x)=\frac1T\,\mathbb{E}\|p(w_t,x)-q(w_t,x)\|_2$ against the teacher's soft target $q$) to check whether per-sample difficulty still varies under frozen soft labels. Introduces Distillation Correlation Score (DCS): given $m$ precomputed subsets of varying quality/IPC, DCS is the Spearman correlation between each subset's downstream test loss (trained and evaluated once, reused as a lookup table) and its distillation-objective loss (e.g. TM's trajectory-matching loss), letting a synthesis objective's scalability be checked without running synthesis or student training. Introduces CAD-Prune: trains one model on the full dataset under the same compute budget/LR schedule as the downstream task, then scores each sample by the standard deviation of its EL2N score over a small window of $J{=}6$ epochs positioned $W{=}2$ epochs from the end of that compute-matched run (capturing samples still being learned at that budget, not early-training difficulty), and selects the difficulty band this implies. CA2D applies CAD-Prune's selected samples to RDED's crop/select/stitch construction instead of RDED's own confidence-only patch heuristic. In the loop: a pretrained ResNet-18 teacher (for soft labels and EL2N/CAD-Prune scores); no bi-level optimization for the paper's own analysis, though it re-runs TM/DATM/DC/DM/SRe2L/DWA/D4M/Minimax Diffusion/RDED as objects of study.

Evidence

ImageNet-1K, ResNet-18, IPC 10/50/100, same subsets scored under all three label regimes (Table 1) — the controlled comparison the atlas tracks: in HL, DD methods vary from far below random real images (SRe2L 9.79%/14.89% vs. random real 28.52%/40.89% at IPC50/100) to above it (RDED 14.34/38.49/44.36 vs. random real 5.10/28.52/40.89 at IPC10/50/100), and the best coreset (EL2N- Best, IPC-dependent difficulty window) reaches 47.18% at IPC100 — beating RDED's 44.36%; full dataset at matched compute is 65.91%. In SL (one frozen soft label per image, same images), all methods compress toward each other and toward random real (SRe2L 37.33 vs. random real 55.65 at IPC100) but a visible method gap remains. In SL+KD (per-crop relabel+KD, same images again), every method and every coreset lands within a narrow 52-59% band at IPC50 regardless of method identity or IPC (e.g. SRe2L 52.58, D4M 58.08, random real 58.10, EL2N-Best 59.16), against a full-dataset reference of 62.50% at matched compute — i.e. label regime alone, not subset choice, explains most of the score once KD is added. TinyImageNet/ConvNet-D4 (Table 2): DATM beats k-center by about 8 points under HL (29.51% vs. 22.02% at IPC50) but the gap vanishes under SL (37.39% vs. 37.45%). CAD-Prune vs. the sliding-window coreset baseline and CA2D vs. RDED, ImageNet-1K HL (Table 3): CAD-Prune reaches 40.21%/47.40% (IPC50/100) vs. sliding-window EL2N-Best's 38.44%/47.76% at 2-4x less compute (no repeated full training runs per candidate window); CA2D reaches 41.72%/46.32% vs. RDED's 38.49%/44.36%.

Limitations

Restricted to image classification (stated as future work: multimodal, tasks beyond classification). TM/DATM's DCS-based conclusion about ResNet-18 non-scalability is inferred from surrogate-loss correlation, not from actually training a competitive TM/DATM ImageNet-1K set (TM/ DATM synthesis itself never scales past TinyImageNet/ConvNet-D4 in the paper). CAD-Prune still requires one full compute-matched training run on the whole dataset to obtain its uncertainty scores, so its efficiency gain is relative to the sliding-window search, not free. The paper states that "expressing non-optimizable DD objectives for evaluation under DCS requires care and is an open problem." No wall-clock/GPU-hour costs are reported anywhere in the paper.