Analysis & theory · Label distillation and soft labels
A label is worth a thousand images
A Label is Worth a Thousand Images in Dataset Distillation
Tian Qin, Zhiwei Deng, David Alvarez-Melis
NeurIPS 2024 · first public 2024-06-15 · arXiv 2406.10485
In one paragraph
Ablates the many disparate dataset-distillation methods and finds that the single largest common factor in their reported performance is the use of soft (probabilistic) labels rather than their specific synthesis techniques; shows the soft labels must carry structured information to help, provides empirical scaling laws relating soft-label effectiveness to images-per-class, and establishes an empirical Pareto frontier for data-efficient learning, challenging the attribution of gains to image-synthesis method alone.
Explained
What came before
By 2024 nearly every method that scaled to ImageNet-1K relied on soft labels from a pretrained teacher without studying why: SRe2L (2023) decouples synthesis from labeling and assigns 300 teacher-generated soft labels per synthetic image (one per augmented crop); Ra-BPTT and FRePo learn soft labels jointly with images but only scale to IPC 1-2 on downsized ImageNet. Method papers reported gains and attributed them to their image-synthesis technique.
The problem
Dataset-distillation methods are structurally very different (bi-level meta-learning, trajectory matching, distribution matching) yet nearly all winning ImageNet-scale methods share one ingredient — soft labels — that had not been isolated as a cause. It was unknown how much of the reported accuracy comes from the synthetic images versus from the labels.
The idea
Freeze the images (use randomly sampled real images, not synthesized ones) and vary only the labels; if a "soft label baseline" built from random real images plus teacher soft labels matches state-of-the-art distilled-image methods, the images are not doing the work the field credits them with. Soft labels help only insofar as they carry structured, semantically meaningful information (e.g. inter-class similarity), not because they are merely non-one-hot.
How it works
Optimizes nothing new: the "soft label baseline" randomly samples real images per class and labels them with the softmax output of a pretrained expert (checkpoint chosen per IPC budget by early-stopping the expert), replicating SRe2L's per-crop CutMix-augmented relabeling scheme. In the loop: a pretrained classifier (ResNet-18/50 on ImageNet, ConvNet on smaller sets) used only as a label generator, no bi-level optimization. Separately, labels are learned directly via MTT's and truncated-BPTT's objectives with images frozen, to test whether distillation objectives recover the same information as expert soft labels. Structured information is isolated with an "i-th label swapping test" (replacing the i-th largest softmax entry with the smallest) and top-k truncation of the softmax vector.
Evidence
ImageNet-1K, hard labels vs random-image + soft-label baseline vs SRe2L, ResNet-18/50 eval (Table 1): hard-label random images reach 0.6% (IPC1)/20.4% (IPC50); the soft-label baseline with ResNet18-generated labels reaches 6.6%/47.9% and with CutMix-augmented soft labels 2.9%/46.8%, matching or beating SRe2L's reported 2.9%/46.8%-level numbers at equal IPC while storing far less (SRe2L's 300 labels/image cost "at least 300x" the soft-label baseline's single label per image). On TinyImageNet/CIFAR-100/CIFAR-10 (Table 2) the soft-label baseline trails Ra-BPTT/MTT/DM at IPC1 but matches or beats them by IPC50-100. Data-knowledge scaling law (Fig. 6): recovering full teacher knowledge needs ~10 IPC with full label information (k=200) but ~40 IPC when the soft label is truncated to the top-32 entries, giving a fitted power law relating soft-label information content to an effective 6x reduction in required data. Zero-shot ablation (Fig. 7): removing all images of a class barely hurts accuracy on that class if its soft-label information is kept elsewhere, but zeroing out a class's entries in the labels (keeping the images) collapses accuracy on that class to 0% at IPC1 — labels carry more of the "content" than images. Label swapping test (Fig. 4) shows swapping top-ranked softmax entries hurts far more than swapping low-ranked ones, confirming the useful information concentrates in a few structured entries. BPTT-learned labels (images frozen) converge to the same labels as an early-stopped expert ensemble (Fig. 8, minimal Jensen-Shannon distance at a consistent epoch), while MTT fails to learn useful labels this way.
Limitations
The paper explicitly restricts to fixing images and only varying labels, so it cannot rule out that jointly learning images and labels does better than either alone; it explicitly notes recent methods still beat the soft-label baseline at low IPC on smaller datasets and concludes both images and labels matter. It only tests image classification (no other modality). The soft-label baseline itself still requires a pretrained expert and per-crop relabeling, so it does not resolve the label-storage cost it implicitly documents (300x for SRe2L-style regenerated labels). No wall-clock/GPU-hour cost comparison is reported.
Written by the atlas from the paper's full text. Check the paper for exact numbers.
Where it sits
- Label distillation and soft labels (Orthogonal design choices)
- Setting: Image classification
Design choices
| Labels | soft-static |
| Optimization regime | decoupled |
| Largest scale evaluated | imagenet-1k, tiny-imagenet |
Abstract (verbatim from arXiv)
Data $\textit{quality}$ is a crucial factor in the performance of machine learning models, a principle that dataset distillation methods exploit by compressing training datasets into much smaller counterparts that maintain similar downstream performance. Understanding how and why data distillation methods work is vital not only for improving these methods but also for revealing fundamental characteristics of "good" training data. However, a major challenge in achieving this goal is the observation that distillation approaches, which rely on sophisticated but mostly disparate methods to generate synthetic data, have little in common with each other. In this work, we highlight a largely overlooked aspect common to most of these methods: the use of soft (probabilistic) labels. Through a series of ablation experiments, we study the role of soft labels in depth. Our results reveal that the main factor explaining the performance of state-of-the-art distillation methods is not the specific techniques used to generate synthetic data but rather the use of soft labels. Furthermore, we demonstrate that not all soft labels are created equal; they must contain $\textit{structured information}$ to be beneficial. We also provide empirical scaling laws that characterize the effectiveness of soft labels as a function of images-per-class in the distilled dataset and establish an empirical Pareto frontier for data-efficient learning. Combined, our findings challenge conventional wisdom in dataset distillation, underscore the importance of soft labels in learning, and suggest new directions for improving distillation methods. Code for all experiments is available at https://github.com/sunnytqin/no-distillation.
BibTeX (generated; prefer the venue's official entry)
@article{qin2024label,
title = {A Label is Worth a Thousand Images in Dataset Distillation},
author = {Tian Qin and Zhiwei Deng and David Alvarez-Melis},
journal = {NeurIPS 2024},
year = {2024}
}Nearby in Label distillation and soft labels
Xiao Cui, Yulei Qin, Wengang Zhou et al. · NeurIPS 2025notablepaper ↗