Labels were always part of the dataset
A distilled image with a one-hot label carries only what its pixels carry. Give it a probability vector over classes and it also carries how similar the classes are, which is information the pixels would otherwise have to encode. Learned labels made this explicit in 2020 by distilling labels for real images instead of distilling images. Kernel methods learned labels alongside their support sets, and TESLA introduced soft labels to help trajectory matching converge on ImageNet-1K’s 1,000 classes. The decisive step was SRe2L’s relabel stage. A pretrained teacher labels every augmented crop of the synthetic images in every epoch, and students learn from those labels. Since 2023 almost every ImageNet-scale headline number has used this regime.
How much of the accuracy is the label?
Separate three regimes: hard labels, fixed soft labels (one stored vector per image), and relabel with knowledge distillation (a teacher labels every crop, every epoch). Papers read in full give consistent answers.
- A label is worth a thousand images gives the same random real images either hard or teacher labels. On ImageNet-1K at 50 IPC with ResNet-18, accuracy goes from 20.4% to 47.9–54.3%, matching or beating SRe2L’s synthesized images. A label-swapping ablation shows the benefit sits in the few top-ranked entries of each vector, not in softness as such.
- Hard truths about soft labels evaluates the same subsets under all three regimes. On ImageNet-1K at 50 IPC, SRe2L images score 9.8% / 27.6% / 52.6% and random real images 28.5% / 48.5% / 58.1%. Under relabeling, every method and coreset tested lands between 52% and 59%.
- NRR-DD gives RDED’s own images 42.0% with soft labels and 16.3% with one-hot labels.
With a teacher supervising every crop, much of what reported accuracy measures is the teacher.
The label bill, and how to shrink it
Soft labels for every crop of every image take space that images-per-class does not count.
| Paper | Approach | Result |
|---|---|---|
| LPLD | Batch synthesis within classes so images are more diverse, then prune labels at random | ImageNet-1K labels at 200 IPC from 113 GB to 2.8 GB; 40×-pruned labels still beat unpruned SRe2L (59.6% vs 57.0%) |
| HeLlO | Replace stored labels with a CLIP-based image-to-label projector (~0.8M parameters) | Beats RDED at 1 and 10 IPC but loses at 50 (52.2% vs 56.5%); RDED’s labels are 28.6 GB at 50 IPC |
| INFER | One static label per instance on real images plus a learned compensator | Up to 99% less label storage |
| NRR-DD | Store two scalar distances per crop instead of a vector | About 70% of the soft-vs-hard gap recovered at roughly 500× less storage |
| HALD | Insert a storage-free, label-smoothed hard-label phase between soft phases | 42.7% vs LPLD’s 33.7% at 285 MB (ImageNet-1K, 50 IPC) |
The pattern is clear. Most of what the soft labels carry can be kept at a small fraction of their storage, but every method still depends on a teacher trained on the full dataset.
The training recipe around labels is itself a variable
GIFT shows that how soft labels are consumed matters as much as having them. With the same images and labels, changing only the optimizer and loss moves RDED at 10 IPC from 1.9% (SGD, standard loss) to 53.4% (SGD with GIFT’s cosine loss). A reported accuracy can therefore depend on an undisclosed optimizer–loss pairing more than on the data.
Risks
Soft labels can leak held-out teacher knowledge: in controlled settings, from toy models up to GPT-2, students trained on a memorizing teacher’s soft labels recover information about training examples they never saw. Higher softmax temperature increases the leakage. This has not been measured on a released condensation dataset. But teacher-relabeled “distilled datasets” should not be assumed to be private, or independent of the teacher.
Where the family stands
Labels are no longer a detail of evaluation; they are half of what is distilled. The evidence favors three practices: report hard-label accuracy alongside soft-label accuracy on the same images; count label storage in the budget; and compare against random real images given the same labels. See Evaluation and the open problems on uncounted label storage and teacher knowledge transfer.