The standard protocol, and what it was meant to measure
In the class-conditional image setting, a distilled set is evaluated by training a freshly initialized network on alone and reporting top-1 accuracy on the real test set, averaged over several runs. The budget is counted in images per class (IPC). Small benchmarks (MNIST, CIFAR-10/100, Tiny-ImageNet) conventionally use a 3–4 layer ConvNet with the differentiable augmentation of DSA. ImageNet-1K uses ResNet-18/50.
The protocol is meant to measure the information in the synthetic data. Since 2023 its evaluation phase has absorbed more and more machinery that also carries information: a pretrained teacher that relabels every augmented crop in every epoch, strong augmentation such as CutMix, long schedules, and tuned batch sizes and learning-rate curves. Everything on this page comes from papers read in full.
Three label regimes, one set of images
Separate three regimes when reading any number:
- Hard labels: one-hot class labels.
- Fixed soft labels: one stored probability vector per image.
- Relabel with KD: a teacher labels every augmented crop at every epoch, so the student learns from the teacher, not only from the data.
Hard truths about soft labels evaluates the same subsets under all three. On ImageNet-1K at 50 IPC with ResNet-18 (Table 1):
| Images | Hard labels | Fixed soft labels | Relabel with KD |
|---|---|---|---|
| SRe2L synthetic images | 9.8% | 27.6% | 52.6% |
| Random real images | 28.5% | 48.5% | 58.1% |
| Full training set (reference) | — | — | 62.5% |
Under hard labels the synthetic images trail random real images by about 19 points. Under relabel-with-KD the gap shrinks to about 5.5 points, and every method and coreset tested falls inside a 52–59% band. Once the teacher supervises every crop, the quality of the images barely matters.
The same effect appears inside individual papers. NRR-DD re-evaluates RDED’s own images: 42.0% with soft labels, 16.3% with one-hot labels (ImageNet-1K, 10 IPC, ResNet-18). The images are identical; the 25.7-point difference is the labels. CIM removes relabeling for several methods at the same setting: SRe2L falls to 1.1% and G-VBSM to 0.8%, while RDED keeps 19.7% and CIM 22.0%. Methods that stay close to real images survive the removal; methods that invert a teacher’s batch-norm statistics do not. A label is worth a thousand images had already shown that random real images given the same soft labels match or beat the synthesized images of state-of-the-art methods at higher IPC.
The recipe moves the same method by 20 points
The table below shows one nominal cell (SRe2L, ImageNet-1K, 10 IPC, ResNet-18, relabel with KD), as reported by different papers that re-ran it:
| Source | SRe2L accuracy |
|---|---|
| SRe2L, original paper | 21.3% |
| CIM, re-run | 31.1% |
| DELT, re-run with CDA’s configuration | 41.9% |
| CV-DD, tuned “SRe2L++” baseline | 43.1% |
That spread is larger than most improvements claimed in this family. RD³ makes it general: across eight decoupled and generative methods the published gap at 10 IPC is 27.3 points, and under one unified protocol (batch size and learning-rate schedule) it shrinks to 6.7 points. Every method’s own accuracy rises under the unified recipe, which means the original papers were under-tuned, not only incomparable. GIFT shows that the optimizer and loss used with soft labels can swing the same images and labels from 1.9% to 53.4% (RDED, 10 IPC, SGD without and with its loss). DC-BENCH found the same pattern in 2022, before soft labels: augmentation alone moved CIFAR-10 accuracy at 10 IPC by 6.6–10.2 points depending on the method.
Against random images and strong coresets
DD-Ranking scores each method against random real images trained with that method’s own recipe. On CIFAR-10, SRe2L’s improvement over random is negative at 1, 10 and 50 IPC, and RDED’s is slightly negative too. Hard-label trajectory matching scores best (+16.7 at 10 IPC). On ImageNet-1K with relabeling, current methods consistently underperform random selection given the same labels.
Do distilled sets outperform coresets? compares seven recent diffusion-based and decoupled methods against two diversity-aware coresets under three label protocols. On ImageNet-1K with hard labels and ResNet-18, the AUM coreset beats the best distilled set at 10 IPC (18.9% vs 15.6%) and at 50 IPC (43.5% vs 39.3%). With soft labels the gap narrows, but the coresets stay slightly ahead. The reversal comes on ImageNette, a small and easy dataset, where diffusion-based synthesis wins even under hard labels (60.0% vs 57.0% at 10 IPC). Coresets also cost a fixed 3–6 GPU-hours to build, while synthesis cost grows with IPC (12.8 to 51.7 hours for ManifoldGD from 10 to 50 IPC).
Where synthesis still clearly helps
The evidence is not that condensation fails. It is that its advantage is regime-specific:
- Small scale, hard labels. Trajectory matching beats k-center coresets on Tiny-ImageNet under hard labels (29.5% vs 22.0% for DATM at 50 IPC), though the gap disappears under fixed soft labels (Hard truths, Table 2).
- Small or easy datasets. Diffusion-based synthesis beats coresets on ImageNette under hard labels (Do distilled sets outperform coresets?).
- Methods anchored in real images. RDED is the one large-scale method in the “Hard truths” study that reliably beats random selection under hard labels, though it can trail the strongest coreset.
The label bill
IPC counts images, not what else is stored. LPLD reports soft labels exceeding the images by more than 30× in ImageNet condensation, and compresses them from 113 GB to 2.8 GB at 200 IPC. HeLlO lists RDED’s soft-label storage at 572 MB, 5.7 GB and 28.6 GB for 1, 10 and 50 IPC, and replaces it with a projector of about 0.8M parameters. That wins at 1 and 10 IPC but loses at 50 (52.2% vs 56.5%). NRR-DD stores two scalars per crop instead of a probability vector, recovering about 70% of the soft-vs-hard gap at roughly 500× less label storage. HALD inserts a storage-free hard-label phase between two soft-label phases, and at 285 MB it beats LPLD by 9 points.
Multimodal evaluation is split too
Image–text distillation reports image↔text Recall@K on Flickr30K and MS-COCO at a fixed number of pairs. Papers differ in which encoders are pretrained, which are trained during evaluation, and which text encoder is used (for example, frozen versus trainable BERT). Numbers from different recipes belong in separate columns; the vision–language setting page records the protocol of each method.
What the evidence recommends
- Report hard-label accuracy first, and fixed-soft and relabel-with-KD accuracy on the same images, so the labels’ contribution is visible.
- Include random real images and a diversity-aware coreset, both trained with exactly the same labels, augmentation, optimizer, batch size, schedule and epochs as the method. A hardest-examples-only coreset is not enough.
- Count everything stored (soft labels, decoders, generators, projectors) and report construction time, stating whether teacher or generator pre-training is excluded.
- Report accuracy on an architecture not used during synthesis, ideally a transformer as well as a CNN. Batch-norm inversion methods degrade sharply on ViTs (RD³).
- Treat cross-paper comparisons as provisional unless the baselines were re-run under one recipe.
The atlas records these facts on each paper where its source states them, and never ranks numbers produced under different answers to questions 1–3.