Where the field is going
Open problems
Each problem is stated as a claim, backed by the papers that show it is a problem and linked to the families that carry it as a weakness. "Partially addressed" means there is published progress that does not yet close the gap.
open · Images
Distilled data works inside the standard evaluation protocol, but does not substitute for real data outside it.
If a distilled set mainly encodes early training dynamics of one training recipe, it will not serve the uses condensation is promoted for (continual learning, architecture search, pre-training) whenever those change the recipe.
Evidence
- What is DD learning?: Distilled data cannot substitute for real data outside the standard evaluation setting, and it retains performance by compressing information about early training dynamics.
open · Graphs
Graph condensation needs full-graph training to produce a small graph, and its standard metrics hide the true cost.
The dominant gradient- and trajectory-matching graph methods train on the full graph to build the condensed one, so the efficiency they report excludes the most expensive step. Node compression ratios ignore storage of features and structure and the cost of condensation itself, and benchmark protocols vary. This mirrors the evaluation reckoning in image condensation.
Evidence
- Graph condensation needs a reset: Argues gradient-matching condensation requires full-dataset training, generalizes poorly across GNN architectures, and relies on misleading metrics such as node compression ratio.
- GC4NC: A 12-method benchmark that documents inconsistent evaluation practices across graph condensation papers.
- GCond: The founding method imitates GNN training on the original graph via gradient matching, the design the position paper critiques.
open · Vision–language, Audio–visual & omni
Multimodal condensation has not scaled beyond about a thousand image–text pairs, and recall stays far below full-data training.
Image classification reached ImageNet-1K and 21K once synthesis was decoupled from the learner. Image–text condensation is still evaluated on Flickr30K and COCO at 100–1,000 pairs, where retrieval recall remains a fraction of what full-data training achieves, and the one attempt at a larger corpus sits near the random floor. The setting has not had its scale-breaking moment.
Evidence
- LoRS: Text→image R@1 of 10.0 on Flickr30K at 500 pairs, under a protocol whose full-data retrieval is several times higher.
- MTT-VL: Reports a full-data NFNet+BERT ceiling of image→text R@1 33.9 and text→image R@1 27.3 on Flickr30K, against 13.3 and 7.9 for its distilled set at 1,000 pairs.
- EDGE: On CC3M at 1,000 pairs its retrieval R@1 is 0.1–0.2, near the random floor.
- HoPA: The omnimodal extension is demonstrated for exactly three modalities; scaling with modality count is argued, not shown.
open · Vision–language
Vision–language condensation results are reported under incompatible evaluation protocols.
Image–text methods differ in which encoders are trained during distillation and evaluation, and in which text encoder is used at all. At least three recipes are in use (frozen BERT, trainable BERT, CLIP text tower), and re-runs of the same baseline at the same budget disagree between papers. Without a shared, versioned benchmark, state-of-the-art claims cannot accumulate across papers as they do for image classification.
Evidence
- CovMatch: Makes the BERT text encoder trainable; its ablation shows freezing BERT alone drops Flickr30K mean recall at 500 pairs from 38.4 to 29.4, so the protocol choice is worth more than many method differences.
- PDS: Evaluates with a frozen CLIP text encoder and non-CLIP image backbones, re-implementing LoRS and TESLA-VL with a CLIP text tower; its numbers cannot be pooled with BERT-based tables.
- PTM-ST: Its re-run of LoRS at 500 pairs (text→image R@1 12.7) differs from LoRS's published 10.0, which RepBlend's re-run reproduces exactly; no paper documents the cause.
- EDGE: Evaluates with a pretrained NFNet and a frozen BERT-base, the protocol shared with LoRS and RepBlend but not with CovMatch or RAHA.
open · Images
Soft labels transfer teacher knowledge that is not in the distilled data at all.
If a student can recover what a teacher memorized about held-out examples from soft labels alone, then a teacher-relabeled distilled set is partly a compressed teacher. That changes both what the accuracy means and what privacy a distilled dataset can offer.
Evidence
- Soft labels leak: Students trained on teacher soft labels reach non-trivial accuracy on held-out memorized data they never observed, sometimes perfect accuracy.
- A label is worth a thousand images: Soft labels must contain structured information to be beneficial.
contested · Images
At large scale, it is not established that synthesized sets beat well-chosen real subsets.
The case for synthesis is that a subset of real images cannot express as much as optimized data. Under hard labels and standardized protocols on ImageNet-scale benchmarks, recent re-evaluations find coresets comparable or better at far lower construction cost.
Evidence
- Do distilled sets beat coresets?: State-of-the-art distilled sets are comparable to or worse than coresets on ImageNet-1K, ImageNet-100 and ImageNette under standardized protocols, and cost substantially more.
- Hard truths about soft labels: With hard labels, only RDED reliably beats random selection on ImageNet-1K among the methods tested, and it can still trail strong coresets.
Attacked by Hard truths about soft labels.
contested · Images
Whether condensed data offers any privacy protection is disputed.
Condensation is often promoted as a way to share data safely, because synthetic images are not real records. The central claim of a privacy benefit and its rebuttal were published the same year and point in opposite directions, and later benchmarking finds leakage reduced but not removed. Without formal guarantees, a distilled dataset should not be treated as private.
Evidence
- Privacy for Free: Argues condensation acts as a private-data generator for free, with a bound on any one raw sample's influence for linear feature extractors, supported by membership-inference experiments.
- No Free Lunch: Argues that evaluation and analysis are flawed and do not show a statistically significant privacy benefit; DP-SGD gives both better accuracy and provably lower membership-attack success.
- A Comprehensive Study on Dataset Distillation: Performance, Privacy, Robustness and Fairness: Reports that distillation reduces but does not eliminate privacy leakage under membership inference.
- Soft labels leak: Teacher soft labels can transfer memorized information about held-out examples to a student.
partially addressed · Images
Reported accuracy does not isolate the quality of the synthetic data.
Modern evaluation pipelines add teacher soft labels, strong augmentation and long schedules. Ablations and re-evaluations show that these additions explain much of the reported accuracy and of the differences between methods, so a better number does not reliably mean a better condensed dataset.
Evidence
- A label is worth a thousand images: Ablations indicate soft labels, not the image-generation technique, are the main factor behind state-of-the-art performance.
- DD-Ranking: Improvements often stem from additional evaluation techniques; even randomly sampled images can do better under them.
- RD3: Much of the performance variation among decoupled methods is attributable to inconsistent post-evaluation protocols.
- Hard truths about soft labels: With abundant soft labels and teacher distillation, performance saturates regardless of subset size or quality.
Attacked by DC-BENCH, DD-Ranking, RD3.
partially addressed · Images, Graphs
Condensation tends to preserve the majority and amplify unfairness toward minority groups.
Matching objectives align the synthetic set with the whole distribution, which the majority dominates. With only a handful of samples per class, minority subgroups can disappear from the distilled data, and models trained on it inherit the gap, for images and graphs alike.
Evidence
- FairDD: Distillation does not fix and typically worsens unfairness toward protected-attribute minority groups, because matching aligns to majority-dominated distributions.
- Fair Graph Distillation: GNNs trained on condensed graphs can show more severe group-fairness problems than GNNs trained on the original graph.
- COBRA: Fairness gaps come from mismatched predictive patterns between subgroups, not only group-size imbalance, and persist after correcting for imbalance.
- A Comprehensive Study on Dataset Distillation: Performance, Privacy, Robustness and Fairness: Reports that distillation can amplify unfairness across classes.
Attacked by FairDD, Fair Graph Distillation, COBRA.
partially addressed · Images, Vision–language
The budget counts images, but not the labels and auxiliary structures that make them work.
Images-per-class ignores soft labels, decoders, bases and generators. At ImageNet scale the stored soft labels can exceed the images many times over, so two methods with the same IPC can differ by orders of magnitude in what they actually store.
Evidence