Why not an existing organization?
Seven surveys and the community index were read in full before this taxonomy was fixed
(notes in the repository under docs/recon/surveys.md). They organize the field along different
axes, and they disagree on basic placements.
| Survey | Organizing axis | Covers decoupled, soft-label, generative, multimodal work? |
|---|---|---|
| Yu, Liu & Wang, TPAMI 2023 | Optimization objective: performance, parameter, distribution matching | No |
| Lei & Tao, TPAMI | Meta-learning versus data-matching frameworks; factorized variants marked across the tree | No |
| Sachdeva & McAuley, TMLR 2023 | Approach family, then data modality | No |
| Geng et al., IJCAI 2023 | Learning frameworks plus common enhancements; rich applications | No |
| Liu & Du, 2025 | Matching, scalable/generative, decoupled | Yes: the only one with first-class sections for all four |
| Gao et al., TKDE 2025 | Graphs only; by evaluation criterion | Graphs only |
| Hashemi et al., IJCAI 2024 | Graphs only; condensation as a sibling of sparsification and coarsening | Graphs only |
Where they disagree.
- Are kernel methods meta-learning or matching? Lei & Tao and Geng et al. put kernel ridge regression under meta-learning, Yu et al. under performance matching, and Sachdeva & McAuley merge unrolled and kernel methods into one “meta-model matching” family. The atlas groups them as P1 bi-level performance matching, with two families: unrolled and closed-form. What the two share is that the inner training problem is solved, not approximated by a surrogate network.
- Is parameterization a family or an axis? Yu et al. and Geng et al. treat it as a cross-cutting enhancement; Sachdeva & McAuley and Lei & Tao give factorization a top-level section. The atlas treats what is stored as an orthogonal design choice (Q3), because it is combined with gradient, trajectory and distribution matching alike.
- Where does SRe2L go? None of the 2023 surveys covers it; Liu & Du make decoupling its own top-level family. The atlas agrees and names the paradigm by its structural property: a frozen teacher, and no student in the synthesis loop.
The Awesome list’s headings, mapped
The Awesome Dataset Distillation list is the discovery source for this atlas, and its headings are a good index. They are not a classification, because they mix kinds of thing:
| Awesome heading | Kind of thing | Where it goes in the atlas |
|---|---|---|
| Gradient/Trajectory Matching, Distribution/Feature Matching, Kernel-Based | Mechanism | Q2 families in P1 and P2 |
| Decoupled Distillation | Mechanism | Q2: P3 decoupled synthesis |
| Generative Distillation (GAN / Diffusion / VAR / Flow) | Mechanism | Q2: P4 families |
| Distilled Dataset Parametrization | What is stored | Q3 parameterization |
| Label Distillation | What the label is | Q3 label distillation |
| Better Optimization | How it is optimized | Q3 optimization recipes, or the family it improves |
| Dataset Quantization | Mechanism (selection) | Q2: P0 |
| Multimodal, Video, Graph, Text, Time Series, … | What data | Q1 settings |
| Better Understanding, Benchmark, Survey | Contribution type | Q4 lens |
| Privacy, Robustness, Fairness, Continual, Federated, Medical, … | Contribution type / use | Q4 lens, plus the mechanism family used |
Two meanings of “multimodal”
The word is used for two different things, and related-work sections often mix them.
- The distilled dataset is multimodal. Image–caption pairs are condensed so that a vision–language model can be trained on them. This is the vision–language setting.
- A multimodal model is a tool. A text-to-image diffusion model, CLIP text prototypes or a vision–language model is used to condense an ordinary image classification dataset. These papers are in the image-classification setting, with a vision–language prior recorded as a design facet.
Why the paradigms are ordered by coupling
The ladder P1 → P4 removes the learner from the synthesis loop one piece at a time. P1 differentiates through training itself. P2 keeps one network in the loop but replaces training with a proxy (gradients, trajectory segments, feature distributions). P3 keeps only a frozen teacher. P4 keeps only a frozen generator and often no classifier at all. Cost fell and achievable scale rose in the same order, which is why the order also reads as a history of the field. It also points to where the field’s current weakness comes from: the less the learner is involved in synthesis, the more the evaluation-time teacher has to supply. See Evaluation.