Q2 · How is the synthetic set obtained?

Method families

A family is a group of methods that share one bet about what makes a small dataset train like a large one. Families are grouped into five paradigms, ordered by how much of the learner sits inside the synthesis loop, plus design choices that attach to any paradigm.Why this organization →

Selection

Which real samples should be kept? The distilled set is a subset of the real data, chosen by a scoring or covering rule.

Coreset selection

Keep a subset of real samples chosen by a scoring or covering rule; no pixels are synthesized.

since 2009 · mature

Dataset quantization

Partition the dataset into non-overlapping bins by a diversity criterion and sample from every bin, optionally dropping uninformative patches.

since 2023 · active

Bi-level performance matching

Does a model trained on the synthetic set perform like one trained on the real set? Optimize the synthetic set through the training of a model on it, by unrolling or in closed form.

Meta-learning through unrolled training

Treat the synthetic data as hyperparameters of training and optimize them by backpropagating through the unrolled inner optimization.

since 2018 · foundational

Kernel and closed-form inner solvers

Replace the inner training loop with kernel ridge regression (NTK, random features, or a trained feature extractor), so the inner problem has a closed-form solution.

since 2021 · mature

Surrogate matching

Does the synthetic set produce the same gradients, trajectories, or features? Replace the bi-level objective with a single-level proxy computed through networks in the loop.

Gradient matching

Make the gradients a network computes on synthetic data match the gradients it computes on real data, step by step along a training run.

since 2021 · mature

Trajectory matching

Train a student for several steps on synthetic data from a point on a precomputed expert trajectory, and make it land where the expert landed after many more real-data steps.

since 2022 · mature

Distribution and feature matching

Make the synthetic set's feature distribution match the real data's, in the embedding spaces of many (often randomly initialized) networks — with no bi-level loop at all.

since 2021 · active

Decoupled synthesis

Can a frozen teacher alone tell us what the synthetic set should contain? Train a teacher once, synthesize or select from it without any bi-level coupling, then train students on its soft labels.

Decoupled teacher-driven synthesis

Train one teacher on the real data, synthesize or select images from the teacher alone, then train students on the teacher's soft labels.

since 2023 · ascendant

Generative priors

Can a pretrained generator supply the space, the artifact, or the sampler? Use a pretrained generative model to parameterize, store, or directly sample the synthetic set.

Optimization in a generative latent space

Keep a pretrained generator frozen and optimize its latent codes, instead of pixels, with a matching objective.

since 2022 · active

The generator as the distilled artifact

Distill the dataset into the weights of a (small) generative model, and sample training data from it on demand.

since 2023 · active

Diffusion-based synthesis

Generate the synthetic set with a pretrained (optionally fine-tuned) diffusion model, steering sampling toward samples that are both representative of the dataset and diverse.

since 2024 · ascendant

Autoregressive and flow-based synthesis

The generative-prior idea carried to visual autoregressive (next-scale) models and flow-matching generators.

since 2026 · ascendant

Orthogonal design choices

What is stored, what is the label, and how is it optimized? Choices that attach to a method in any paradigm — parameterization, label handling, optimization recipes.

Synthetic-set parameterization

Change what is stored — codes and decoders, bases and hallucinators, frequency coefficients, sparse codes, neural fields — so that a fixed storage budget decodes into more or better training samples.

since 2022 · active

Label distillation and soft labels

Treat labels as part of what is distilled — learned labels, soft teacher labels, relabeling at every epoch — and account for what that costs.

since 2020 · contested

Optimization and training recipes

Improvements to how any condensation objective is optimized — initialization, model augmentation, sample weighting, curricula, multi-size and prunable sets.

since 2023 · active