The generator as the distilled artifact

DiM · arXiv 2023

Stores the dataset in a generator trained so that a pool of models gives matching logits on real and generated images; one generator then serves any distillation ratio and larger architectures.

Point Cloud DD — Point Cloud Dataset Distillation

Deyu Bo, Xinchao Wang · ICML 2025notableOther datapaper ↗

This paper adapts dataset distillation to unstructured 3D point clouds by theoretically showing that matching rotation-invariant features between real and synthetic data matters for 3D distillation, introducing a plug-and-play point cloud rotator that aligns point clouds to a canonical orientation, and replacing fixed-size synthetic data with a point-wise generator that produces point clouds at multiple resolutions instead of optimizing one fixed set directly.

GTN — Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data

Felipe Petroski Such, Aditya Rawal, Joel Lehman et al. · ICML 2020notablepaper ↗code ↗

Generative Teaching Networks train a generator network to produce synthetic training data (and, in principle, training environments or curricula) on which a freshly initialized learner takes a few SGD steps before being evaluated on the target task, with the generator's parameters updated by differentiating through that whole short training process via meta-gradients. Used to speed up architecture evaluation in neural architecture search, GTN-NAS is reported to improve on the NAS state of the art for a given search proposal mechanism while using orders of magnitude less computation than typical NAS methods.

Generative Dataset Distillation Based on Self-knowledge Distillation

Longzhen Li, Guang Li, Ren Togo et al. · ICASSP 2025paper ↗

Improves generative dataset distillation (a follow-up to the same authors' global/local-balancing GAN method) by integrating self-knowledge distillation for more precise logit-based distribution matching between synthetic and original data, adding a standardization step on logits before matching to keep their range consistent; reports gains over existing generative distillation baselines.

Dataset-Distillation Generative Model for Speech Emotion Recognition

Fabian Ritter-Gutierrez, Kuan-Po Huang, Jeremy H. M Wong et al. · Interspeech 2024Other datapaper ↗

Introduces the first dataset-distillation approach for speech, applied to Speech Emotion Recognition on IEMOCAP: a GAN is trained not to reconstruct real utterances but to distill discriminative information useful for downstream training, then replaces the dataset as a sampler of any desired synthetic size; reports comparable performance under the original class imbalance and a 0.3-point absolute UAR gain under balanced classes, with about 95% less storage and faster downstream training.

Generative Dataset Distillation: Balancing Global Structure and Local Details

Longzhen Li, Guang Li, Ren Togo et al. · CVPR 2024 Workshoppaper ↗

Trains a conditional GAN to serve as the distilled artifact while explicitly balancing global structural attributes against local details such as texture and shape, which prior generative distillation methods weighted toward high-level semantics alone; continuously optimizes the generator toward a more information-dense synthetic dataset.

DiLM — DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation

Aru Maekawa, Satoshi Kosugi, Kotaro Funakoshi et al. · NAACL 2024Textpaper ↗code ↗

Proposes DiLM, which trains a language model to generate readable synthetic text examples for dataset distillation instead of optimizing word-embedding sequences directly, so the distilled data is not tied to one model's embedding table; reports it outperforms coreset-selection baselines on text classification and generalizes to training different model types and to in-context learning with large language models.

MedSynth — MedSynth: Leveraging Generative Model for Healthcare Data Sharing

Renuga Kanagavelu · MICCAI 2024paper ↗

Condenses the knowledge in large medical datasets into a small generative model, combining an attention-based generator with a vision transformer so the model can produce a compact set of representative synthetic medical images; sharing the generator across hospitals avoids disclosing raw patient data. Reports outperforming state-of-the-art comparison methods and successfully defending against state-of-the-art membership-inference attacks.

D2M — Data-to-Model Distillation: Data-Efficient Learning Framework

Ahmad Sajedi, Samir Khaki, Lucy Z. Liu et al. · ECCV 2024paper ↗code ↗

Transfers knowledge into the parameters of a pretrained generative model by matching real and synthetic representations, rather than storing pixels, so the same trained generator serves any distillation ratio without retraining; scales to ImageNet-1K at 128x128, validated on 15 datasets with an application to neural architecture search.

Dataset Condensation via Generative Model

David Junhao Zhang, Heng Wang, Chuhui Xue et al. · arXiv 2023paper ↗

Condenses a dataset into a trained generative model rather than pixels, so storage stays roughly constant as classes or resolution grow, with intra-class and inter-class losses that push same-class samples apart and separate class centers to increase diversity and discriminability; reports the first successful ImageNet-1K condensation using a generative-model artifact.

MGDD — MGDD: A Meta Generator for Fast Dataset Distillation

Songhua Liu, Xinchao Wang · NeurIPS 2023paper ↗

Produces synthetic images from a generator network conditioned on a dataset-distillation initialization, with synthetic labels solved in closed form via least squares in feature space, and meta-trains the generator across many datasets so that adapting to a new target needs only a few steps; reports parity with state-of-the-art distillation baselines at 22x less computation and strong generalization to synthetic-set sizes unseen during adaptation.