Meta-learning through unrolled training

DD · arXiv 2018

Poses the task. Synthetic points need not come from the data distribution; they only have to reproduce, through a few gradient steps from a fixed initialization, the model trained on the full data. MNIST's 60,000 training images are compressed into 10 synthetic images.

RaT-BPTT · ICLR 2024

Returns to backpropagation through time, traces its difficulty to gradient variance, compute and long-range dependencies, and stabilizes it with randomly placed truncation windows. Also finds that subsets of a distilled set perform much worse than a set distilled directly at the smaller size.

Offline Behavior Distillation

Shiye Lei, Sen Zhang, Dacheng Tao · NeurIPS 2024notableOther datapaper ↗code ↗

This paper formulates offline behavior distillation (OBD), synthesizing a small set of expert-like behavioral data from suboptimal offline RL data so that a policy can be learned quickly, and shows that naive decision-difference objectives are hard to optimize through the intractable bi-level problem and give only quadratic-discount-complexity guarantees. It introduces action-value weighted PBC (Av-PBC), proving an equivalence between policy performance and the action-value weighted decision difference, and reports faster convergence, better distillation performance and robust cross-architecture/optimizer generalization than the naive objectives on multiple D4RL datasets.

GTN — Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data

Felipe Petroski Such, Aditya Rawal, Joel Lehman et al. · ICML 2020notablepaper ↗code ↗

Generative Teaching Networks train a generator network to produce synthetic training data (and, in principle, training environments or curricula) on which a freshly initialized learner takes a few SGD steps before being evaluated on the target task, with the generator's parameters updated by differentiating through that whole short training process via meta-gradients. Used to speed up architecture evaluation in neural architecture search, GTN-NAS is reported to improve on the NAS state of the art for a given search proposal mechanism while using orders of magnitude less computation than typical NAS methods.

InfMatch — Dataset Distillation by Influence Matching

Haoru Tan, Wang Wang, Sitong Wu et al. · CVPR 2026Vision–languagepaper ↗code ↗

Replaces process-level surrogates (per-step gradients or training trajectories) with a differentiable, sample-level estimator of each synthetic point's influence on the converged model parameters, computed in linear time by unrolling the optimization dynamics with a first-order Taylor approximation, then learns the synthetic set by matching this influence to that of the real data; reports 31.5% accuracy on Tiny-ImageNet at IPC-10 (a 4.7-point gain over NCFM) and extends to vision-language distillation on Flickr30K.

CD^2 — CD^2: Constrained Dataset Distillation for Few-Shot Class-Incremental Learning

Kexin Bao, Daichi Zhang, Hansong Zhang et al. · IJCAI 2025paper ↗

CD^2 targets few-shot class-incremental learning by pairing a classifier-guided dataset distillation module, which condenses each new session's few-shot samples into highly compacted exemplars, with a distillation constraint module that regularizes the previously learned class distribution so old sessions' distilled knowledge is not overwritten. The paper reports improvements over other class-incremental methods on three public benchmarks.

Beyond Random: Automatic Inner-loop Optimization in Dataset Distillation

Muquan Li, Hang Gou, Dongyang Zhang et al. · NeurIPS 2025paper ↗

Replaces the random truncation used in backpropagation-through-time dataset distillation with Automatic Truncated BPTT (AT-BPTT), which chooses truncation windows and sizes from stage-aware gradient statistics and a low-rank Hessian approximation, reporting a 6.16-point average accuracy gain over baselines on CIFAR-10/100, Tiny-ImageNet and ImageNet-1K along with a 3.9x speedup and 63% less memory.

Distilling Reinforcement Learning into Single-Batch Datasets

Connor Wilhelm, Dan Ventura · ECAI 2025Other datapaper ↗

Extends dataset distillation to reinforcement learning by distilling RL environments into a single batch of supervised-learning examples, using a novel extension of proximal policy optimization for meta-learning to fit the bi-level objective; demonstrates the approach on an extended cart-pole task, MuJoCo environments and Atari games, compressing complex RL environments into one-step supervised training and examining generalization across learner architectures.

TD3 — TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendation

Jiaqing Zhang, Mingjia Yin, Hao Wang et al. · WWW 2025Other datapaper ↗code ↗

TD3 distills sequential-recommendation interaction data within a meta-learning bi-level framework, using Tucker decomposition to store the synthetic sequence summary as four compact factors (synthetic user latent factor, temporal dynamics factor, shared item latent factor, and a relation core) instead of a full interaction tensor, and adds a surrogate feature-space alignment objective on top of naive performance matching, optimized with RaT-BPTT. The paper reports superior and cross-architecture generalizable performance versus prior baselines on multiple public recommendation datasets.

Behaviour Distillation

Andrei Lupu, Chris Lu, Jarek Liesen et al. · ICLR 2024Other datapaper ↗code ↗

Formalizes behaviour distillation: discovering and condensing the information needed to train an expert policy into a synthetic set of state-action pairs without access to expert data. Introduces HaDES, which searches such datasets with evolution strategies rather than backpropagation, and reports that as few as four state-action pairs can train continuous-control agents to competitive performance, generalizing out of distribution to other architectures and hyperparameters, and to zero-shot multi-task training.

Dataset Distillation with Attention Labels for Fine-tuning BERT

Aru Maekawa, Naoki Kobayashi, Kotaro Funakoshi et al. · ACL 2023Textpaper ↗code ↗

Proposes an attention-label distillation method for fine-tuning BERT on condensed text data, transferring attention-probability supervision from a teacher alongside a small number of learned synthetic examples in a bi-level, few-gradient-step distillation setup; reports 98.5% of full-dataset performance from a single example per class and a single gradient step, including 93.2% accuracy on AG News.

Towards Robust Dataset Learning

Yihan Wu, Xinda Li, Florian Kerschbaum et al. · arXiv 2022paper ↗

Poses learning a dataset such that any classifier naturally trained on it is adversarially robust as a tri-level optimization problem, avoiding the cost of adversarial training on the downstream task and showing the robustness property transfers between models and data. Proves the method learns a provably robust dataset under an abstraction that separates robust from non-robust features, and reports effectiveness across different network initializations and architectures on MNIST, CIFAR-10 and Tiny-ImageNet.

Compressed Gastric Image Generation Based on Soft-Label Dataset Distillation for Medical Data Sharing

Guang Li, Ren Togo, Takahiro Ogawa et al. · CMPB 2022paper ↗code ↗

Applies soft-label dataset distillation to gastric X-ray images so that tens of thousands of images compress into a handful of anonymized synthetic images plus a distilled model a fraction of the original size, enabling medical data sharing without exposing patient images. Reports high detection performance from the small compressed set alongside large reductions in image count and stored-model size.

On Implicit Bias in Overparameterized Bilevel Optimization

Paul Vicol, Jonathan P. Lorraine, Fabian Pedregosa et al. · ICML 2022paper ↗

Studies how the algorithmic choices of bi-level optimization — cold-start versus warm-start inner-loop initialization, and the hypergradient approximation used — implicitly bias which solution gradient-based bi-level methods converge to, in settings including hyperparameter optimization, meta-learning and dataset distillation, and shows warm-start solutions can retain substantial information about the outer objective even in low-dimensional problems.

Gastric SLDD — Soft-Label Anonymous Gastric X-ray Image Distillation

Guang Li, Ren Togo, Takahiro Ogawa et al. · ICIP 2020paper ↗code ↗

Applies gradient-descent dataset distillation, jointly optimizing distilled images, distilled soft labels and the learning rate, to gastric X-ray images so the resulting tiny distilled set both compresses the dataset and anonymizes the original patient images by construction. Reports the compressed images no longer carry identifiable patient information while remaining useful for training.

Distilled Replay: Overcoming Forgetting through Synthetic Samples

Andrea Rosasco, Antonio Carta, Andrea Cossu et al. · IJCAI 2021 Workshoppaper ↗code ↗

Builds a rehearsal buffer for continual learning by distilling the full dataset into one highly informative synthetic pattern per class, rather than storing real exemplars, keeping the buffer extremely small. Reports effectiveness against popular replay-based continual learning baselines on four benchmarks.

CCMCL — Condensed Composite Memory Continual Learning

Felix Wiewel, Bin Yang · IJCNN 2021paper ↗code ↗

For rehearsal-based continual learning, learns a small set of shared components combined into per-class synthetic exemplars rather than learning each replay image directly, increasing memory efficiency of the stored rehearsal buffer. Reports improved performance over prior rehearsal baselines, especially at very small buffer sizes.

New Properties of the Data Distillation Method When Working With Tabular Data

Dmitry Medvedev, Alexander D'yakonov · AIST 2020Other datapaper ↗code ↗

This paper applies the original backpropagation-through-time dataset distillation method to tabular data and finds that a model trained on the distilled samples can outperform one trained on the original dataset, but that the distilled data generalizes poorly to models with different hyperparameters than the one used during distillation. It shows that distilling with multiple architectures simultaneously mitigates this poor cross-hyperparameter generalization.

DOSFL — Distilled One-Shot Federated Learning

Yanlin Zhou, George Pu, Xiyao Ma et al. · arXiv 2020paper ↗

Each client distills its private dataset into synthetic data and uploads only that in a single round, training a global model without exchanging model weights or gradients; the distilled data is reported useless to anyone without the exact initial model weights, since it is tied to the specific training trajectory it was distilled for. Reports up to three orders of magnitude less communication than FedAvg while retaining 93-99% of centralized-training performance across vision and language tasks.

Federated Learning via Synthetic Data

Jack Goetz, Ambuj Tewari · arXiv 2020paper ↗

Proposes transmitting a small amount of synthetic data from client to server in place of a gradient update, avoiding the multi-million-parameter cost of standard federated learning updates. Reports more than an order-of-magnitude reduction in communication cost with minimal model degradation in early experiments.

Reducing catastrophic forgetting with learning on synthetic data

Wojciech Masarczyk, Ivona Tautkute · CVPR 2020 Workshoppaper ↗

Generates synthetic replay data via a two-step meta-gradient optimization process specifically so that learning tasks on it in sequence does not cause catastrophic forgetting. Reports that training on such synthetic data in sequence avoids catastrophic forgetting on Split-MNIST, and that the generation method is robust across different learning scenarios.