Resources

Graph condensation needs a reset — Position: Graph Condensation Needs a Reset -- Move Beyond Full-dataset Training and Model-Dependence

Mridul Gupta, Samyak Jain, Vansh Ramani et al. · arXiv 2026notableGraphspaper ↗

A position paper arguing that mainstream graph condensation contradicts its own goal: gradient-matching methods must train on the full graph to produce the condensed one, generalize poorly across GNN architectures, and are judged by node compression ratios that ignore real resource savings and condensation overhead. It calls for lightweight, architecture-agnostic methods and resource-based evaluation.

Text dataset distillation report — Technical Report on Text Dataset Distillation

Keith Ando Ogawa, Bruno Lopes Yamamoto, Lucas Lauton de Alcantara et al. · arXiv 2025notableTextpaper ↗

A review of dataset distillation for text, tracing its path from adaptations of vision methods on embeddings, through transformer-based methods and the generation of discrete synthetic text, to distillation for decoder-only models above one billion parameters, and naming benchmark standardization, the discreteness of text, and complex tasks as the main open problems.

Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions

Luyang Fang, Xiaowei Yu, Jiazhang Cai et al. · arXiv 2025Textpaper ↗

Surveys knowledge distillation (KD) and dataset distillation (DD) as complementary strategies for compressing large language models. Covers KD methodologies (task-specific alignment, rationale-based training, multi-teacher frameworks) alongside DD techniques that synthesize compact fine-tuning data via optimization-based gradient matching, latent-space regularization, and generative synthesis, discusses combining the two, surveys applications in healthcare and education, and identifies open challenges in preserving emergent reasoning and linguistic diversity and in establishing evaluation protocols.

Liu & Du survey — The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions

Ping Liu, Jiawei Du · arXiv 2025corepaper ↗

This survey focuses on post-2023 advances that scale dataset distillation to ImageNet-1K and ImageNet-21K, organizing progress into trajectory matching, gradient matching, distribution matching, scalable generative approaches and decoupled optimization, and highlights SRe2L, soft-label strategies and lossless-distillation techniques as the field's key breakthroughs. It also reviews robustness to adversarial and backdoor attacks, non-IID data, and emerging applications in video, audio, multimodal learning, medical imaging and scientific computing.

Shang et al. survey — Dataset Distillation in the Era of Large-Scale Data: Methods, Analysis, and Future Directions

Xinyi Shang · arXiv 2025Vision–languagepaper ↗

This survey identifies four shifts since 2023 that earlier surveys predate: scaling distillation to large datasets, moving from optimizing distilled samples alone to improving the whole distillation pipeline, developing theoretical and empirical understanding of core mechanisms, and expanding from images to diverse modalities. It analyzes methods for each stage, benchmarks contemporary methods on large-scale datasets, summarizes distillation across unimodal and multimodal data, and lays out open challenges and directions.

A Survey on Graph Condensation

Hongjia Xu, Liangliang Zhang, Yao Ma et al. · arXiv 2024Graphspaper ↗code ↗

Surveys graph condensation, giving a formal problem definition and a taxonomy that splits methods by optimization objective into three types and by output into graph-modifying versus fully synthetic approaches, and reviews the datasets and evaluation metrics used in the field.

Graph reduction survey — A Comprehensive Survey on Graph Reduction: Sparsification, Coarsening, and Condensation

Mohammad Hashemi, Shengbo Gong, Juntong Ni et al. · IJCAI 2024Graphspaper ↗code ↗

This survey unifies graph sparsification, coarsening and condensation under one definition, introduces a hierarchical taxonomy of the challenges each addresses, reviews the technical details and practical applications of methods in all three families, and lays out research directions for keeping graph reduction effective as graphs continue to grow.

Graph condensation survey — Graph Condensation: A Survey

Xinyi Gao, Junliang Yu, Tong Chen et al. · TKDE 2025Graphspaper ↗code ↗

This survey organizes graph condensation research into five categories aligned with evaluation criteria (effectiveness, generalization, efficiency, fairness, robustness), examines optimization strategies and condensed-graph generation as the two core components across methods, empirically compares representative methods against the five criteria, and surveys applications and open-source libraries in the field.

Geng et al. survey — A Survey on Dataset Distillation: Approaches, Applications and Future Directions

Jiahui Geng, Zongxiong Chen, Yuandou Wang et al. · IJCAI 2023corepaper ↗code ↗

This survey proposes a taxonomy of dataset distillation approaches and systematically reviews the methods, data modalities, and applications (continual learning, neural architecture search, privacy protection) covered by the field to date, and summarizes open challenges and future directions.

Yu et al. review — Dataset Distillation: A Comprehensive Review

Ruonan Yu, Songhua Liu, Xinchao Wang · TPAMI 2023corepaper ↗code ↗

This review formalizes dataset distillation with an overall algorithmic framework common to existing methods, proposes a systematic taxonomy of methodologies and discusses their theoretical interconnections, and presents current challenges and future directions supported by extensive experiments.

Lei & Tao survey — A Comprehensive Survey of Dataset Distillation

Shiye Lei, Dacheng Tao · TPAMI 2023corepaper ↗code ↗

This survey taxonomizes dataset distillation methods into meta-learning and data-matching frameworks based on whether they explicitly mimic target-data performance, covers distillation algorithms, factorized dataset distillation, performance comparisons and applications, and identifies limitations around high-resolution data and complex label spaces as open challenges.

Sachdeva & McAuley survey — Data Distillation: A Survey

Noveen Sachdeva, Julian McAuley · TMLR 2023coreGraphsOther datapaper ↗

This survey presents a formal framework for data distillation with a detailed taxonomy of existing approaches, and covers the method across three data modalities: images, graphs, and user-item interactions (recommender systems), identifying current challenges and future research directions for each.

BPS — Balanced Dataset Distillation via Modeling Multiple Visual Pattern Distribution

Guanghui Shi, Xuefeng Liang, Qixiang Wen · CVPR 2026paper ↗code ↗

Models each class as multiple visual-pattern clusters via a hierarchical semantic structure, instead of assuming one cluster per class, and selects a balanced coreset of complementary class-general (center) and marginal (boundary) real samples from every cluster; proves the selected coreset aligns with the original dataset in information coverage and reports state-of-the-art results across four benchmarks with model-agnostic, one-shot-for-all-IPCs selection.

DeepCore — DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning

Chengcheng Guo, Bo Zhao, Yanbing Bai · DEXA 2022paper ↗

Contributes a code library unifying coreset-selection methods for deep learning and an empirical study of them on CIFAR-10 and ImageNet under matched settings. Reports that although individual methods have advantages in specific settings, random selection remains a strong baseline overall.

EL2N / GraNd — Deep Learning on a Data Diet: Finding Important Examples Early in Training

Mansheej Paul, Surya Ganguli, Gintare Karolina Dziugaite · NeurIPS 2021paper ↗

Introduces two per-example difficulty scores computable early in training, the Gradient Norm (GraNd) and the Error L2-Norm (EL2N), and shows they identify which training examples can be pruned without hurting generalization. Reports pruning half of CIFAR-10 with EL2N scores from a few epochs of training while slightly improving test accuracy, and that scores computed with one architecture generalize to others.

Forgetting — An Empirical Study of Example Forgetting during Deep Neural Network Learning

Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes et al. · ICLR 2019notablepaper ↗

Defines a 'forgetting event' as a training example transitioning from correctly to incorrectly classified over the course of training, and studies these events across benchmark datasets. Reports that some examples are forgotten frequently and others never, that a dataset's unforgettable examples generalize across architectures, and that a significant fraction of examples can be removed from training, ranked by forgetting frequency, without hurting generalization — the basis of the forgetting-score coreset baseline.

k-Center coreset — Active Learning for Convolutional Neural Networks: A Core-Set Approach

Ozan Sener, Silvio Savarese · ICLR 2018notablepaper ↗

Poses active learning for CNNs as core-set selection: choose a subset such that a model trained on it is competitive with one trained on the full pool, and derives a bound on this gap from the geometry of the selected points. Selects points with a k-Center greedy covering criterion in feature space, used throughout dataset condensation as the covering-based coreset baseline.

Herding — Herding Dynamical Weights to Learn

Max Welling · ICML 2009notablepaper ↗

Defines herding, a deterministic weakly-chaotic dynamical system that converts a set of observed moments directly into a sequence of pseudo-samples respecting those moments, without first fitting a joint probability model over the data. Used in the coreset-selection literature as the mean-matching baseline: samples are added greedily to minimize the distance between the running average of selected samples' features and the true class-mean feature.

C^2R — Mind Your Margin and Boundary: Are Your Distilled Datasets Truly Robust?

Muquan Li, Yingyi Ma, Yihong Huang et al. · ICML 2026notablepaper ↗

Argues prior robust dataset distillation trades off accuracy for robustness poorly because it treats all adversarial examples uniformly and does not explicitly widen inter-class decision-boundary separation; C^2R instead derives a robust-margin perturbation score to build an attack-aware curriculum that prioritizes the smallest-margin adversaries driving robust error, paired with a class-balanced contrastive robustness loss that widens boundary separation. Reports the best robust accuracy across six attacks on CIFAR-10/100, Tiny-ImageNet and ImageNet-1K subsets, beating prior robust DD by 2.8% on average.

SubPopMark — From Compression to Accountability: Harmless Copyright Protection for Dataset Distillation

Yan Liang, Ziyuan Yang, Mengyu Sun et al. · arXiv 2026paper ↗

Observes that networks trained on distilled data still memorize subpopulation-level prediction biases, and uses this to build SubPopMark, a harmless (non-backdoor) copyright and provenance marker for distilled datasets: a Copyright Verification Marker stage injects a class-consistent subpopulation bias while preserving the original distillation trajectory, and a User-Specific Tracing Marker stage adds user-distinguishable perturbations on top. Provenance is verified black-box by comparing a suspicious model's output-behavior signature against a reference bank built from standard and subpopulation-shifted test sets.

COBRA — Fair Dataset Distillation via Cross-Group Barycenter Alignment

Mohammad Hossein Moslemi, Nima Hosseini Dashtbayaz, Zhimin Mei et al. · ICML 2026notablepaper ↗code ↗

Argues that fairness gaps in distilled data come from mismatches between demographic subgroups' predictive patterns, not just group-size imbalance, and so persist even after correcting for imbalance; proposes distilling toward a group-imbalance-agnostic barycenter of the predictive information so subgroups induce similar representations. The plug-in is compatible with existing distillation methods and reported to substantially reduce the bias dataset distillation introduces.

Sneakdoor — SNEAKDOOR: Stealthy Backdoor Attacks against Distribution Matching-based Dataset Condensation

He Yang, Dongyi Lv, Song Ma et al. · NeurIPS 2025notablepaper ↗code ↗

Targets distribution-matching-based dataset condensation with a backdoor attack designed for stealth, exploiting the vulnerability of class decision boundaries and using a generative module that builds input-aware triggers aligned to local feature geometry to hide artifacts in both the condensed data and the triggered inference samples. Reports a better balance of attack success rate, clean accuracy and stealthiness than prior condensation backdoor attacks, evading both human inspection and statistical detection.

Attention Hijacking — Attention Hijacking: Backdooring Text Dataset Distillation via Semantic Anchors

Hang Ren · ICML 2026Textpaper ↗

Proposes a backdoor attack on text dataset distillation built on a "Semantic Anchoring Hypothesis": the attack reshapes gradients into input embeddings so the synthetic data evolves to turn a trigger word into an adversarial feature, integrated into the bi-level distillation loop so the attack satisfies both the clean-task and backdoor objectives at once.

DP-GENG — DP-GENG: Differentially Private Dataset Distillation Guided by DP-Generated Data

Shuo Shi, Jinghuai Zhang, Shijie Jiang et al. · AAAI 2026notablepaper ↗

Initializes the distilled dataset with DP-generated data for realism, then refines it under a small privacy budget with a DP-feature-matching (distribution-matching) objective and aligns examples to their class distribution with an expert model, alongside a privacy-budget allocation strategy with a theoretical DP guarantee. Reports outperforming prior differentially-private dataset-distillation methods in both utility and robustness to membership-inference attacks.

Dosser — Improving Noise Efficiency in Privacy-preserving Dataset Distillation

Runkai Zheng, Vishnu Asutosh Dasu, Yinong Oliver Wang et al. · ICCV 2025notablepaper ↗code ↗

Identifies that prior differentially-private dataset distillation wastes privacy budget because sampling and optimization are synchronized and training signals come from noisy, randomly-initialized networks; decouples sampling from optimization for better convergence and matches in an informative subspace to reduce the impact of DP noise. Reports a 10.0% accuracy improvement at 50 images per class on CIFAR-10 and an 8.3% gain using only one-fifth the distilled-set size of prior state-of-the-art private DD methods.

Soft labels leak — Dataset distillation for memorized data: Soft labels can leak held-out teacher knowledge

Freya Behrens, Lenka Zdeborová · ICLR 2026paper ↗

Shows that students trained on a teacher's soft labels can achieve non-trivial, sometimes near-perfect, accuracy on held-out data the teacher memorized but never generalized to, including on i.i.d. random datasets where generalization is a priori impossible, and that with enough soft labels the student can recover the teacher's predictions on all inputs, including the memorized held-out ones; the effect depends strongly on the softmax temperature used for the labels but persists across network capacities, architectures and dataset compositions.

UniDetox — UniDetox: Universal Detoxification of Large Language Models via Dataset Distillation

Huimin Lu, Masaru Isonuma, Junichiro Mori et al. · ICLR 2025notableTextpaper ↗code ↗

UniDetox distills a small synthetic text dataset that carries detoxifying signal, generated from one source model (GPT-2) via contrastive decoding, and fine-tunes any target LLM on it to reduce toxicity without per-model hyperparameter tuning. The paper reports that text distilled from GPT-2 successfully detoxifies larger, unrelated models (OPT, Falcon, LLaMA-2) with a single shared hyperparameter configuration, and that the distilled text also reduces politically biased content.

Dark Distillation: Backdooring Distilled Datasets without Accessing Raw Data

Ziyuan Yang, Ming Yan, Yi Zhang et al. · AAAI 2026notablepaper ↗

Shows that an attacker who intercepts an already-distilled dataset in transit, with no access to the raw data it came from, can still inject a working backdoor by reconstructing per-class conceptual archetypes from a model trained on the distilled set and updating those archetypes with a hybrid loss that adds backdoor information while preserving the original optimization trajectory. Reports that distilled datasets remain highly vulnerable to this raw-data-free attack across multiple raw datasets and distillation methods.

ROME — ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck

Zheng Zhou · ICML 2025notablepaper ↗code ↗

Applies the Information Bottleneck principle to distillation robustness with two loss terms — a performance-aligned term to preserve clean accuracy and a robustness-aligned term that aligns feature distributions between synthetic and adversarially perturbed images — and introduces an Improved Robustness Ratio (I-RR) metric to evaluate DD robustness more precisely. Reports up to roughly 40% and 35% I-RR improvements over existing DD methods under white-box and black-box attacks respectively on CIFAR-10/100.

RDC — Robust Dataset Condensation using Supervised Contrastive Learning

Nicole Hee-Yeon Kim · ICCV 2025notablepaper ↗code ↗

RDC is an end-to-end dataset-condensation method that mitigates label noise without a separate noise-reduction preprocessing step, using a "golden MixUp contrast" that applies contrastive learning tailored for robust condensation: synthetic samples sharpen class boundaries and reduce noisy representations, while an augmentation strategy pulls clean samples out of the noisy training data to enrich the synthetic images with real-data diversity. Reports outperforming other condensation approaches on CIFAR-10/100 across symmetric, asymmetric and real-world noise.

FairDD — FairDD: Fair Dataset Distillation

Qihang Zhou, Shenhao Fang, Shibo He et al. · NeurIPS 2025notablepaper ↗code ↗

Shows that dataset distillation does not fix, and typically worsens, unfairness toward protected-attribute minority groups because matching-based objectives align synthetic data to the whole distribution, which majority groups dominate; FairDD instead synchronously matches synthetic data to each protected-attribute group separately, without changing the underlying matching architecture. Reported to significantly improve fairness with a favorable accuracy trade-off across both distribution-matching and gradient-matching DD methods.

DD_LNL — Dataset Distillers Are Good Label Denoisers In the Wild

Lechao Cheng, Kaifeng Chen, Jiyang Li et al. · arXiv 2024paper ↗code ↗

Evaluates three existing dataset distillation methods (DATM, DANCE, RCIG) as an offline, feedback-loop-free alternative to noise evaluation for learning with noisy labels. Reports that distillation denoises effectively under symmetric random noise but struggles with structured asymmetric noise, which can be absorbed into the distilled samples, and that clean tail-class samples can be lossily compressed away.

Bias in DD — Mitigating Bias in Dataset Distillation

Justin Cui, Ruochen Wang, Yuanhao Xiong et al. · ICML 2024paper ↗

Finds that color and background biases present in the original dataset are amplified by dataset distillation while corruption bias is instead suppressed, and introduces a kernel-density-estimation-based sample reweighting scheme that reduces this amplification, reporting 91.5% accuracy versus 23.8% for vanilla DM on biased CMNIST (5% bias-conflict ratio) at IPC-50.

Adaptive Backdoor Attacks Against Dataset Distillation for Federated Learning

Ze Chai, Zhipeng Gao, Yijing Lin et al. · ICC 2024paper ↗

Proposes three adaptive dataset-condensation-based backdoor attacks against dataset distillation used in federated learning, in which the trigger is dynamically modified during the distillation process rather than fixed in advance.

GUARD — Towards Adversarially Robust Dataset Distillation by Curvature Regularization

Eric Xue, Yijiang Li, Haoyang Liu et al. · AAAI 2025notablepaper ↗code ↗

Adds curvature regularization to the distillation process so that models trained on the resulting distilled set are adversarially robust without the cost of full adversarial training. Reports outperforming standard adversarial training on both accuracy and robustness at lower computational overhead, and withstanding various adversarial attacks.

Group Distributionally Robust Dataset Distillation with Risk Minimization

Saeed Vahidian, Mingyu Wang, Jianyang Gu et al. · ICLR 2025notablepaper ↗code ↗

Argues that matching training-set convergence, the usual DD criterion, does not guarantee that a model trained on the synthetic set generalizes to under-represented subgroups, since coverage of low-density regions matters more than training error there; proposes combining clustering with minimization of a distributionally-robust risk measure on the loss during distillation. Reports a theoretical rationale and improved generalization and robustness across subgroups compared to standard DD.

Differentially Private Dataset Condensation

Tianhang Zheng, Baochun Li · NDSS 2024 Workshoppaper ↗

Proposes two differentially-private dataset condensation algorithms: LDPDC, a linear DC method that runs on a low-end CPU, and NDPDC, which uses a neural network for representation extraction under a DP feature/gradient-matching objective. Reports LDPDC performs comparably to recent privacy-preserving generative methods, while NDPDC gives acceptable DP guarantees with only mild utility loss relative to plain distribution matching.

Rethinking Backdoor Attacks on Dataset Distillation: A Kernel Method Perspective

Ming-Yu Chung, Sheng-Yen Chou, Chia-Mu Yu et al. · ICLR 2024notablepaper ↗

Counters the view that kernel-based dataset distillation counteracts backdoor risk, deriving two new theory-driven trigger-generation methods specialized for kernel-based distillation from a theoretical analysis of backdoor attacks under kernel methods. Reports its optimization-based trigger design produces resilient backdoor attacks that evade conventional backdoor detection and mitigation methods.

Rethinking Data Distillation: Do Not Overlook Calibration

Dongyao Zhu, Bowen Lei, Jie Zhang et al. · ICCV 2023notablepaper ↗

Finds that networks trained on distilled data resist standard calibration methods (temperature scaling, mixup), tracing the failure to a more concentrated distribution of maximum logits and to loss of semantically meaningful but classification-irrelevant information in the distilled data; proposes Masked Temperature Scaling and Masked Distillation Training to correct for these effects while preserving distillation efficiency.

TrustDD — Towards Trustworthy Dataset Distillation

Shijie Ma, Fei Zhu, Zhen Cheng et al. · PR 2024paper ↗code ↗

Observes that dataset distillation has only ever targeted closed-world in-distribution classification, ignoring out-of-distribution detection, and proposes Trustworthy Dataset Distillation (TrustDD), which distills both in-distribution samples and outliers so the condensed set trains models competent at both tasks; Pseudo-Outlier Exposure corrupts in-distribution samples to stand in for real outlier data when none is available. Reports POE surpasses the state-of-the-art Outlier Exposure method and that TrustDD generalizes better to open-world scenarios than preceding DD.

Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation

Noel Loo, Ramin Hasani, Mathias Lechner et al. · ICLR 2024notablepaper ↗

Strengthens the dataset-reconstruction attack on trained network parameters and proves it can recover the entire training set in the infinite-width Neural Tangent Kernel regime, showing empirically that success depends on deviation from the frozen infinite-width limit and that easily-reconstructed images tend to be dataset outliers. Turns this into a dataset distillation method by retraining on the reconstructed images, reporting high predictive accuracy from the reconstructed-then-distilled set.

DP-KIP-ScatterNet — Differentially Private Kernel Inducing Points using features from ScatterNets (DP-KIP-ScatterNet) for Privacy Preserving Data Distillation

Margarita Vinaroz, Mi Jung Park · arXiv 2023paper ↗code ↗

Applies DP-SGD to KIP (kernel inducing points), but finds that the convolutional NTK features KIP needs for good accuracy are too expensive to compute privately at scale; DP-KIP-ScatterNet substitutes fixed wavelet ScatterNet features, which need no pretraining (avoiding an extra privacy cost) and run on a single GPU. Reports superior performance to prior differentially-private data-distillation methods across several image datasets under formal DP guarantees.

Backdoor attacks on DD — Backdoor Attacks Against Dataset Distillation

Yugeng Liu, Zheng Li, Michael Backes et al. · NDSS 2023notablepaper ↗code ↗

Performs the first backdoor attack on models trained on dataset-distilled data, injecting triggers into the synthetic data during the distillation procedure itself rather than during model training; NAIVEATTACK adds a fixed trigger at the start of distillation, while DOORPING iteratively updates the trigger throughout the whole distillation process. Reports DOORPING reaches near-1.0 attack success rate across datasets, architectures and distillation techniques, and that the resulting backdoors evade multiple existing defense mechanisms.

Fair Graph Distillation

Qizhang Feng, Zhimeng Jiang, Ruiquan Li et al. · NeurIPS 2023Graphspaper ↗

Shows that GNNs trained on graphs condensed by existing graph-distillation methods can exhibit more severe group-fairness problems than GNNs trained on the original graph, then proposes a fair graph-distillation approach that generates small distilled graphs that are both fair and informative.

Towards Robust Dataset Learning

Yihan Wu, Xinda Li, Florian Kerschbaum et al. · arXiv 2022paper ↗

Poses learning a dataset such that any classifier naturally trained on it is adversarially robust as a tri-level optimization problem, avoiding the cost of adversarial training on the downstream task and showing the robustness property transfers between models and data. Proves the method learns a provably robust dataset under an abstraction that separates robust from non-robust features, and reports effectiveness across different network initializations and architectures on MNIST, CIFAR-10 and Tiny-ImageNet.

Private Set Generation with Discriminative Information

Dingfan Chen, Raouf Kerkouche, Mario Fritz · NeurIPS 2022notablepaper ↗code ↗

Rather than fitting a full private generative model to the data distribution, directly optimizes a small set of representative samples under differential privacy, supervised by discriminative information from the downstream task, which the paper argues is an easier and more DP-training-friendly target than full-distribution generative modeling. Reports greatly improved sample utility over prior state-of-the-art differentially private generation approaches for high-dimensional data.

Can we achieve robustness from data alone?

Nikolaos Tsilivis, Jingtong Su, Julia Kempe · ICML 2022 Workshoppaper ↗

Poses adversarially robust classification as a bi-level kernel-ridge-regression problem over Neural Tangent Kernels, optimizing a dataset once so that plain gradient descent on it yields a robust model with no specialized robust training algorithm. The paper reports experiments across standard vision benchmarks and models, and revisits a prior data-optimization approach to argue that robustness from standard training on an optimized dataset is harder to obtain than previously claimed.

Privacy for Free — Privacy for Free: How does Dataset Condensation Help Privacy?

Tian Dong, Bo Zhao, Lingjuan Lyu · ICML 2022notablepaper ↗

Argues dataset condensation, though designed for training efficiency, also serves as a private-data generator "for free," and proves for linear (then extended to non-linear) feature extractors that any one of the n raw samples has only O(m/n) influence on the parameter distribution of networks trained on m condensed samples. Empirically supports this with loss-based and likelihood-based membership-inference attacks against DC-synthesized data. This claim's empirical and theoretical basis is disputed by no-free-lunch-privacy-2022.

DO-ALL — Distill Once, Adapt Life-Long: Exploring Dataset Distillation for Continual Test-Time Adaptation

Hyun-Kurl Jang, Jihun Kim, Hyeokjun Kweon et al. · ECCV 2026notablepaper ↗code ↗

Before deployment, DO-ALL distills the source data once into a small set of synthetic anchors; during continual test-time adaptation, each target sample is matched to its closest anchor, which supplies a stable reference for source replay, representation alignment and manifold-smoothing regularization without retaining raw source data. As a plug-in, it is reported to consistently improve long-term robustness of existing CTTA algorithms on CIFAR100-C, ImageNet-C and CCC.

BTM — Geometric Characterisation and Structured Trajectory Surrogates for Clinical Dataset Condensation

Pafue Christy Nganjimi, Andrew Soltan, Danielle Belgrave et al. · arXiv 2026Other datapaper ↗

Gives a geometric account of why trajectory matching is hard to supervise with a small fixed synthetic set: such a set can only reproduce a limited, low-rank span of the parameter changes real SGD training induces, so a spectrally broad supervision signal creates a representability bottleneck. Proposes Bezier Trajectory Matching (BTM), which replaces SGD expert trajectories with quadratic Bezier surrogates between initial and final model states, reducing trajectory storage and better matching what a fixed synthetic set can represent. Reports BTM matching or improving on standard trajectory matching across five clinical tabular-EHR and time-series datasets (three NHS emergency-department cohorts, eICU and MIMIC-III), with the largest gains at low prevalence and low synthetic-data budgets.

DIET — DIET: Learning to Distill Dataset Continually for Recommender Systems

Jiaqing Zhang, Hao Wang, Mingjia Yin et al. · arXiv 2026Other datapaper ↗

Formulates streaming dataset distillation for recommender systems and proposes DIET, which maintains an evolving distilled memory updated stage-wise via a bi-level optimization framework with influence-guided initialization from influential samples and selective, influence-aware memory updates, so the distilled set stays aligned with long-term training dynamics without reprocessing the full streaming data; reports compressing training data to 1-2% of the original size while preserving full-data performance trends and cutting model-iteration cost by up to 60x, with generalization across model architectures.

UniRain — UniRain: Unified Image Deraining with RAG-based Dataset Distillation and Multi-objective Reweighted Optimization

Qianfeng Yang, Qiyuan Guan, Xiang Chen et al. · CVPR 2026Dense predictionpaper ↗code ↗

UniRain is a unified image-deraining framework that trains one asymmetric mixture-of-experts model across rain streak/raindrop and day/night conditions; to build its mixed training set it uses a retrieval-augmented-generation-based dataset distillation pipeline that selects high-quality samples from all public deraining datasets, alongside a multi-objective reweighted optimization strategy. The paper reports favorable performance against state-of-the-art deraining models on its proposed benchmarks and multiple public datasets.

DPD — Towards Realistic Remote Sensing Dataset Distillation with Discriminative Prototype-guided Diffusion

Yonghao Xu, Pedram Ghamisi, Qihao Weng · arXiv 2026paper ↗code ↗

Introduces dataset distillation to remote sensing image interpretation for the first time with discriminative prototype-guided diffusion (DPD): per-category prototypes in latent space anchor hyperspherical semantic regions that guide the reverse denoising trajectory, and multiple candidates per prototype are ranked by a latent classifier's logit margin so only the most discriminative generated samples form the distilled set. Reports realistic, diverse and discriminative distilled samples across three high-resolution remote sensing scene-classification benchmarks.

CD^2 — CD^2: Constrained Dataset Distillation for Few-Shot Class-Incremental Learning

Kexin Bao, Daichi Zhang, Hansong Zhang et al. · IJCAI 2025paper ↗

CD^2 targets few-shot class-incremental learning by pairing a classifier-guided dataset distillation module, which condenses each new session's few-shot samples into highly compacted exemplars, with a distillation constraint module that regularizes the previously learned class distribution so old sessions' distilled knowledge is not overwritten. The paper reports improvements over other class-incremental methods on three public benchmarks.

Asymmetric Synthetic Data Update for Domain Incremental Dataset Distillation

Minyoung Oh, Sim · ICLR 2026paper ↗code ↗

Introduces Domain Incremental Dataset Distillation, where domains arrive over time and must be continually distilled into a single synthetic set rather than distilled once from a fixed dataset; naively overwriting the synthetic set with each new domain causes catastrophic forgetting. Proposes an Asymmetric Synthetic Data Update strategy that adjusts per-sample update rates to balance stability against plasticity.

Secure and Explainable Fraud Detection in Finance via Hierarchical Multi-source Dataset Distillation

Yiming Qian, Thorsten Neumann, Xueyining Huang et al. · ICAIFW 2025Other datapaper ↗

Converts a trained random forest into transparent axis-aligned rule regions (leaf hyperrectangles) and generates synthetic transactions by uniformly sampling within each region, producing a compact, auditable surrogate dataset with rule-based explanations and calibrated uncertainty for multi-institution fraud detection. On the IEEE-CIS fraud dataset, reports 85-93% data volume reduction with competitive precision/micro-F1 and only a modest AUC drop, chance-level membership-inference attack success, and improved cross-cluster metrics when institutions share the distilled data.

HoP-TM — High-Order Progressive Trajectory Matching for Medical Image Dataset Distillation

Le Dong, Jinghao Bian, Jingyang Hou et al. · MICCAI 2025paper ↗code ↗

Extends trajectory matching for medical image distillation by matching a shape-wise potential that captures the geometry of intermediate points along expert parameter trajectories, not just terminal states, and by an easy-to-complex strategy that progressively matches parameters in order of complexity. Reports improved distillation performance on medical image classification while preserving privacy and keeping accuracy comparable to training on the original datasets.

Distilling Reinforcement Learning into Single-Batch Datasets

Connor Wilhelm, Dan Ventura · ECAI 2025Other datapaper ↗

Extends dataset distillation to reinforcement learning by distilling RL environments into a single batch of supervised-learning examples, using a novel extension of proximal policy optimization for meta-learning to fit the bi-level objective; demonstrates the approach on an extended cart-pole task, MuJoCo environments and Atari games, compressing complex RL environments into one-step supervised training and examining generalization across learner architectures.

D2C — Accelerating Diffusion Model Training under Minimal Budgets: A Condensation-Based Perspective

Rui Huang, Shitong Shao, Zikai Zhou et al. · CVPR 2026notablePre-training & transferpaper ↗

Applies a condensation-based perspective to training diffusion models efficiently rather than to classification: a Select phase scores real images with a diffusion difficulty score combined with interval sampling to pick a compact, informative subset, and an Attach phase enriches each selected image with additional semantic and visual conditioning signals. Reports an FID of 4.3 after 40k steps on ImageNet 256x256 with SiT-XL/2 using only 0.8% of the training images, about 233x and 100x faster than training vanilla SiT-XL/2 and SiT-XL/2 + REPA respectively.

FedWSIDD — FedWSIDD: Federated Whole Slide Image Classification via Dataset Distillation

Haolong Jin, Shenglin Liu, Cong Cong et al. · MICCAI 2025paper ↗code ↗

Each client distills its whole-slide images into a compact set of synthetic slides, using a distillation algorithm that adds stain normalization for histopathology, and transmits the synthetic slides (not model parameters) to a server that aggregates and redistributes them; clients then train locally on synthetic plus real slides. Reports improved local WSI classification, tolerance of heterogeneous local models, and preserved patient privacy on CAMELYON16/17.

Data-Distill-Net — Data-Distill-Net: A Data Distillation Approach Tailored for Reply-based Continual Learning

Wenyang Liao, Quanziang Wang, Yichen Wu et al. · arXiv 2025paper ↗

For replay-based continual learning, keeps a learnable memory buffer of real replay images but, instead of optimizing the images themselves, trains a lightweight module that generates learned soft labels distilling both the current task's data and knowledge accumulated in the prior buffer. Reports competitive accuracy and reduced forgetting across continual-learning benchmarks relative to standard replay baselines, at lower overhead than distilling the buffer images directly.

GPS — GPS: Distilling Compact Memories via Grid-based Patch Sampling for Efficient Online Class-Incremental Learning

Mingchuan Ma, Yuhao Zhou, Jindi Lv et al. · arXiv 2025paper ↗

For online class-incremental replay, generates memory samples by sampling a grid of pixels from each real image into a compact low-resolution representation, with no trainable model or optimization, then reassembles the grid at replay time — avoiding the bi-level optimization cost of prior distilled-memory replay methods. Reports 3-4 percentage-point average end-accuracy gains when substituted into existing replay frameworks under memory-constrained settings, with limited computational overhead.

FedVCK — FedVCK: Non-IID Robust and Communication-Efficient Federated Learning via Valuable Condensed Knowledge for Medical Image Analysis

Guochen Yan, Luyuan Xie, Xinyi Gao et al. · AAAI 2025notablepaper ↗code ↗

FedVCK condenses each client's data with latent distribution constraints to improve knowledge quality, and each round selectively condenses only knowledge the current global model has not yet assimilated, cutting redundant communication; on the server, relational supervised contrastive learning gives extra supervision for the global update. Reports outperforming state-of-the-art federated learning methods on medical tasks under non-IID data and limited communication budgets.

Offline Behavior Distillation

Shiye Lei, Sen Zhang, Dacheng Tao · NeurIPS 2024notableOther datapaper ↗code ↗

This paper formulates offline behavior distillation (OBD), synthesizing a small set of expert-like behavioral data from suboptimal offline RL data so that a policy can be learned quickly, and shows that naive decision-difference objectives are hard to optimize through the intractable bi-level problem and give only quadratic-discount-complexity guarantees. It introduces action-value weighted PBC (Av-PBC), proving an equivalence between policy performance and the action-value weighted decision difference, and reports faster convergence, better distillation performance and robust cross-architecture/optimizer generalization than the naive objectives on multiple D4RL datasets.

HFLDD — Dataset Distillation-based Hybrid Federated Learning on Non-IID Data

Xiufang Shi, Wei Zhang, Yuheng Li et al. · IEEE TNSE 2026paper ↗

For federated learning with skewed label distributions, groups clients into heterogeneous clusters whose combined labels are balanced, has cluster heads collect data distilled by their members, and trains with the server as if the pooled, distilled data were IID. Reports higher test accuracy and lower communication cost than baseline federated learning methods under severe label imbalance.

Histo-DD — Dataset Distillation for Histopathology Image Classification

Cong Cong, Shiyu Xuan, Sidong Liu et al. · arXiv 2024paper ↗

Adapts dataset distillation to histopathology whole-slide images by adding stain normalization and model augmentation to the distillation process, addressing the high color heterogeneity of histopathology patches. Reports more informative synthetic patches than coreset-selection and patch-sampling baselines on Camelyon16, TCGA-IDH and UniToPath at both patch- and slide-level classification, with architecture-agnostic synthetic samples.

CollabDM — One-Shot Collaborative Data Distillation

William Holland, Chandra Thapa, Sarah Ali Siddiqui et al. · ECAI 2024paper ↗code ↗

CollabDM captures the global data distribution across distributed clients in a single round of client-server communication, rather than merging independently-distilled local sets (which the paper shows degrades under client data heterogeneity). Reports outperforming the state-of-the-art one-shot distillation method on skewed distributed data and demonstrates benefit on 5G network attack detection.

Dataset Distillation for Offline Reinforcement Learning

Jonathan Light, Yuanzhe Liu, Ziniu Hu · ICML 2024 WorkshopOther datapaper ↗code ↗

Applies dataset distillation to offline reinforcement learning, distilling a smaller synthetic dataset (DDRL) from offline trajectories so that a policy trained on it matches one trained on the full offline dataset or with percentile behavioral cloning.

Behaviour Distillation

Andrei Lupu, Chris Lu, Jarek Liesen et al. · ICLR 2024Other datapaper ↗code ↗

Formalizes behaviour distillation: discovering and condensing the information needed to train an expert policy into a synthetic set of state-action pairs without access to expert data. Introduces HaDES, which searches such datasets with evolution strategies rather than backpropagation, and reports that as few as four state-action pairs can train continuous-control agents to competitive performance, generalizing out of distribution to other architectures and hyperparameters, and to zero-shot multi-task training.

InfoDist — Image Distillation for Safe Data Sharing in Histopathology

Zhe Li, Bernhard Kainz · MICCAI 2024paper ↗code ↗

Trains a latent diffusion model on histopathology data and builds a distilled, human-readable synthetic dataset by selecting the most informative generated images through graph community analysis of the representation space, addressing prior distillation methods' unreadable outputs and weak downstream performance. Reports downstream classifiers trained on the synthetic set reaching performance suitable for practical application, comparable to models trained on real data.

HCDC — Calibrated Dataset Condensation for Faster Hyperparameter Search

Mucong Ding, Yuancheng Xu, Tahseen Rabbani et al. · arXiv 2024paper ↗

Rather than matching gradients to preserve accuracy, HCDC condenses a synthetic validation set so that the ranking of models by hyperparameter/architecture choice is preserved between the condensed and real validation sets, matching hyperparameter gradients via implicit differentiation and an approximate inverse Hessian. The paper reports the condensed validation set keeps model rankings stable and speeds up hyperparameter and architecture search on both image and graph tasks.

DeSA — Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchors

Chun-Yin Huang, Kartik Srinivas, Xin Zhang et al. · ICML 2024notablepaper ↗code ↗

In serverless decentralized federated learning, DeSA synthesizes global anchor data from each client's raw data distribution and uses it for two local regularizers: aligning each client's latent embeddings to the anchors, and a knowledge-distillation loss that lets clients learn from one another, grounded in domain-adaptation and KD theory. Reports improved inter- and intra-domain accuracy across clients with diverse data and model heterogeneity.

FedAF — An Aggregation-Free Federated Learning for Tackling Data Heterogeneity

Yuan Wang, Huazhu Fu, Renuga Kanagavelu et al. · CVPR 2024notablepaper ↗

FedAF replaces iterative model aggregation with a collaborative distribution-matching condensation step, where clients jointly learn condensed data and soft labels from peer knowledge that the server then trains the global model on directly. It reports faster convergence and higher global-model accuracy than prior one-shot and iterative FL baselines under label- and feature-skew heterogeneity.

Distilled Datamodel with Reverse Gradient Matching

Jingwen Ye, Ruonan Yu, Songhua Liu et al. · CVPR 2024notablepaper ↗

Approximates each training sample's influence on a pretrained model without leave-one-out retraining, by first distilling the training set into a synset via reverse gradient matching (offline) and then using that synset to speed up leave-one-out evaluation and compute an attribution matrix (online). Reports training-data-attribution and data-quality evaluations comparable to direct retraining at substantially lower cost.

Progressive trajectory matching for medical dataset distillation

Zhen Yu, Yang Liu, Qingchao Chen · arXiv 2024paper ↗

Finds that plain trajectory matching (randomly matching parts of real training trajectories) is unstable and gives inferior results on medical images, so proposes a progressive trajectory-matching strategy for stability plus a dynamic overlap mitigation module that eliminates overlap across synthetic images and retrains parts of them to recover the diversity lost to the more stable matching. Reports 8.33% average improvement over prior state-of-the-art methods, and 11.7% at 2 images per class, on a new medical dataset distillation benchmark across multiple modalities.

Dataset Condensation Driven Machine Unlearning

Junaid Iqbal Khan · arXiv 2024paper ↗code ↗

Proposes new dataset condensation techniques used as a building block of an unlearning scheme, condensing training data so that an approximate-unlearning update is cheaper to compute while balancing privacy, utility and efficiency; also proposes using the resulting "condensed model" to retrain arbitrary models unaffected by the forgotten samples and applies the scheme to defend against membership inference and model inversion attacks.

MedSynth — MedSynth: Leveraging Generative Model for Healthcare Data Sharing

Renuga Kanagavelu · MICCAI 2024paper ↗

Condenses the knowledge in large medical datasets into a small generative model, combining an attention-based generator with a vision transformer so the model can produce a compact set of representative synthetic medical images; sharing the generator across hospitals avoids disclosing raw patient data. Reports outperforming state-of-the-art comparison methods and successfully defending against state-of-the-art membership-inference attacks.

DCFL — DCFL: Non-IID awareness Data Condensation aided Federated Learning

Shaohan Sha, YaFeng Sun · IJCNN 2024paper ↗code ↗

Groups federated clients by Centered Kernel Alignment similarity, then uses non-IID-aware dataset condensation so that clients within the same group share each other's condensed data as complementary information; CKA-guided client selection, filtering and data-enhancement techniques are added to make use of the condensed data efficiently. Reports competitive performance on MNIST, FashionMNIST, SVHN and CIFAR-10 against existing FL protocols.

FedDG — Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents

Yuqi Jia, Saeed Vahidian, Jingwei Sun et al. · ECCV 2024notablepaper ↗code ↗

Moves dataset distillation for federated learning from the client to the server: instead of clients distilling and uploading synthetic data, the server uses prior knowledge from pretrained deep generative models' latents to synthesize data representations from heterogeneous client model architectures, letting local devices train smaller surrogate models while the server trains a larger global model on a multi-modal distribution. Reports up to 40% accuracy improvement over non-distillation FL and 18% over existing dataset-distillation FL methods under high heterogeneity, plus faster convergence, with a theoretical argument that the process asymptotically resembles centralized training.

Discovering Galaxy Features via Dataset Distillation

Haowen Guan, Xuan Zhao, Zishi Wang et al. · NeurIPS 2023 Workshoppaper ↗code ↗

Uses dataset distillation on a class-balanced Galaxy Zoo 2 subset to visualize what a galaxy-morphology classifier relies on, treating the synthesized prototypical images as human-inspectable summaries of the features a neural net uses to classify galaxy morphology. Introduces a self-adaptive variant of trajectory matching to automate the distillation process, reporting enhanced performance on standard computer-vision benchmarks as a byproduct.

CaT — CaT: Balanced Continual Graph Learning with Graph Condensation

Yilun Liu, Ruihong Qiu, Zi Huang · ICDM 2023Graphspaper ↗code ↗

Proposes CaT for continual graph learning: before each model update, the newly arrived graph is condensed into a small synthetic replay graph stored in a Condensed Graph Memory alongside condensed replays of past graphs, and the model is updated by training directly on this memory (Training in Memory) rather than on the full incoming graph, which addresses the scale imbalance between new and historical data; reports superior effectiveness and efficiency over replay-based continual graph learning baselines on four benchmarks.

Multi-Source Domain Adaptation meets Dataset Distillation through Dataset Dictionary Learning

Eduardo Fernandes Montesuma, Fred Ngolè Mboula, Antoine Souloumiac · ICASSP 2024paper ↗

Poses multi-source domain adaptation and dataset distillation as one joint problem (MSDA-DD), combining Wasserstein Barycenter Transport and Dataset Dictionary Learning from the MSDA literature with the distribution-matching distillation objective. Reports state-of-the-art adaptation performance on four benchmarks with as little as one distilled sample per class.

IEM — Towards Efficient Deep Hashing Retrieval: Condensing Your Data via Feature-Embedding Matching

Tao Feng, Jie Zhang, Huashan Liu et al. · arXiv 2023paper ↗

Adapts dataset condensation to deep hashing retrieval with IEM (Information-intensive feature Embedding Matching), a distribution-matching-centered method that adds model and data augmentation to strengthen the condensed hashing-space features, since retrieval training does not benefit directly from condensation methods designed for classification accuracy. Reports superior performance and efficiency relative to applying mainstream condensation methods to deep hashing retrieval.

SSD — Summarizing Stream Data for Memory-Constrained Online Continual Learning

Jianyang Gu, Kai Wang, Wei Jiang et al. · AAAI 2024notablepaper ↗code ↗

SSD summarizes the incoming data stream into more informative replay samples by distilling the training characteristics of real images, maintaining consistency of training gradients and relationship to past tasks so summarized samples represent the stream better than raw exemplars at the same buffer size. Reports more than a 3% accuracy boost on sequential CIFAR-100 under an extremely restricted memory buffer, with limited extra computational overhead.

FedLGD — Federated Learning on Virtual Heterogeneous Data with Local-global Distillation

Chun-Yin Huang, Ruinan Jin, Can Zhao et al. · TMLR 2024paper ↗code ↗

Observes that distilling each client's local data independently can worsen federated heterogeneity, so FedLGD instead uses iterative distribution matching to inpaint global information into local synthetic ("virtual") data and federated gradient matching to distill global virtual anchor points that rectify heterogeneous local training. Reports outperforming prior heterogeneous-FL methods on benchmark and real-world datasets, including at large numbers of class-imbalanced clients.

FedLAP-DP — FedLAP-DP: Federated Learning by Sharing Differentially Private Loss Approximations

Hui-Po Wang, Dingfan Chen, Raouf Kerkouche et al. · arXiv 2023paper ↗code ↗

Clients synthesize a small set of samples that approximate their local loss landscape by simulating the gradients of real images within a local region; these loss surrogates, rather than model updates, are aggregated on the server to recover the global loss landscape, which the paper shows lets record-level differential privacy be enforced at the same privacy cost as gradient-sharing FL but with a better privacy-utility trade-off and faster convergence.

BIB — Bidirectional Learning for Offline Model-based Biological Sequence Design

Can Chen, Yingxue Zhang, Xue Liu et al. · ICML 2023Other datapaper ↗code ↗

Extends the bidirectional (forward/backward) offline model-based optimization idea to biological sequence design by replacing the NTK proxy with a linearized pretrained language model, giving a closed-form loss that keeps the biophysical information the NTK could not represent. A bi-level auxiliary model learns how to weight the forward and backward mappings, and a derived learning-rate-adaptation module is reported to improve DNA/protein sequence design over the NTK-only predecessor.

Dataset Distillation for Medical Dataset Sharing

Guang Li, Ren Togo, Takahiro Ogawa et al. · AAAI 2023 Workshoppaper ↗code ↗

Applies trajectory-matching dataset distillation (the linked code builds on MTT) to a medical imaging dataset to produce a small, shareable synthetic set for cross-institution medical data sharing, following the same authors' prior soft-label distillation work on gastric X-ray images.

GDD-FL — Communication-Efficient Federated Skin Lesion Classification with Generalizable Dataset Distillation

Yuchen Tian, Jiacheng Wang, Yueming Jin et al. · MICCAI 2023 Workshoppaper ↗

GDD-FL condenses thousands of skin-lesion images per client into one synthetic image per class, modeling dataset features as an uncertain Gaussian distribution so the synthetic images capture diverse semantics and resist distribution drift across clients; only these few synthesized images are transmitted once to train a global model. Reports reduced communication cost with performance superior to classical federated learning and existing dataset-distillation baselines.

DynaFed — DYNAFED: Tackling Client Data Heterogeneity with Global Dynamics

Renjie Pi, Weizhong Zhang, Yueqi Xie et al. · CVPR 2023notablepaper ↗code ↗

Instead of gathering client data, DynaFed reserves a short trajectory of global-model snapshots on the server and synthesizes a pseudo dataset so that a model trained on it mimics that trajectory's dynamics, then uses the synthetic data once to correct client drift during aggregation in later rounds. Reports effectiveness across extensive non-IID federated learning benchmarks without needing an external server-side dataset.

Meta Knowledge Condensation for Federated Learning

Ping Liu, Xin Yu, Joey Tianyi Zhou · ICLR 2023notablepaper ↗

Instead of exchanging model parameters, clients extract a "meta knowledge" representation, weighted by a dynamic per-sample assignment mechanism that adapts each training sample's contribution over the course of training, and send only this condensed representation to the server for model updates; clients also exchange meta knowledge with each other as conditional initialization to further reduce heterogeneity. Reports raising accuracy from 74.07% to 92.95% on MNIST under a restricted 10-round communication budget.

Compressed Gastric Image Generation Based on Soft-Label Dataset Distillation for Medical Data Sharing

Guang Li, Ren Togo, Takahiro Ogawa et al. · CMPB 2022paper ↗code ↗

Applies soft-label dataset distillation to gastric X-ray images so that tens of thousands of images compress into a handful of anonymized synthetic images plus a distilled model a fraction of the original size, enabling medical data sharing without exposing patient images. Reports high detection performance from the small compressed set alongside large reductions in image count and stored-model size.

BDI — Bidirectional Learning for Offline Infinite-width Model-based Optimization

Can Chen, Yingxue Zhang, Jie Fu et al. · NeurIPS 2022Other datapaper ↗code ↗

BDI adds a backward mapping to offline model-based optimization that distills information from the static design dataset into the high-scoring candidate designs it proposes, alongside the usual forward proxy. Using an infinite-width network and its neural tangent kernel makes the backward loss closed-form, which the paper reports mitigates the out-of-distribution failure of proxy-only gradient ascent across several design tasks.

FedD3 — Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environments

Rui Song, Dai Liu, Dave Zhenyu Chen et al. · IJCNN 2023paper ↗code ↗

FedD3 needs only one-shot communication: each client distills its local dataset independently and the server aggregates the decentralized distilled datasets directly for model training, instead of exchanging model updates iteratively. Reports large communication savings over other one-shot federated learning approaches, e.g. over 71% higher accuracy at similar communication volume, or 98% less communication volume at the same accuracy, training AlexNet on non-IID CIFAR-10 with 10 clients.

FedDM — FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning

Yuanhao Xiong, Ruochen Wang, Minhao Cheng et al. · CVPR 2023notablepaper ↗code ↗

Each client builds a local surrogate loss landscape by matching the distribution of a small synthetic set to its own real data, and only this synthetic data (not model weights) is transmitted so the server assembles a global training objective from multiple clients' surrogate landscapes. Reports fewer communication rounds and better model quality than other federated learning methods, and shows the synthetic-data transmission can be made differentially private under a Gaussian mechanism at the same privacy budget.

OLCGM — Sample Condensation in Online Continual Learning

Mattia Sangermano, Antonio Carta, Andrea Cossu et al. · IJCNN 2022paper ↗code ↗

OLCGM is a replay-based online continual learning strategy that continuously compresses its memory buffer with knowledge-condensation techniques as new data streams in, rather than dropping old samples outright when the buffer fills. Reports improved final accuracy over state-of-the-art replay strategies when the memory budget is small relative to the data's complexity.

FedSynth — FedSynth: Gradient Compression via Synthetic Data in Federated Learning

Shengyuan Hu, Jack Goetz, Kshitiz Malik et al. · arXiv 2022paper ↗

Instead of transmitting a sparsified model update, each client learns and uploads a small synthetic dataset such that training on it reproduces the client's local update, which the server recovers and aggregates as usual. Reports comparable or better upstream compression than random-masking baselines on three common federated learning benchmarks.

Wearable ImageNet: Synthesizing Tileable Textures via Dataset Distillation

George Cazenavette, Tongzhou Wang, Antonio Torralba et al. · CVPR 2022 Workshoppaper ↗code ↗

Generates tileable distilled "textures" by sampling random crops from a toroidal canvas of trajectory-matching-distilled pixels while enforcing that every crop is itself effective distilled training data for its class, producing infinite repeating patterns that visually summarize an ImageNet category and are suitable for printing on fabric or clothing.

ADD-GNN — Learning from Designers: Fashion Compatibility Analysis Via Dataset Distillation

Yulan Chen · ICIP 2022paper ↗

Proposes an Attention-based Dataset Distillation Graph Neural Network (ADD-GNN) that distills knowledge from designer-generated outfit data, rather than noisy user-generated social-media data, to guide feature representation learning, and models overall outfit compatibility with an attention-based graph neural network jointly optimized with the distillation component.

DENSE — DENSE: Data-Free One-Shot Federated Learning

Jie Zhang, Chen Chen, Bo Li et al. · NeurIPS 2022notablepaper ↗code ↗

For one-shot federated learning with heterogeneous client architectures, DENSE trains a generator to invert an ensemble of the uploaded client models into synthetic images, then distills that ensemble's knowledge into a single global model on the synthetic data, needing no public dataset and no extra information beyond model parameters. Reports outperforming the strongest prior one-shot FL baseline by 5.08% on CIFAR-10.

Gastric SLDD — Soft-Label Anonymous Gastric X-ray Image Distillation

Guang Li, Ren Togo, Takahiro Ogawa et al. · ICIP 2020paper ↗code ↗

Applies gradient-descent dataset distillation, jointly optimizing distilled images, distilled soft labels and the learning rate, to gastric X-ray images so the resulting tiny distilled set both compresses the dataset and anonymizes the original patient images by construction. Reports the compressed images no longer carry identifiable patient information while remaining useful for training.

Distilled Replay: Overcoming Forgetting through Synthetic Samples

Andrea Rosasco, Antonio Carta, Andrea Cossu et al. · IJCAI 2021 Workshoppaper ↗code ↗

Builds a rehearsal buffer for continual learning by distilling the full dataset into one highly informative synthetic pattern per class, rather than storing real exemplars, keeping the buffer extremely small. Reports effectiveness against popular replay-based continual learning baselines on four benchmarks.

CCMCL — Condensed Composite Memory Continual Learning

Felix Wiewel, Bin Yang · IJCNN 2021paper ↗code ↗

For rehearsal-based continual learning, learns a small set of shared components combined into per-class synthetic exemplars rather than learning each replay image directly, increasing memory efficiency of the stored rehearsal buffer. Reports improved performance over prior rehearsal baselines, especially at very small buffer sizes.

DOSFL — Distilled One-Shot Federated Learning

Yanlin Zhou, George Pu, Xiyao Ma et al. · arXiv 2020paper ↗

Each client distills its private dataset into synthetic data and uploads only that in a single round, training a global model without exchanging model weights or gradients; the distilled data is reported useless to anyone without the exact initial model weights, since it is tied to the specific training trajectory it was distilled for. Reports up to three orders of magnitude less communication than FedAvg while retaining 93-99% of centralized-training performance across vision and language tasks.

Federated Learning via Synthetic Data

Jack Goetz, Ambuj Tewari · arXiv 2020paper ↗

Proposes transmitting a small amount of synthetic data from client to server in place of a gradient update, avoiding the multi-million-parameter cost of standard federated learning updates. Reports more than an order-of-magnitude reduction in communication cost with minimal model degradation in early experiments.

Reducing catastrophic forgetting with learning on synthetic data

Wojciech Masarczyk, Ivona Tautkute · CVPR 2020 Workshoppaper ↗

Generates synthetic replay data via a two-step meta-gradient optimization process specifically so that learning tasks on it in sequence does not cause catastrophic forgetting. Reports that training on such synthetic data in sequence avoids catastrophic forgetting on Split-MNIST, and that the generation method is robust across different learning scenarios.

GTN — Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data

Felipe Petroski Such, Aditya Rawal, Joel Lehman et al. · ICML 2020notablepaper ↗code ↗

Generative Teaching Networks train a generator network to produce synthetic training data (and, in principle, training environments or curricula) on which a freshly initialized learner takes a few SGD steps before being evaluated on the target task, with the generator's parameters updated by differentiating through that whole short training process via meta-gradients. Used to speed up architecture evaluation in neural architecture search, GTN-NAS is reported to improve on the NAS state of the art for a given search proposal mechanism while using orders of magnitude less computation than typical NAS methods.