Optimization and training recipes

C^2R — Mind Your Margin and Boundary: Are Your Distilled Datasets Truly Robust?

Muquan Li, Yingyi Ma, Yihong Huang et al. · ICML 2026notablepaper ↗

Argues prior robust dataset distillation trades off accuracy for robustness poorly because it treats all adversarial examples uniformly and does not explicitly widen inter-class decision-boundary separation; C^2R instead derives a robust-margin perturbation score to build an attack-aware curriculum that prioritizes the smallest-margin adversaries driving robust error, paired with a class-balanced contrastive robustness loss that widens boundary separation. Reports the best robust accuracy across six attacks on CIFAR-10/100, Tiny-ImageNet and ImageNet-1K subsets, beating prior robust DD by 2.8% on average.

COBRA — Fair Dataset Distillation via Cross-Group Barycenter Alignment

Mohammad Hossein Moslemi, Nima Hosseini Dashtbayaz, Zhimin Mei et al. · ICML 2026notablepaper ↗code ↗

Argues that fairness gaps in distilled data come from mismatches between demographic subgroups' predictive patterns, not just group-size imbalance, and so persist even after correcting for imbalance; proposes distilling toward a group-imbalance-agnostic barycenter of the predictive information so subgroups induce similar representations. The plug-in is compatible with existing distillation methods and reported to substantially reduce the bias dataset distillation introduces.

PTM-ST — Multimodal Dataset Distillation via Phased Teacher Models

Shengbin Guo, Hang Zhao, Senqiao Yang et al. · ICLR 2026notableVision–languagepaper ↗code ↗

PTM-ST distills vision-language pairs by matching a stage-aware model of the teacher's training dynamics rather than a single trajectory, using a shortcut-based trajectory construction strategy to fit distinct training phases and stabilize the match, addressing performance gaps between phases and unstable teacher trajectories seen in prior multimodal trajectory-matching methods. On Flickr30K and MS-COCO the paper reports surpassing prior state-of-the-art methods by up to 13.5 percentage points absolute, with an average gain of 9.53 points on Flickr30K, while also reducing storage overhead.

FD2 — FD$^2$: A Dedicated Framework for Fine-Grained Dataset Distillation

Hongxu Ma, Guang Li, Shijie Wang et al. · ECCV 2026notablepaper ↗

Improves decoupled dataset distillation for fine-grained image datasets, where coarse class-label supervision otherwise yields distilled samples with large intra-class variation and limited inter-class separation; FD2 uses counterfactual attention learning during pretraining to build discriminative class prototypes, then applies a fine-grained characteristic constraint that aligns each sample with its prototype while repelling others, plus a similarity constraint that diversifies attention across same-class samples; reports improvements when combined with existing decoupled methods on fine-grained and general benchmarks.

PRISM — PRISM: Video Dataset Condensation with Progressive Refinement and Insertion for Sparse Motion

Jaehyun Choi, Jiwan Hur, Gyojin Han et al. · CVPR 2026notableVideopaper ↗

PRISM treats a condensed video as one fully coupled spatiotemporal structure rather than disentangling it into separate static and dynamic components, starting from minimal temporal anchors and progressively inserting key-frames only where linear interpolation between them fails to capture non-linear motion, guided by gradient misalignments between synthetic and real data. The paper reports competitive accuracy with state-of-the-art storage efficiency on standard video-distillation benchmarks by allocating representational capacity only where the motion needs it.

ROME — ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck

Zheng Zhou · ICML 2025notablepaper ↗code ↗

Applies the Information Bottleneck principle to distillation robustness with two loss terms — a performance-aligned term to preserve clean accuracy and a robustness-aligned term that aligns feature distributions between synthetic and adversarially perturbed images — and introduces an Improved Robustness Ratio (I-RR) metric to evaluate DD robustness more precisely. Reports up to roughly 40% and 35% I-RR improvements over existing DD methods under white-box and black-box attacks respectively on CIFAR-10/100.

RDC — Robust Dataset Condensation using Supervised Contrastive Learning

Nicole Hee-Yeon Kim · ICCV 2025notablepaper ↗code ↗

RDC is an end-to-end dataset-condensation method that mitigates label noise without a separate noise-reduction preprocessing step, using a "golden MixUp contrast" that applies contrastive learning tailored for robust condensation: synthetic samples sharpen class boundaries and reduce noisy representations, while an augmentation strategy pulls clean samples out of the noisy training data to enrich the synthetic images with real-data diversity. Reports outperforming other condensation approaches on CIFAR-10/100 across symmetric, asymmetric and real-world noise.

FairDD — FairDD: Fair Dataset Distillation

Qihang Zhou, Shenhao Fang, Shibo He et al. · NeurIPS 2025notablepaper ↗code ↗

Shows that dataset distillation does not fix, and typically worsens, unfairness toward protected-attribute minority groups because matching-based objectives align synthetic data to the whole distribution, which majority groups dominate; FairDD instead synchronously matches synthetic data to each protected-attribute group separately, without changing the underlying matching architecture. Reported to significantly improve fairness with a favorable accuracy trade-off across both distribution-matching and gradient-matching DD methods.

GUARD — Towards Adversarially Robust Dataset Distillation by Curvature Regularization

Eric Xue, Yijiang Li, Haoyang Liu et al. · AAAI 2025notablepaper ↗code ↗

Adds curvature regularization to the distillation process so that models trained on the resulting distilled set are adversarially robust without the cost of full adversarial training. Reports outperforming standard adversarial training on both accuracy and robustness at lower computational overhead, and withstanding various adversarial attacks.

Group Distributionally Robust Dataset Distillation with Risk Minimization

Saeed Vahidian, Mingyu Wang, Jianyang Gu et al. · ICLR 2025notablepaper ↗code ↗

Argues that matching training-set convergence, the usual DD criterion, does not guarantee that a model trained on the synthetic set generalizes to under-represented subgroups, since coverage of low-density regions matters more than training error there; proposes combining clustering with minimization of a distributionally-robust risk measure on the loss during distillation. Reports a theoretical rationale and improved generalization and robustness across subgroups compared to standard DD.

RaT-BPTT — Embarassingly Simple Dataset Distillation

Yunzhen Feng, Ramakrishna Vedantam, Julia Kempe · ICLR 2024notablepaper ↗code ↗

Revisits backpropagation-through-time dataset distillation directly as a bi-level optimization problem and introduces Random Truncated BPTT, combining gradient truncation with a random window over the unrolled steps to stabilize gradients and cover long-term dependencies, setting new state-of-the-art results across standard benchmarks and revealing pronounced intercorrelation among the resulting distilled points that a boosting mechanism can exploit to build near-optimal subsets at multiple budgets.

Rethinking Data Distillation: Do Not Overlook Calibration

Dongyao Zhu, Bowen Lei, Jie Zhang et al. · ICCV 2023notablepaper ↗

Finds that networks trained on distilled data resist standard calibration methods (temperature scaling, mixup), tracing the failure to a more concentrated distribution of maximum logits and to loss of semantically meaningful but classification-irrelevant information in the distilled data; proposes Masked Temperature Scaling and Masked Distillation Training to correct for these effects while preserving distillation efficiency.

InfMatch — Dataset Distillation by Influence Matching

Haoru Tan, Wang Wang, Sitong Wu et al. · CVPR 2026Vision–languagepaper ↗code ↗

Replaces process-level surrogates (per-step gradients or training trajectories) with a differentiable, sample-level estimator of each synthetic point's influence on the converged model parameters, computed in linear time by unrolling the optimization dynamics with a first-order Taylor approximation, then learns the synthetic set by matching this influence to that of the real data; reports 31.5% accuracy on Tiny-ImageNet at IPC-10 (a 4.7-point gain over NCFM) and extends to vision-language distillation on Flickr30K.

DGS — Difficulty-guided Sampling: Bridging the Target Gap between Dataset Distillation and Downstream Tasks

Mingzhuo Li, Guang Li, Linfeng Ye et al. · arXiv 2026paper ↗

Argues that existing distillation objectives match dataset-level statistics but ignore task-specific information needed for the downstream classifier, creating a target gap; introduces difficulty-guided sampling (DGS), a plug-in post-processing module that resamples an existing method's pool of distilled images to match a target difficulty distribution, and difficulty-aware guidance (DAG), which steers a diffusion generation process by the same difficulty signal. Reports consistent improvements when added to multiple existing distillation settings.

DiRe — DiRe: Diversity-promoting Regularization for Dataset Condensation

Saumyaranjan Mohanty, Aravind Reddy, Konda Reddy Mopuri · WACV 2026paper ↗code ↗

Proposes DiRe, a diversity regularizer combining cosine similarity and Euclidean distance terms that plugs into existing condensation methods off the shelf to reduce redundancy among synthesized samples; reports consistent generalization and diversity-metric improvements when added to state-of-the-art condensation methods from CIFAR-10 to ImageNet-1K.

DDTime — DDTime: Dataset Distillation with Spectral Alignment and Information Bottleneck for Time-Series Forecasting

Yuqi Li, Kuiye Ding, Chuanguang Yang et al. · arXiv 2025Time seriespaper ↗

Proposes DDTime, a lightweight plug-in for time-series-forecasting dataset distillation that corrects autocorrelation-induced value-alignment bias with a frequency-domain alignment mechanism and increases synthetic-sample diversity with an information-bottleneck-inspired inter-sample regularizer; reports about 30% relative accuracy gains over prior distillation methods across 20 benchmark datasets and several forecasting architectures, at roughly 2.49% computational overhead.

Beyond Random: Automatic Inner-loop Optimization in Dataset Distillation

Muquan Li, Hang Gou, Dongyang Zhang et al. · NeurIPS 2025paper ↗

Replaces the random truncation used in backpropagation-through-time dataset distillation with Automatic Truncated BPTT (AT-BPTT), which chooses truncation windows and sizes from stage-aware gradient statistics and a low-rank Hessian approximation, reporting a 6.16-point average accuracy gain over baselines on CIFAR-10/100, Tiny-ImageNet and ImageNet-1K along with a 3.9x speedup and 63% less memory.

Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation

Sheng-Feng Yu, Jia-Jiun Yao, Wei-Chen Chiu · ICLR 2025Pre-training & transferpaper ↗

Improves self-supervised dataset distillation with a low-dimensional basis parameterization shared by images and their self-supervised representation targets, fixed (non-random) augmentations to stabilize training against a source of instability specific to self-supervised objectives, and a lightweight network that links the representations of augmented views from the same image for more compact pairs; reports gains in distillation efficiency, cross-architecture generalization and downstream transfer performance.

FedWSIDD — FedWSIDD: Federated Whole Slide Image Classification via Dataset Distillation

Haolong Jin, Shenglin Liu, Cong Cong et al. · MICCAI 2025paper ↗code ↗

Each client distills its whole-slide images into a compact set of synthetic slides, using a distillation algorithm that adds stain normalization for histopathology, and transmits the synthetic slides (not model parameters) to a server that aggregates and redistributes them; clients then train locally on synthetic plus real slides. Reports improved local WSI classification, tolerance of heterogeneous local models, and preserved patient privacy on CAMELYON16/17.

DAViD — Dynamic-Aware Video Distillation: Optimizing Temporal Resolution Based on Video Semantics

Yinjie Zhao, Heng Zhao, Bihan Wen et al. · arXiv 2025Videopaper ↗

Trains a reinforcement-learning agent, updated with a teacher-in-the-loop reward, to predict a per-class temporal resolution for synthetic video clips instead of assuming uniform temporal redundancy across classes, which the paper identifies as a limitation of prior video distillation methods. Reports substantial accuracy improvements over existing video dataset distillation methods.

Temporal Saliency-Guided Distillation: A Scalable Framework for Distilling Video Datasets

Xulin Gu, Xinhao Zhong, Zhixing Wei et al. · arXiv 2025Videopaper ↗

Optimizes synthetic video clips directly against a single pretrained model rather than through a bi-level inner training loop, and adds a temporal-saliency-guided filtering mechanism that uses inter-frame differences to concentrate distillation on informative temporal cues while suppressing redundant frames. Reports state-of-the-art results on standard video dataset-distillation benchmarks.

CCFS — Curriculum Coarse-to-Fine Selection for High-IPC Dataset Distillation

Yanda Chen, Gongwei Chen, Miao Zhang et al. · CVPR 2025paper ↗code ↗

Addresses the accuracy decay of dataset distillation at high images-per-class by curriculum-selecting real images to combine with the synthetic set at each stage, coarse-to-fine and conditioned on the current synthetic data rather than choosing real data once and independently, reporting gains of up to 6.6 points on CIFAR-10 and 5.8 on CIFAR-100 over prior real+synthetic combination methods at high IPC.

CMI — Going Beyond Feature Similarity: Effective Dataset Distillation based on Class-Aware Conditional Mutual Information

Xinhao Zhong, Bin Chen, Hao Fang et al. · ICLR 2025paper ↗code ↗

Introduces conditional mutual information, estimated in the feature space of pretrained networks, as a class-aware complexity regularizer added to an existing dataset-distillation loss to counter synthetic sets that are needlessly hard to learn from, reporting improved accuracy and training efficiency as a general add-on to distribution-matching-based methods.

EDF — Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios

Kai Wang, Zekai Li, Zhi-Qi Cheng et al. · CVPR 2025paper ↗code ↗

Uses Grad-CAM activation maps to up-weight discriminative image regions and down-weights low-loss (easy, redundant) supervision signals during synthesis, targeting the small, high-activation regions typical of complex datasets, and introduces the Comp-DD benchmark of sixteen ImageNet-1K subsets (eight easy, eight hard) to measure this, reporting consistent gains over prior methods on the hard subsets.

Teddy — Teddy: Efficient Large-Scale Dataset Distillation via Taylor-Approximated Matching

Ruonan Yu, Songhua Liu, Jingwen Ye et al. · ECCV 2024paper ↗

Replaces the multi-step-gradient dependence of bi-level, trajectory-style dataset distillation with a memory-efficient first-order Taylor approximation, and replaces repeated per-iteration model training with a pre-cached pool of weak models generated from a single base model, reporting up to 12.8-point gains and 46.6% less runtime over prior methods on Tiny-ImageNet and full-size ImageNet-1K.

Towards Model-Agnostic Dataset Condensation by Heterogeneous Models

Jun-Yeong Moon, Jung Uk Kim, Gyeong-Moon Park · ECCV 2024paper ↗code ↗

Addresses the tendency of condensed images to overfit to the particular network used during distillation by condensing with multiple heterogeneous models simultaneously, using a Gradient Balance Module to equalize each model's contribution and a Mutual Distillation with spatial-semantic decomposition to keep their semantics aligned, improving the cross-architecture usability of condensed images built on distribution-alignment and trajectory-tracking methods.

UDD — UDD: Dataset Distillation via Mining Underutilized Regions

Shiguang Wang, Zhongyu Zhang, Jian Cheng · PRCV 2024paper ↗

Identifies underutilized regions inside synthetic images as wasted capacity in multi-formation-style distillation, and introduces two utilization-sensitive search policies (response-based and data-jittering-based) that dynamically locate and fill these regions during optimization, together with a category-wise feature contrastive loss for inter-class separability. Reports 4.0% and 3.7% accuracy gains over the next-best method on CIFAR-10 and CIFAR-100 at IPC=1.

Not All Samples Should Be Utilized Equally: Towards Understanding and Improving Dataset Distillation

Shaobo Wang, Yantai Yang, Qilong Wang et al. · CVPR 2025 Workshoppaper ↗

Measures sample difficulty by gradient norm and finds that different matching-based distillation methods implicitly favor samples of different difficulty, then extends neural scaling laws for data pruning to dataset distillation to argue that prioritizing easier real samples during synthesis improves the distilled set, especially at low IPC, packaging this as a Sample Difficulty Correction plug-in that improves 7 distillation methods across 6 datasets.

Histo-DD — Dataset Distillation for Histopathology Image Classification

Cong Cong, Shiyu Xuan, Sidong Liu et al. · arXiv 2024paper ↗

Adapts dataset distillation to histopathology whole-slide images by adding stain normalization and model augmentation to the distillation process, addressing the high color heterogeneity of histopathology patches. Reports more informative synthetic patches than coreset-selection and patch-sampling baselines on Camelyon16, TCGA-IDH and UniToPath at both patch- and slide-level classification, with architecture-agnostic synthetic samples.

FYI — FYI: Flip Your Images for Dataset Distillation

Byunggwan Son, Youngmin Oh, Donghyeon Baek et al. · ECCV 2024paper ↗code ↗

Identifies bilateral equivalence -- object parts appearing on one side of a real image tend to recur, mirrored, on other images in the dataset -- which pushes synthesis to duplicate discriminative parts on both sides of synthetic images and limits fine-grained discrimination; FYI embeds horizontal flipping into the distillation process to counter this without changing training objectives or architectures, and integrates into several state-of-the-art methods for consistent gains on CIFAR-10/100, Tiny-ImageNet and ImageNet.

Bias in DD — Mitigating Bias in Dataset Distillation

Justin Cui, Ruochen Wang, Yuanhao Xiong et al. · ICML 2024paper ↗

Finds that color and background biases present in the original dataset are amplified by dataset distillation while corruption bias is instead suppressed, and introduces a kernel-density-estimation-based sample reweighting scheme that reduces this amplification, reporting 91.5% accuracy versus 23.8% for vanilla DM on biased CMNIST (5% bias-conflict ratio) at IPC-50.

CondTSF — CondTSF: One-line Plugin of Dataset Condensation for Time Series Forecasting

Jianrong Ding, Zhanyu Liu, Guanjie Zheng et al. · NeurIPS 2024Time seriespaper ↗code ↗

Analyzes why classification-style dataset-condensation objectives are misaligned with time-series forecasting, where evaluation requires closeness of predictions rather than matching discrete labels, and proposes CondTSF, a one-line plugin that can be added to existing time-series condensation methods to shrink the gap between full-data and synthetic-data forecast predictions; reports consistent gains across eight benchmark datasets, largest at low condensation ratios.

BACON — BACON: Bayesian Optimal Condensation Framework for Dataset Distillation

Zheng Zhou, Hongbo Zhao, Guangliang Cheng et al. · arXiv 2024paper ↗code ↗

Casts dataset distillation as minimizing an expected risk functional over the joint data distribution under a Bayesian framework and derives a numerically tractable lower bound as an approximate solution, reporting a 3.46-point accuracy gain over IDM at IPC-10 on CIFAR-10 and 3.10 points on Tiny-ImageNet when combined with existing methods.

MDC — Multisize Dataset Condensation

Yang He, Lingao Xiao, Joey Tianyi Zhou et al. · ICLR 2024paper ↗code ↗

Compresses N separate condensation runs at N target sizes into a single condensation process by adding an adaptive subset loss that keeps any prefix subset of the condensed images representative on its own, addressing the 'subset degradation problem' where a subset of a larger condensed set underperforms a set condensed directly at that smaller size; reports 5.22-6.40 point average accuracy gains condensing CIFAR-10 to 10 images per class.

Model Pool — Improve Cross-Architecture Generalization on Dataset Distillation

Binglin Zhou, Linhao Zhong, Wentao Chen · arXiv 2024paper ↗

Selects distillation models from a diverse pool according to a probability distribution during synthesis, rather than optimizing against one fixed network, so the distilled images are less biased toward a single architecture, and applies knowledge distillation when evaluating the distilled set. Reports improved test accuracy across a range of evaluation architectures compared to single-model distillation baselines.

IADD — Importance-Aware Adaptive Dataset Distillation

Guang Li, Ren Togo, Takahiro Ogawa et al. · NN 2024paper ↗

Observes that parameter-/gradient-matching distillation treats every network parameter as equally important and proposes IADD, which automatically assigns importance weights to different parameters during distillation to synthesize more robust distilled sets. Reports outperforming other parameter-matching SOTA methods on multiple benchmarks and in cross-architecture generalization, and validates the method on real-world COVID-19 detection.

MIM4DD — MIM4DD: Mutual Information Maximization for Dataset Distillation

Yuzhang Shang, Zhihang Yuan, Yan Yan · NeurIPS 2023paper ↗

Adds a mutual-information objective, estimated via an InfoNCE-style contrastive loss between same-label (positive) and different-label (negative) real/synthetic sample pairs, as a plug-in regularizer for existing gradient- and trajectory-matching distillation methods, on the argument that maximizing shared information is a more principled compression criterion than the heuristic indicators these methods already match.

SeqMatch — Sequential Subset Matching for Dataset Distillation

Jiawei Du, Qin Shi, Joey Tianyi Zhou · NeurIPS 2023paper ↗code ↗

Argues that optimizing an entire synthetic dataset as one static, uniformly-updated entity couples its instances together and prevents later-training-epoch high-level features from being captured, and instead generates the synthetic instances sequentially so each new subset acquires knowledge the earlier subsets have not, improving over prior matching-based methods on SVHN, CIFAR-10/100 and Tiny-ImageNet.

DREAM+ — DREAM+: Efficient Dataset Distillation by Bidirectional Representative Matching

Yanqing Liu, Jianyang Gu, Kai Wang et al. · arXiv 2023paper ↗code ↗

Extends DREAM's representative-sample selection to bidirectional matching across multiple dimensions of the distillation objective, plugging into gradient-, distribution- and trajectory-matching frameworks alike and cutting the number of distillation iterations needed by more than 15x without hurting accuracy.

YOCO — You Only Condense Once: Two Rules for Pruning Condensed Datasets

Yang He, Lingao Xiao, Joey Tianyi Zhou · NeurIPS 2023paper ↗code ↗

Introduces two pruning rules — a Low LBPE (Loss-Based Prediction Error) Score and Balanced Construction — that shrink an already-condensed dataset to smaller sizes on demand without any extra condensation runs, for on-device settings with varying and limited compute; on CIFAR-10 at IPC-10, reports 6.98-8.89 point gains over condensation baselines and 6.31-23.92 point gains over pruning baselines.

Progressive DD — Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality

Xuxi Chen, Yu Yang, Zhangyang Wang et al. · ICLR 2024paper ↗code ↗

Argues a single synthetic subset cannot represent the drastically different training dynamics of early versus late training, and instead synthesizes multiple small synthetic subsets in sequence, each conditioned on the ones before it, training on their cumulative union without extra training time; improves existing distillation methods by up to 4.3 points and is the first to scale distillation to considerably larger synthetic datasets.

Gold from Ores — Distill Gold from Massive Ores: Bi-level Data Pruning towards Efficient Dataset Distillation

Yue Xu, Yong-Lu Li, Kaitong Cui et al. · ECCV 2024paper ↗code ↗

Studies which real training samples are essential to the distillation process itself, proposing a data-pruning criterion based on empirical loss value refined by each sample's causal effect on the distillation outcome, which improves several distillation algorithms' results, including on full ImageNet-1K and Kinetics-400.

DREAM — DREAM: Efficient Dataset Distillation by Representative Matching

Yanqing Liu, Jianyang Gu, Kai Wang et al. · ICCV 2023paper ↗code ↗

Replaces the random sampling of real images used as matching targets in gradient-, distribution- and trajectory-matching distillation with a representative-sample selection strategy chosen for even coverage and diversity, reducing the number of distillation iterations needed by more than 8x and improving final accuracy when trained longer.

Translative pre-training — Few-Shot Dataset Distillation via Translative Pre-Training

Songhua Liu, Xinchao Wang · ICCV 2023paper ↗

Learns a 'distillation space' via a translator network, pretrained on large datasets with image-to-image translation, that maps synthetic images optimized cheaply in an arbitrary fixed network's space into the space of the target few-shot distillation network, needing only a few adaptation steps per new target dataset; reports about 15x faster distillation with performance comparable to iterative bi-level baselines, and satisfactory generalization across datasets, budgets and class counts.

Acc-DD — Accelerating Dataset Distillation via Model Augmentation

Lei Zhang, Jie Zhang, Bowen Lei et al. · CVPR 2023paper ↗code ↗

Speeds up gradient-matching dataset distillation with two model-augmentation strategies — sampling early-stage (partially trained) models and perturbing model parameters — instead of the thousands of randomly initialized models gradient matching otherwise needs, reporting up to 20x faster synthesis with accuracy comparable to prior gradient-matching methods.

Parameter pruning DD — Dataset Distillation Using Parameter Pruning

Guang Li, Ren Togo, Takahiro Ogawa et al. · IEICE Transactions on Fundamentals 2023paper ↗

Prunes parameters that are difficult to match during the distillation process, on the premise that this yields synthetic datasets that are more robust to which parameters are matched. Reports improved distillation performance over baselines on two benchmark datasets.