Decoupled teacher-driven synthesis

SRe2L · NeurIPS 2023

Squeeze, recover, relabel: decouples model training from synthetic-data optimization, so resolution and evaluation architecture are free. Reports 60.8% on ImageNet-1K at 50 images per class, with synthesis about 52 times faster than trajectory matching on ConvNet-4.

CDA · TMLR 2024

Observes that the first recovery iterations fix each image's outline, and schedules augmentation from global to local. Reports 63.2% on ImageNet-1K at 50 IPC and the first distillation of ImageNet-21K at 224×224.

G-VBSM · CVPR 2024

Replaces one backbone, one layer set and one statistic with matching across several of each, and reports strong results on small and large datasets with the same method.

EDC · NeurIPS 2024

Maps the design space (soft category-aware matching, learning-rate schedule and other choices) and reports 48.6% on ImageNet-1K with ResNet-18 at 10 IPC.

DWA · NeurIPS 2024

Analyzes diversity among independently synthesized batches and perturbs teacher weights in directed ways so that each batch reflects a different large subset of the data.

DELT · CVPR 2025

Splits each class's budget into subtasks optimized for different numbers of iterations, breaking the uniformity that shared global supervision imposes on all images of a class.

FADRM · NeurIPS 2025

Adds data-level residual connections that keep local information from real images during pixel optimization, and reports halving training time and peak GPU memory.

RDED · CVPR 2024

Names realism, diversity and efficiency as the requirements at scale and builds synthetic images from selected real content instead of inversion: ImageNet-1K at 10 IPC in 7 minutes on one RTX 4090, reported at 42% top-1 with ResNet-18.

CIM — Condensing Large-Scale Datasets Directly with Minimal Information Loss

Xinyi Shang, Peng Sun, Bei Shi et al. · ECCV 2026notablepaper ↗code ↗

Argues that the squeeze-recover-relabel pipeline's implicit dual compression (data to model, model back to images) causes a distribution shift that makes the pretrained model an unreliable relabeler; CIM instead directly quantifies and minimizes a metric-driven information gap between real and synthetic distributions, reporting 48.7% top-1 on ImageNet-1K at IPC=10 in 80 minutes on one RTX-4090, ahead of NRR-DD and DELT by 2.6 and 2.9 points respectively.

FD2 — FD$^2$: A Dedicated Framework for Fine-Grained Dataset Distillation

Hongxu Ma, Guang Li, Shijie Wang et al. · ECCV 2026notablepaper ↗

Improves decoupled dataset distillation for fine-grained image datasets, where coarse class-label supervision otherwise yields distilled samples with large intra-class variation and limited inter-class separation; FD2 uses counterfactual attention learning during pretraining to build discriminative class prototypes, then applies a fine-grained characteristic constraint that aligns each sample with its prototype while repelling others, plus a similarity constraint that diversifies attention across same-class samples; reports improvements when combined with existing decoupled methods on fine-grained and general benchmarks.

Fixed Anchors Are Not Enough: Dynamic Retrieval and Persistent Homology for Dataset Distillation

Muquan Li, Hang Gou, Yingyi Ma et al. · CVPR 2026notablepaper ↗

Identifies a 'pull-to-anchor' effect in residual-matching decoupled pipelines that rely on static real patches, which reduces intra-class diversity; RETA's Dynamic Retrieval Connection selects a real patch from a prebuilt pool by minimizing a fit-complexity score in teacher feature space before injecting it via a residual connection, and Persistent Topology Alignment penalizes topological discrepancies between real and synthetic sets via persistence images over a mutual k-NN feature graph; reports 64.3% top-1 on ImageNet-1K with ResNet-18 at 50 images per class, +3.1 points over the prior best.

Grounding and Enhancing Informativeness and Utility in Dataset Distillation

Shaobo Wang, Yantai Yang, Guo Chen et al. · ICLR 2026notablepaper ↗

Defines Informativeness (information within one sample) and Utility (how essential a sample is to the training set) to mathematically characterize optimal dataset distillation, replacing the field's heuristic balancing of efficiency and quality; InfoUtil instantiates both with game-theoretic Shapley-value attribution to extract key per-sample information and gradient-norm-based selection of globally influential samples, reporting a 6.1% accuracy improvement over the prior state of the art on ImageNet-1K with ResNet-18.

OGM — Beyond Soft Label: Dataset Distillation via Orthogonal Gradient Matching

Deyu Bo, Xinchao Wang · CVPR 2026notablepaper ↗

Shows theoretically that batch-norm statistic matching in decoupled synthesis mainly aligns the scale of real and synthetic gradients but overlooks their direction, though experiments show direction, not scale, is what drives training; Orthogonal Gradient Matching (OGM) orthogonalizes real and synthetic gradients by fixing all singular values to one and matches their singular vectors with a forward-pass-only least-squares loss, reporting 47.0% top-1 with soft labels and 16.7% with hard labels at IPC=10 on ImageNet-1K, ahead of RDED.

HALD — Hard Labels In! Rethinking the Role of Hard Labels in Mitigating Local Semantic Drift

Jiacheng Cui, Bingkui Tong, Xinyue Bi et al. · ICML 2026notablepaper ↗code ↗

Shows theoretically and empirically that limiting soft-label crops per image to save storage causes local semantic drift -- visually ambiguous crops pull soft supervision away from the image's true label -- and that hybridizing hard labels back in as a content-invariant semantic anchor corrects it; HALD uses hard labels as an intermediate corrective signal alongside sparse soft labels, reaching 42.7% top-1 on ImageNet-1K with only 285MB of soft-label storage (a 100x reduction) and a 9.0-point gain over LPLD.

RLDD — Rethinking Long-tailed Dataset Distillation: A Uni-Level Framework with Unbiased Recovery and Relabeling

Xiao Cui, Yulei Qin, Xinyue Li et al. · AAAI 2026notablepaper ↗code ↗

Moves long-tailed dataset distillation away from trajectory matching to a statistical-alignment perspective, recalibrating Batch Normalization statistics via a full forward pass with dynamically adjusted momentum, using separate observer and teacher expert models for reliable statistics and soft-label generation, and initializing synthetic images by incrementally selecting high-confidence, diverse augmentations. Reports 15.6% top-1 accuracy improvement on CIFAR-100-LT and 11.8% on Tiny-ImageNet-LT at IPC=10, IF=10, over prior state-of-the-art long-tailed distillation methods.

OD3 — OD3: Optimization-free Dataset Distillation for Object Detection

Salwa K. Al Khatib, Ahmed ElHagry, Shitong Shao et al. · ICLR 2026notableDense predictionpaper ↗code ↗

OD3 distills an object-detection dataset without any bi-level optimization, in two stages it iteratively places real object instances at suitable locations in synthesized images (candidate selection), then screens out low-confidence placements with a pretrained observer model (candidate screening). On MS COCO and PASCAL VOC at compression ratios from 0.25% to 5%, the paper reports new state-of-the-art results, surpassing the prior detection-distillation method by more than 14 points of mAP50 at a 1.0% compression ratio.

INFER — Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator

Xin Zhang, Jiawei Du, Ping Liu et al. · ICLR 2025notablepaper ↗code ↗

Breaks the one-hot, per-class synthesis barrier of prior distillation with a Universal Feature Compensator (UFC) that enhances feature integration across classes and can generate multiple additional synthetic instances from a single UFC input, plus label interpolation across classes that shrinks stored soft-label size toward zero; reports outperforming SRe2L by 34.5% at IPC=50 on ImageNet-1K with ResNet-18.

DENSE — DENSE: Data-Free One-Shot Federated Learning

Jie Zhang, Chen Chen, Bo Li et al. · NeurIPS 2022notablepaper ↗code ↗

For one-shot federated learning with heterogeneous client architectures, DENSE trains a generator to invert an ensemble of the uploaded client models into synthetic images, then distills that ensemble's knowledge into a single global model on the synthetic data, needing no public dataset and no extra information beyond model parameters. Reports outperforming the strongest prior one-shot FL baseline by 5.08% on CIFAR-10.

ProtoBlend — Efficient Video Dataset Distillation via Cluster-Guided Prototype Blending

Chongle Ren, Guang Li, Wenbo Huang et al. · arXiv 2026Videopaper ↗

Builds distilled video sets without any gradient-based optimization of the stored clips: a teacher model selects one high-confidence temporal segment per source video, cluster-guided allocation partitions the selected clips per class in the teacher's feature space and assigns one distilled slot per cluster, and each slot is blended with an in-cluster anchor clip while their teacher predictions are mixed with the same coefficient. Reports a competitive accuracy-efficiency trade-off against iteratively optimized video distillation methods on four trimmed action-recognition benchmarks.

E2D — Accelerating Large-Scale Dataset Distillation via Exploration-Exploitation Optimization

Muhammad J. Alahmadi, Peng Gao, Feiyi Wang et al. · arXiv 2026paper ↗code ↗

Speeds up decoupled dataset distillation with a two-phase Exploration-Exploitation Distillation (E2D) schedule: initialization from full real images preserves semantic content, an exploration phase applies uniform updates to locate high-loss regions, and an exploitation phase concentrates further updates there. Reports surpassing prior decoupled methods on ImageNet-1K while running about 18x faster, and improving accuracy on ImageNet-21K at roughly 4.3x the speed.

PRISM — PRISM: Diversifying Dataset Distillation by Decoupling Architectural Priors

Brian B. Moser, Shalini Sarode, Federico Raue et al. · TMLR 2026paper ↗code ↗

Argues that decoupled distillation inherits the inductive bias of whichever teacher supervises it, over-smoothing intra-class diversity as dataset size grows; PRISM decouples logit-matching from batch-normalization-alignment supervision, assigning them to different teacher architectures (a primary model for logits, a stochastic subset for BN alignment), with a scalable cross-class batch-formation scheme for fast parallel synthesis; on ImageNet-1K it reproducibly outperforms single-teacher SRe2L and multi-teacher G-VBSM at low and mid IPC with measurably richer intra-class diversity.

Temporal Saliency-Guided Distillation: A Scalable Framework for Distilling Video Datasets

Xulin Gu, Xinhao Zhong, Zhixing Wei et al. · arXiv 2025Videopaper ↗

Optimizes synthetic video clips directly against a single pretrained model rather than through a bi-level inner training loop, and adds a temporal-saliency-guided filtering mechanism that uses inter-frame differences to concentrate distillation on informative temporal cues while suppressing redundant frames. Reports state-of-the-art results on standard video dataset-distillation benchmarks.

NRR-DD — Enhancing Dataset Distillation via Non-Critical Region Refinement

Minh-Tuan Tran, Trung Le, Xuan-May Le et al. · CVPR 2025paper ↗code ↗

Enriches synthetic images' non-critical regions with class-general information while preserving instance-specific fine-grained detail elsewhere, instead of the common all-or-nothing choice between the two feature types; also introduces Distance-Based Representative knowledge transfer, which trains without soft labels by using the distance between synthetic-data predictions and one-hot labels, storing only two distances per instance while remaining competitive across small- and large-scale settings.

CV-DD — Dataset Distillation via Committee Voting

Jiacheng Cui, Zhaoyi Li, Xiaochen Ma et al. · arXiv 2025paper ↗code ↗

Aggregates distributions and predictions from a committee of multiple pretrained models to produce higher-quality soft labels and reduce model-specific bias in decoupled synthesis; reports consistent gains over single- and multi-model distillation baselines and improved generalization to non-training-based frameworks and synthetic-to-real transfer tasks.

FocusDD — FocusDD: Real-World Scene Infusion for Robust Dataset Distillation

Youbing Hu, Yun Cheng, Olga Saukh et al. · arXiv 2025paper ↗

Uses a pretrained Vision Transformer to locate key information patches in real images and stitches several such multi-object patches into one resolution-independent distilled image, augmented with a downsampled view of the original for generalization; the first method to apply distilled datasets to object detection as well as classification, reporting 71.0%/62.6% ImageNet-1K validation accuracy for ResNet50/MobileNet-v2 at IPC 100, and 24.4%/32.1% mAP for YOLOv11n/s on COCO2017 at IPC 50.

Video DC study — A Large-Scale Study on Video Action Dataset Condensation

Yang Chen, Sheng Guo, Bo Zheng et al. · arXiv 2024Videopaper ↗code ↗

This large-scale study systematically examines video dataset condensation along three axes: temporal processing of video data, the evaluation protocol, and how condensation algorithms adapt to the space-time domain. It finds that labeling method matters a great deal for condensation performance, that simple sliding-window sampling suffices for temporal processing, and that dataset-distillation methods win in harder scenarios while sample-selection methods win in easier ones; it also proposes a unified evaluation protocol and reports state-of-the-art results on HMDB51, UCF101, SSv2 and K400 under it.

Curriculum Dataset Distillation

Zhiheng Ma, Anjia Cao, Funing Yang et al. · TIP 2025paper ↗code ↗

Synthesizes decoupled ImageNet-scale images along a simple-to-complex curriculum with curriculum evaluation, plus an adversarial term against the distilling network, to counter the homogeneity and simplicity of prior batch-to-global synthesis; reports gains of 11.1% on Tiny-ImageNet, 9.0% on ImageNet-1K and 7.3% on ImageNet-21K over prior large-scale distillation, with improved cross-architecture generalization and robustness to noise.

SC-DD — Self-supervised Dataset Distillation: A Good Compression Is All You Need

Muxin Zhou, Zeyuan Yin, Shitong Shao et al. · arXiv 2024Pre-training & transferpaper ↗code ↗

SC-DD observes that as the pretrained model used for SRe2L-style inversion grows, supervised pretraining's batch-norm statistics flatten and become less informative for recovering data, whereas self-supervised pretrained models keep larger BN-statistic variance that yields stronger gradient signal during synthesis; it therefore performs decoupled inversion from self-supervised pretrained models instead of supervised ones. On CIFAR-100, Tiny-ImageNet and ImageNet-1K, the paper reports outperforming prior supervised decoupled-distillation methods (SRe2L, MTT, TESLA, DC, CAFE and others) by large margins when larger backbones are used, under the same recovery and post-training budgets.

Information Compensation: A Fix for Any-scale Dataset Distillation

Peng Sun · ICLR 2024 Workshoppaper ↗

Addresses information loss that arises when decoupled ('squeeze-recover-relabel' style) dataset distillation is applied across different compression scales or budgets. Method details could not be independently verified: OpenReview returned only a browser-verification challenge, and no alternative fetchable copy of the abstract was found.

DCOD — Fetch and Forge: Efficient Dataset Condensation for Object Detection

Ding Qi, Jian Li, Jinlong Peng et al. · NeurIPS 2024Dense predictionpaper ↗

Proposes DCOD, the first dataset-condensation framework for object detection, in a two-stage Fetch-and-Forge pipeline: localization and classification information from the real dataset is first embedded into model parameters (Fetch), then synthetic images are reconstructed by inverting that model (Forge); a Foreground-Background Decoupling strategy and an Incremental PatchExpand step increase multi-instance diversity within a fixed storage budget. At a 1% compression rate it reports 46.4% and 24.7% AP50 on VOC and COCO respectively.