Timeline

Herding · ICML 2009 · Coreset selectionnotable

Defines herding, a deterministic weakly-chaotic dynamical system that converts a set of observed moments directly into a sequence of pseudo-samples respecting those moments, without first fitting a joint probability model over the data. Used in the coreset-selection literature as the mean-matching baseline: samples are added greedily to minimize the distance between the running average of selected samples' features and the true class-mean feature.

k-Center coreset · ICLR 2018 · Coreset selectionnotable

Poses active learning for CNNs as core-set selection: choose a subset such that a model trained on it is competitive with one trained on the full pool, and derives a bound on this gap from the geometry of the selected points. Selects points with a k-Center greedy covering criterion in feature space, used throughout dataset condensation as the covering-based coreset baseline.

DD · arXiv 2018 · Meta-learning through unrolled traininglandmark

Introduces dataset distillation: a small set of synthetic training images, not necessarily on the real-data manifold, optimized by backpropagating through several steps of gradient descent so that training a fixed-initialization network on them approximates training on the full dataset, compressing 60,000 MNIST images into 10 synthetic images with close to the original accuracy.

Forgetting · ICLR 2019 · Coreset selectionnotable

Defines a 'forgetting event' as a training example transitioning from correctly to incorrectly classified over the course of training, and studies these events across benchmark datasets. Reports that some examples are forgotten frequently and others never, that a dataset's unforgettable examples generalize across architectures, and that a significant fraction of examples can be removed from training, ranked by forgetting frequency, without hurting generalization — the basis of the forgetting-score coreset baseline.

GTN · ICML 2020 · The generator as the distilled artifactnotable

Generative Teaching Networks train a generator network to produce synthetic training data (and, in principle, training environments or curricula) on which a freshly initialized learner takes a few SGD steps before being evaluated on the target task, with the generator's parameters updated by differentiating through that whole short training process via meta-gradients. Used to speed up architecture evaluation in neural architecture search, GTN-NAS is reported to improve on the NAS state of the art for a given search proposal mechanism while using orders of magnitude less computation than typical NAS methods.

DC · ICLR 2021 · Gradient matchinglandmark

Introduces dataset condensation as gradient matching: at every training step, match the gradients a network computes on a small synthetic set to the gradients it computes on the corresponding class of real data, learning the synthetic images by backpropagating through this per-step matching loss rather than unrolling full training.

KIP · ICLR 2021 · Kernel and closed-form inner solverslandmark

Introduces Kernel Inducing Points: treats dataset distillation as meta-learning support points for kernel ridge regression under the infinite-width neural-tangent-kernel correspondence, giving a closed-form inner solution and compressing MNIST and CIFAR-10 by one to two orders of magnitude while the resulting synthetic data remains usable to train finite-width networks.

DSA · ICML 2021 · Gradient matchinglandmark

Adds differentiable Siamese augmentation — applying the same random augmentation to a real and a synthetic batch before computing the gradient-matching loss — reporting roughly 7-point accuracy gains on CIFAR-10 and CIFAR-100 over the unaugmented gradient-matching baseline.

KIP-ConvNet · NeurIPS 2021 · Kernel and closed-form inner solversnotable

Extends KIP's kernel-ridge-regression meta-learning to infinitely wide convolutional networks via a distributed NTK computation, reporting over 65% CIFAR-10 test accuracy from just 10 datapoints — up from a prior best of 40% — with matching state-of-the-art results across MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100 and SVHN.

DM · WACV 2023 · Distribution and feature matchinglandmark

Introduces distribution matching: instead of matching gradients or unrolling training, synthesize images so their class-wise feature means match those of real images across many randomly sampled embedding networks, removing the bi-level optimization entirely and making condensation cheap enough to scale to larger, more realistic datasets.

GCond · ICLR 2022 · Gradient matchinglandmark

Introduces GCond, which condenses a large graph into a small synthetic graph and features by imitating the GNN training trajectory on the original graph through a gradient-matching loss, jointly synthesizing node features and structure so that GNNs trained on the condensed graph transfer across architectures; reports approximating original test accuracy by 95.3% on Reddit, 99.8% on Flickr and 99.0% on Citeseer while shrinking graph size by over 99.9%.

DENSE · NeurIPS 2022 · Decoupled teacher-driven synthesisnotable

For one-shot federated learning with heterogeneous client architectures, DENSE trains a generator to invert an ensemble of the uploaded client models into synthetic images, then distills that ensemble's knowledge into a single global model on the synthetic data, needing no public dataset and no extra information beyond model parameters. Reports outperforming the strongest prior one-shot FL baseline by 5.08% on CIFAR-10.

CAFE · CVPR 2022 · Distribution and feature matchingcore

Aligns multi-scale feature statistics between real and synthetic batches instead of matching gradients directly, while preserving synthetic-set discriminability with a classification loss and a dynamic bi-level optimization schedule that adaptively prevents over- and under-fitting, reporting up to an 11-point accuracy gain over prior methods on SVHN.

MTT · CVPR 2022 · Trajectory matchinglandmark

Introduces trajectory matching: instead of matching single-step gradients, optimize synthetic data so that training a network on it for several steps lands close to where a network trained on real data lands after many more steps, using precomputed, cached expert training trajectories to avoid recomputing them, and shows this outperforms prior methods and scales to higher-resolution images.

IT-GAN · NeurIPS 2022 Workshop · Optimization in a generative latent spacecore

Freezes a pretrained GAN and optimizes the latent vectors that generate informative training images, rather than optimizing for visual realism, so synthesized samples are chosen to accelerate and improve downstream network training; verified to let networks train faster and reach better performance than with plain GAN samples, offering a path to dataset condensation without directly optimizing pixels.

IDC · ICML 2022 · Synthetic-set parameterizationcore

Proposes an efficient synthetic-data parameterization that packs multiple down-sampled or augmented image formations into the storage of one, exploiting spatial data regularity, together with an improved optimization for gradient-matching-based condensation; substantially improves condensed-data quality over the prior state of the art on CIFAR-10, ImageNet subsets, and Speech Commands.

Privacy for Free · ICML 2022notable

Argues dataset condensation, though designed for training efficiency, also serves as a private-data generator "for free," and proves for linear (then extended to non-linear) feature extractors that any one of the n raw samples has only O(m/n) influence on the parameter distribution of networks trained on m condensed samples. Empirically supports this with loss-based and likelihood-based membership-inference attacks against DC-synthesized data. This claim's empirical and theoretical basis is disputed by no-free-lunch-privacy-2022.

FRePo · NeurIPS 2022 · Kernel and closed-form inner solverscore

Replaces the closed-form kernel-ridge-regression solver of prior kernel-based distillation with regression onto the features of a pool of finite-width networks (neural Feature Regression with Pooling), reducing memory by an order of magnitude and training time by two orders of magnitude while improving accuracy on CIFAR-100, Tiny-ImageNet and ImageNet-1K.

Distill-CF · NeurIPS 2022 · Kernel and closed-form inner solversnotable

Uses the Neural Tangent Kernel equivalence of an infinitely-wide autoencoder (infinity-AE) to give recommendation-model training a closed-form solution, then builds Distill-CF on top of it with differentiable Gumbel-sampling to synthesize tiny, high-fidelity summaries of large, sparse user-item interaction matrices; reports 96-105% of infinity-AE's full-data performance using as little as 0.1% of the original interactions.

Addressable memories · NeurIPS 2022 · Synthetic-set parameterizationnotable

Compresses a dataset into compact addressable memories -- a set of bases ('memories') shared across classes and combined through learned addressing functions to generate diverse training examples -- so compressed size need not grow linearly with the number of classes, and generalized queries beyond the original classes become possible; reports up to 16.5% and 9.7% retained-accuracy gains on CIFAR-10/100 respectively and, applied to continual learning, a 23.2% accuracy improvement on the MANY benchmark.

FedDM · CVPR 2023 · Distribution and feature matchingnotable

Each client builds a local surrogate loss landscape by matching the distribution of a small synthetic set to its own real data, and only this synthetic data (not model weights) is transmitted so the server assembles a global training objective from multiple clients' surrogate landscapes. Reports fewer communication rounds and better model quality than other federated learning methods, and shows the synthetic-data transmission can be made differentially private under a Gaussian mechanism at the same privacy budget.

DC-BENCH · NeurIPS 2022core

Introduces DC-BENCH, the first large-scale standardized benchmark for dataset condensation, evaluating methods under controlled data-augmentation and architecture choices so that gains attributable to the condensation method itself can be separated from evaluation-pipeline confounds, and releases evaluators, baselines and generated datasets for reproducible comparison.

No Free Lunch · arXiv 2022notable

A rebuttal of Dong et al. (2022, "Privacy for Free"), arguing its empirical evaluation and theoretical analysis of dataset condensation's privacy benefit both have major flaws, so the original paper does not provide statistically significant evidence that DC improves privacy over a naive baseline. Points out that DP-SGD, the standard privacy-preserving training approach, simultaneously gives better accuracy and a provably lower membership-attack success rate than the condensation-based claims it disputes.

Meta Knowledge Condensation for Federated Learning · ICLR 2023notable

Instead of exchanging model parameters, clients extract a "meta knowledge" representation, weighted by a dynamic per-sample assignment mechanism that adapts each training sample's contribution over the course of training, and send only this condensed representation to the server for model updates; clients also exchange meta knowledge with each other as conditional initialization to further reduce heterogeneity. Reports raising accuracy from 74.07% to 92.95% on MNIST under a restricted 10-round communication budget.

RFAD · NeurIPS 2022 · Kernel and closed-form inner solversnotable

Replaces KIP's exact O(|S|^2) neural tangent kernel computation with a random feature approximation of the Neural Network Gaussian Process kernel that scales as O(|S|), giving at least a 100x speedup that runs on a single GPU while matching KIP's accuracy across kernel-regression and finite-width training, including on model-interpretability and privacy-preservation tasks.

HaBa · NeurIPS 2022 · Synthetic-set parameterizationcore

Factorizes a distilled dataset into a small set of shared 'bases' and per-image 'hallucination networks' that recombine them into training images, a plug-and-play representation portable to any distillation baseline, with adversarial contrastive constraints on the bases and hallucinators to keep generated images diverse and discriminative; reports improved downstream accuracy while cutting compressed parameters by up to 65%, and about 10 points higher cross-architecture accuracy than baselines.

Private Set Generation with Discriminative Information · NeurIPS 2022notable

Rather than fitting a full private generative model to the data distribution, directly optimizes a small set of representative samples under differential privacy, supervised by discriminative information from the downstream task, which the paper argues is an easier and more DP-training-friendly target than full-distribution generative modeling. Reports greatly improved sample utility over prior state-of-the-art differentially private generation approaches for high-dimensional data.

TESLA · ICML 2023 · Trajectory matchingcore

Derives a way to compute MTT's unrolled trajectory-matching gradient exactly with constant memory (about 6x less), letting trajectory matching scale to ImageNet-1K, and introduces a soft-label assignment that improves convergence on datasets with many classes; scales to 50 images per class on ImageNet-1K on a single GPU (versus 2 IPC for prior methods) with only a 5.9-point accuracy drop from full-data training using 4.2% of the data.

FTD · CVPR 2023 · Trajectory matchingnotable

Identifies the accumulated trajectory error of trajectory-matching distillation — the mismatch between the short synthetic trajectory optimized during distillation and the long trajectory used at evaluation — and regularizes expert trajectories toward flatter minima so that weights trained on the synthetic data are more robust to this error, improving accuracy by up to 4.7 points on a higher-resolution ImageNet subset.

DynaFed · CVPR 2023 · Trajectory matchingnotable

Instead of gathering client data, DynaFed reserves a short trajectory of global-model snapshots on the server and synthesizes a pseudo dataset so that a model trained on it mimics that trajectory's dynamics, then uses the synthetic data once to correct client drift during aggregation in later rounds. Reports effectiveness across extensive non-IID federated learning benchmarks without needing an external server-side dataset.

An Efficient Dataset Condensation Plugin and Its Application to Continual Learning · NeurIPS 2023 · Synthetic-set parameterizationnotable

Observes that dataset-condensation matching objectives operate in high-dimensional pixel space even though natural images are locally connected and have lower intrinsic dimension, and proposes a plugin that instead condenses images into two low-rank matrices, matching raw and synthetic data in this low-dimensional manifold for higher condensation efficiency. Demonstrated as a continual-learning replay buffer, presented at NeurIPS 2023.

Backdoor attacks on DD · NDSS 2023notable

Performs the first backdoor attack on models trained on dataset-distilled data, injecting triggers into the synthetic data during the distillation procedure itself rather than during model training; NAIVEATTACK adds a fixed trigger at the start of distillation, while DOORPING iteratively updates the trigger throughout the whole distillation process. Reports DOORPING reaches near-1.0 attack success rate across datasets, architectures and distillation techniques, and that the resulting backdoors evade multiple existing defense mechanisms.

Sachdeva & McAuley survey · TMLR 2023core

This survey presents a formal framework for data distillation with a detailed taxonomy of existing approaches, and covers the method across three data modalities: images, graphs, and user-item interactions (recommender systems), identifying current challenges and future research directions for each.

Lei & Tao survey · TPAMI 2023core

This survey taxonomizes dataset distillation methods into meta-learning and data-matching frameworks based on whether they explicitly mimic target-data performance, covers distillation algorithms, factorized dataset distillation, performance comparisons and applications, and identifies limitations around high-resolution data and complex label spaces as open challenges.

Yu et al. review · TPAMI 2023core

This review formalizes dataset distillation with an overall algorithmic framework common to existing methods, proposes a systematic taxonomy of methodologies and discusses their theoretical interconnections, and presents current challenges and future directions supported by extensive experiments.

Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation · ICLR 2024 · Kernel and closed-form inner solversnotable

Strengthens the dataset-reconstruction attack on trained network parameters and proves it can recover the entire training set in the infinite-width Neural Tangent Kernel regime, showing empirically that success depends on deviation from the frozen infinite-width limit and that easily-reconstructed images tend to be dataset outliers. Turns this into a dataset distillation method by retraining on the reconstructed images, reporting high predictive accuracy from the reconstructed-then-distilled set.

RCIG · ICML 2023 · Kernel and closed-form inner solversnotable

Formulates dataset distillation as a bi-level problem solved with implicit gradients, convexifies the meta-gradient computation by learning on top of a frozen finite-width neural tangent kernel, and reduces implicit-gradient bias by analytically computing final-layer parameters from the body parameters; reports a 108% average improvement over prior distillation algorithms at one image per class on resized ImageNet, 66% on Tiny-ImageNet and 37% on CIFAR-100.

DiM · arXiv 2023 · The generator as the distilled artifactcore

Trains a generative model to store the target dataset by minimizing the difference in logits, from a pool of models, between real and generated images, so the same trained generator produces samples for any distillation ratio and large architectures without re-optimizing; reports state-of-the-art results on four datasets, including higher accuracy on complex architectures (75.1% ResNet-18) than simple ones (72.6% ConvNet-3) at 10 images per class on CIFAR-10.

GLaD · CVPR 2023 · Optimization in a generative latent spacelandmark

Distills into a small number of intermediate feature vectors in a pretrained deep generative model's latent space instead of pixels, augmenting existing matching-based distillation objectives with this generative prior; significantly improves cross-architecture generalization across settings and is among the first to combine a generative prior with dataset-distillation matching objectives at this scale.

Geng et al. survey · IJCAI 2023core

This survey proposes a taxonomy of dataset distillation approaches and systematically reviews the methods, data modalities, and applications (continual learning, neural architecture search, privacy protection) covered by the field to date, and summarizes open challenges and future directions.

SSD · AAAI 2024 · Gradient matchingnotable

SSD summarizes the incoming data stream into more informative replay samples by distilling the training characteristics of real images, maintaining consistency of training gradients and relationship to past tasks so summarized samples represent the stream better than raw exemplars at the same buffer size. Reports more than a 3% accuracy boost on sequential CIFAR-100 under an extremely restricted memory buffer, with limited extra computational overhead.

SFGC · NeurIPS 2023 · Trajectory matchingnotable

SFGC drops the joint node-and-structure optimization of prior graph condensation methods and instead distills only a condensed node set with topology reduced to the identity matrix (graph-free data), implicitly folding structural information into the node attributes; a training-trajectory meta-matching scheme aligns the long-term GNN learning behavior between the original and condensed graph, and a closed-form graph neural feature score dynamically evaluates candidate condensed data during training. The paper reports superior performance across different condensation ratios versus prior graph condensation baselines.

SRe2L · NeurIPS 2023 · Decoupled teacher-driven synthesislandmark

Introduces Squeeze, Recover and Relabel (SRe2L), decoupling the bilevel optimization between model and synthetic data: a teacher is trained once on the real data (squeeze), synthetic images are then optimized by matching the frozen teacher's batch-norm statistics and predictions (recover), and student networks are trained against the teacher's soft labels (relabel); the pipeline that first made ImageNet-1K-scale condensation practical, reaching 60.8% top-1 accuracy on ImageNet-1K and 42.5% on Tiny-ImageNet at IPC=50, more than 30 points above prior methods, and 16-52x faster than MTT with far less memory.

IDM · CVPR 2023 · Distribution and feature matchingcore

Identifies two weaknesses of plain distribution matching — an imbalanced number of features per matching term and embeddings not validated for the distance computation — and addresses them with partitioning-and-expansion augmentation, a richer pool of partially trained sampled models, and class-aware distribution regularization, outperforming most optimization-oriented distillation methods at a fraction of their compute.

Rethinking Data Distillation: Do Not Overlook Calibration · ICCV 2023 · Optimization and training recipesnotable

Finds that networks trained on distilled data resist standard calibration methods (temperature scaling, mixup), tracing the failure to a more concentrated distribution of maximum logits and to loss of semantically meaningful but classification-irrelevant information in the distilled data; proposes Masked Temperature Scaling and Masked Distillation Training to correct for these effects while preserving distillation efficiency.

MTT-VL · TMLR 2024 · Trajectory matchinglandmark

This is the first vision-language dataset distillation method, extending trajectory matching to image-text pairs by jointly distilling them in a contrastive formulation (since there are no discrete classes to condition on) and using LoRA matching for efficient trajectory matching in large vision-language models. Against adapted vision-language coreset-selection baselines, the paper reports nearly doubling Flickr30K image-to-text recall@1 (5.6% to 9.9%) while using 100 distilled pairs versus 1000 selected ones.

DQ · ICCV 2023 · Dataset quantizationcore

Partitions a dataset into non-overlapping bins by a diversity-driven criterion and samples from every bin, decoupling the compressed subset from any specific matching architecture; reports the first successful ImageNet-1K-scale, distillation-free compression at state-of-the-art ratios, and shows that 60% of ImageNet plus 20% of Alpaca instruction data trains vision and language models with negligible accuracy loss across classification, segmentation, detection and instruction tuning.

DataDAM · ICCV 2023 · Distribution and feature matchingnotable

Matches spatial attention maps computed at multiple layers of randomly initialized networks between real and synthetic batches, instead of matching gradients or single-layer features, reporting up to 6.5-point gains on CIFAR-100 and 4.1-point gains on ImageNet-1K over prior methods.

DATM · ICLR 2024 · Trajectory matchingcore

Aligns the training-stage difficulty of the expert-trajectory segments matched during trajectory-matching distillation with the size of the synthetic set — early, easy-pattern segments for small budgets and late, hard-pattern segments for larger ones — letting trajectory matching keep improving as the synthetic set grows and reporting the first near-lossless dataset-distillation results.

KRR-ST · ICLR 2024 · Kernel and closed-form inner solverscore

KRR-ST distills an unlabeled dataset for self-supervised pre-training by replacing the naive bilevel SSL objective, whose synthetic-sample gradient is biased by augmentation/masking randomness, with an inner objective that regresses learnable target feature representations (MSE, no randomness) and an outer objective that matches the resulting model's representations to a self-supervised target model trained on the full dataset. With the feature extractor fixed, the final linear head is obtained in closed form via kernel ridge regression, and the method is validated on several transfer-learning applications.

GDEM · ICML 2024 · Distribution and feature matchingnotable

Proposes GDEM for graph distillation, which aligns the eigenbasis and node features of the synthetic and real graphs directly and replicates the real graph's spectrum, instead of relying on GNN-derived gradients, representations or trajectories that bias the synthetic spectrum and tie the result to one GNN architecture; adds a discrimination constraint to balance effectiveness and generalization, reporting state-of-the-art results with strong cross-architecture generalization and a theoretical guarantee that the synthetic graphs are restricted spectral approximations of the real ones.

SGDD · NeurIPS 2023notable

Analyzes graph condensation methods in the spectral domain and finds that prior approaches induce a substantial Laplacian Energy Distribution (LED) shift between synthetic and real graphs, hurting cross-architecture generalization and tasks like anomaly detection and link prediction; proposes SGDD, which broadcasts the original graph's structural information into synthetic-graph generation to reduce this shift, reporting 98.6% of full-graph accuracy on YelpChi at a 1000x size reduction and 17.6-31.4% smaller LED shifts across nine datasets.

Mirage · ICLR 2024 · Dataset quantizationnotable

Mirage condenses a graph-classification dataset by decomposing every input graph into the multiset of computation trees a message-passing GNN would build on it, then compressing the (typically skewed) frequency distribution of these computation trees into a concise summary, rather than emulating gradient flows on the full dataset. Because it needs no full-dataset training and no target architecture during distillation, Mirage is unsupervised and architecture-agnostic, and the paper reports better generalization accuracy, compression and distillation efficiency than prior graph distillation baselines.

RaT-BPTT · ICLR 2024 · Meta-learning through unrolled trainingnotable

Revisits backpropagation-through-time dataset distillation directly as a bi-level optimization problem and introduces Random Truncated BPTT, combining gradient truncation with a random window over the unrolled steps to stabilize gradients and cover long-term dependencies, setting new state-of-the-art results across standard benchmarks and revealing pronounced intercorrelation among the resulting distilled points that a boosting mechanism can exploit to build near-optimal subsets at multiple budgets.

FreD · NeurIPS 2023 · Synthetic-set parameterizationnotable

Parameterizes each synthetic instance in the frequency domain instead of pixel space, selecting a subset of frequency dimensions by explained variance so that a limited budget is spent only on the most informative components; theoretically and empirically more budget-efficient than prior parameterizations, and composes with existing distillation methods for consistent gains across benchmarks.

Minimax Diffusion · CVPR 2024 · Diffusion-based synthesiscore

Fine-tunes a diffusion model with additional minimax criteria that push generated images toward representativeness and diversity, modeled theoretically as hierarchical diffusion control, replacing the sample-wise iterative optimization of prior distillation methods; under 100 IPC on ImageNet-Woof, needs under one-twentieth the distillation time of prior methods while achieving better performance.

Rethinking Backdoor Attacks on Dataset Distillation: A Kernel Method Perspective · ICLR 2024 · Kernel and closed-form inner solversnotable

Counters the view that kernel-based dataset distillation counteracts backdoor risk, deriving two new theory-driven trigger-generation methods specialized for kernel-based distillation from a theoretical analysis of backdoor attacks under kernel methods. Reports its optimization-based trigger design produces resilient backdoor attacks that evade conventional backdoor detection and mitigation methods.

G-VBSM · CVPR 2024 · Decoupled teacher-driven synthesisnotable

Argues that SRe2L's single-backbone, single-layer 'local-match-global' matching limits generalization, and generalizes it to match densities across many backbones, layers and statistics at once; the first algorithm to perform well at both small and large scale, reporting 38.7% on CIFAR-100 (IPC 10), 47.6% on Tiny-ImageNet (IPC 50), and 31.4% on full 224x224 ImageNet-1K (IPC 10), beating prior state of the art by 3.9-10.1 points.

WMDD · ICCV 2025 · Distribution and feature matchingnotable

Computes the Wasserstein barycenter of pretrained-classifier features from the real data and optimizes synthetic images to align with this barycenter in feature space, combined with per-class batch-norm statistics to preserve intra-class variation, reporting state-of-the-art results across several high-resolution dataset-distillation benchmarks while keeping the efficiency of distribution-matching approaches.

CDA · TMLR 2024 · Decoupled teacher-driven synthesisnotable

Extends SRe2L's decoupled synthesis with curriculum data augmentation that refines the gradient update from global to local over the synthesis iterations, since the first few update steps determine the final image outline; reports 63.2% top-1 on ImageNet-1K at IPC 50 and 36.1% on ImageNet-21K at IPC 20, the first reported ImageNet-21K distillation at 224x224 resolution, beating SRe2L, TESLA and MTT by more than 4 points.

Static-dynamic video DD · CVPR 2024 · Gradient matchingcore

This paper gives the first systematic study of video distillation, finding that the temporal dimension of synthetic clips is usually not well learned and contributes little, and responds by disentangling static and dynamic information: a static learning stage distills videos into still images via gradient matching, and a dynamic fine-tuning stage compensates motion with a separately stored, learnable dynamic memory block rather than distilling every frame. The paper reports state-of-the-art results on video datasets at multiple scales with a notably smaller memory storage budget than prior methods.

FedDG · ECCV 2024 · Optimization in a generative latent spacenotable

Moves dataset distillation for federated learning from the client to the server: instead of clients distilling and uploading synthetic data, the server uses prior knowledge from pretrained deep generative models' latents to synthesize data representations from heterogeneous client model architectures, letting local devices train smaller surrogate models while the server trains a larger global model on a multi-modal distribution. Reports up to 40% accuracy improvement over non-distillation FL and 18% over existing dataset-distillation FL methods under high heterogeneity, plus faster convergence, with a theoretical argument that the process asymptotically resembles centralized training.

RDED · CVPR 2024 · Decoupled teacher-driven synthesiscore

Identifies realism, diversity and efficiency as the properties large-scale dataset distillation needs and shows existing methods trade off against them; RDED selects and stitches realistic real-image crops scored by a teacher rather than optimizing synthetic pixels, distilling all of ImageNet-1K to 10 images per class in 7 minutes and reaching 42% top-1 with ResNet-18 on a single RTX-4090, versus 21% in 6 hours for the prior state of the art.

M3D · AAAI 2024 · Distribution and feature matchingnotable

Argues that prior distribution-matching methods only align first-moment (mean) statistics and so under-match the real and synthetic feature distributions, and proposes minimizing the maximum mean discrepancy in a reproducing kernel Hilbert space to align all orders of moments, surpassing the optimization-oriented method IDC on high-resolution ImageNet.

GSDD · AAAI 2024 · Optimization in a generative latent spacenotable

GSDD distills a low-resolution/high-resolution image-pair dataset for super-resolution by optimizing codes in the latent space of a pretrained GAN via GAN-inversion, rather than selecting or synthesizing pixels directly, so the stored artifact is a small set of latent codes plus the frozen generator. The paper reports super-resolution performance comparable to prior state-of-the-art distillation methods with about an 8x increase in training efficiency and roughly 93.2% less storage, and shows generalization to real-world degraded images.

AVDD · TMLR 2024 · Distribution and feature matchingcore

Extends distribution matching to audio-visual data, first matching visual-only and audio-only feature distributions with randomly initialized networks separately, then adding a joint matching loss that implicitly cross-matches real audio against synthetic visual features (and vice versa) and a modality-gap matching loss that aligns the audio-visual gap between real and synthetic data, plus herding-based initialization; on VGGS-10K at 10 images-per-class it reports 54.99% recognition accuracy versus 43.85% for plain distribution matching and 68.24% for the full dataset (Table 3), and raises audio-to-visual retrieval Recall@1 from 8.66% (DM) to 19.33% (Table 5).

Group Distributionally Robust Dataset Distillation with Risk Minimization · ICLR 2025 · Optimization and training recipesnotable

Argues that matching training-set convergence, the usual DD criterion, does not guarantee that a model trained on the synthetic set generalizes to under-represented subgroups, since coverage of low-density regions matters more than training error there; proposes combining clustering with minimization of a distributionally-robust risk measure on the loss during distillation. Reports a theoretical rationale and improved generalization and robustness across subgroups compared to standard DD.

GEOM · ICML 2024 · Trajectory matchingnotable

Identifies that the prior state-of-the-art graph trajectory-matching method supplies biased and restricted supervision signals from the original graph, limiting both the scale and efficacy of the condensed graph, and proposes GEOM, which trains expert trajectories with a curriculum-learning strategy for more diverse supervision and transfers that information into the condensed graph with an expanding-window matching scheme; reports state-of-the-art results toward lossless graph condensation across several datasets.

LD3M · NeurIPS 2025 · Optimization in a generative latent spacenotable

The first method to learn gradient-based distilled latents and class embeddings end-to-end through a pretrained latent diffusion model, rather than only sampling from a fixed prior or relying on GANs/autoencoders; a linearly decaying skip connection injected at every reverse step preserves gradient signal across dozens of timesteps without fine-tuning the diffusion weights, improving downstream accuracy by up to 4.8 points (1 IPC) and 4.2 points (10 IPC) over the prior state of the art on ImageNet subsets at 128x128 and 256x256.

GUARD · AAAI 2025 · Optimization and training recipesnotable

Adds curvature regularization to the distillation process so that models trained on the resulting distilled set are adversarially robust without the cost of full adversarial training. Reports outperforming standard adversarial training on both accuracy and robustness at lower computational overhead, and withstanding various adversarial attacks.

EDC · NeurIPS 2024 · Decoupled teacher-driven synthesiscore

Systematically explores the design space of decoupled dataset condensation across scales, contributing soft category-aware matching and a tuned learning-rate schedule grounded in empirical and theoretical analysis; establishes a benchmark for small- and large-scale condensation, reporting 48.6% top-1 on ImageNet-1K with ResNet-18 at IPC 10 (0.78% compression ratio), ahead of SRe2L, G-VBSM and RDED by 27.3, 17.2 and 6.6 points respectively.

Distilled Datamodel with Reverse Gradient Matching · CVPR 2024 · Gradient matchingnotable

Approximates each training sample's influence on a pretrained model without leave-one-out retraining, by first distilling the training set into a synset via reverse gradient matching (offline) and then using that synset to speed up leave-one-out evaluation and compute an attribution matrix (online). Reports training-data-attribution and data-quality evaluations comparable to direct retraining at substantially lower cost.

FedAF · CVPR 2024 · Distribution and feature matchingnotable

FedAF replaces iterative model aggregation with a collaborative distribution-matching condensation step, where clients jointly learn condensed data and soft labels from peer knowledge that the server then trains the global model on directly. It reports faster convergence and higher global-model accuracy than prior one-shot and iterative FL baselines under label- and feature-skew heterogeneity.

DeSA · ICML 2024 · Distribution and feature matchingnotable

In serverless decentralized federated learning, DeSA synthesizes global anchor data from each client's raw data distribution and uses it for two local regularizers: aligning each client's latent embeddings to the anchors, and a knowledge-distillation loss that lets clients learn from one another, grounded in domain-adaptation and KD theory. Reports improved inter- and intra-domain accuracy across clients with diverse data and model heterogeneity.

ReLA · NeurIPS 2024 · Label distillation and soft labelsnotable

ReLA (Representation Learning Accelerator) argues that a publicly available, task- and architecture-agnostic "prior model" can be used to produce efficient training signal that is transportable across tasks and architectures, substituting learned representation targets for the labels/pipeline a representation-learning method would otherwise need. The paper reports that using a CIFAR-10-pretrained ResNet-18 as the prior model to inform ResNet-50 training on ImageNet-1K cuts computational cost by 50% while matching the accuracy of the original BYOL recipe trained at full cost.

GIFT · ICLR 2025 · Label distillation and soft labelsnotable

Shows that models trained on distilled data are highly sensitive to the loss function used for soft labels, and introduces GIFT, a plug-and-play soft-label refinement plus cosine-similarity loss that extracts more of the label information already present; reports consistent gains across distillation methods and scales, including a 30.8% cross-optimizer generalization improvement over RDED on ImageNet-1K at IPC=10, at no extra compute cost.

SelMatch · ICML 2024 · Trajectory matchingnotable

Finds that trajectory-matching distillation fails to close the coverage gap between easy and hard test samples as images-per-class grows, and addresses this by initializing the synthetic set with selection-based (coreset) samples and only partially updating them through trajectory matching, tuning the selection-to-update ratio to the target IPC; consistently outperforms leading selection-only and distillation-only methods on CIFAR-10/100 and Tiny-ImageNet from 5% to 30% subset ratios.

What is DD learning? · ICML 2024notable

Finds that distilled data cannot substitute for real data outside the standard dataset-distillation evaluation setting, that the distillation process retains high task performance mainly by compressing information about real models' early training dynamics, and provides an interpretation framework showing individual distilled data points carry meaningful semantic information.

LoRS · ICML 2024 · Trajectory matchingcore

LoRS distills a ground-truth image-text similarity matrix alongside the synthetic pairs, instead of assuming one caption strictly matches one image, and stores that similarity structure efficiently via low-rank factorization. The paper reports significant improvements over prior vision-language distillation algorithms and proposes LoRS as a foundational synthetic-data setup for image-text dataset distillation.

H-GLaD · CVPR 2025 · Optimization in a generative latent spacenotable

Extends GLaD-style generative-prior distillation by systematically exploring hierarchical feature layers within a pretrained GAN, rather than a single fixed latent space, since different layers offer different informative guidance for distillation; also introduces a class-relevant feature-distance metric that lowers the cost of evaluating synthetic sets. Matches or surpasses diffusion-based generative distillation at extreme compression (IPC=1, IPC=10) at comparable time cost.

A label is worth a thousand images · NeurIPS 2024 · Label distillation and soft labelslandmark

Ablates the many disparate dataset-distillation methods and finds that the single largest common factor in their reported performance is the use of soft (probabilistic) labels rather than their specific synthesis techniques; shows the soft labels must carry structured information to help, provides empirical scaling laws relating soft-label effectiveness to images-per-class, and establishes an empirical Pareto frontier for data-efficient learning, challenging the attribution of gains to image-synthesis method alone.

GC-Bench · NeurIPS 2024notable

Introduces GC-Bench, a benchmark that systematically evaluates 12 graph-condensation methods on node- and graph-level tasks across 12 datasets along effectiveness, transferability and complexity, and releases a reusable library for training and evaluating graph condensation methods.

D4M · CVPR 2024 · Diffusion-based synthesislandmark

Argues that architecture-dependent bi-level/uni-level matching objectives are the reason cross-architecture generalization degrades, and instead condenses the dataset into a latent diffusion model's space, incorporating label information into per-category prototypes so the same distilled representation serves any evaluation architecture without regenerating separate datasets; reports superior performance and robust generalization across most benchmarks compared with prior state-of-the-art methods.

PAD · arXiv 2024 · Trajectory matchingnotable

Identifies that trajectory-matching distillation extracts and embeds 'misaligned' information from its agent (expert) model, and fixes this by pruning the target dataset to match the compression ratio before computing expert trajectories and by using only the agent model's deep layers for the matching loss, avoiding excess low-level information; reports state-of-the-art results built on trajectory matching.

INFER · ICLR 2025 · Decoupled teacher-driven synthesisnotable

Breaks the one-hot, per-class synthesis barrier of prior distillation with a Universal Feature Compensator (UFC) that enhances feature integration across classes and can generate multiple additional synthetic instances from a single UFC input, plus label interpolation across classes that shrinks stored soft-label size toward zero; reports outperforming SRe2L by 34.5% at IPC=50 on ImageNet-1K with ResNet-18.

HeLlO · ICCV 2025 · Label distillation and soft labelsnotable

Replaces stored per-crop soft labels with an image-to-label projector distilled from a foundation model (e.g. CLIP) via LoRA-style fine-tuning, so synthetic labels are generated online from synthetic images instead of stored, plus an image-optimization step that narrows the gap between original and distilled label generators; reports comparable accuracy to state-of-the-art distillation with only about 0.003% of the storage a full soft-label set would need on large-scale datasets.

LTDD · CVPR 2025 · Trajectory matchingnotable

Identifies why trajectory-matching distillation fails on long-tailed data — biased expert trajectories from imbalanced training propagate into biased synthetic sets, and experts trained on tail classes give poor guidance and soft labels — then proposes Distribution-agnostic Matching, which keeps the student trajectory away from the biased expert path, and Expert Decoupling, which matches backbone and classifier separately to improve tail-class guidance and soft-label quality. Reported as the first effective method for long-tailed dataset distillation.

DWA · NeurIPS 2024 · Decoupled teacher-driven synthesisnotable

Provides a theoretical and empirical analysis of diversity within synthesized datasets and argues that the parallelizable but isolated synthesis common to decoupled methods needs an explicit diversity mechanism; introduces dynamic, directed weight adjustment during synthesis so each synthetic instance mirrors a large, varying subset of the real data, improving performance on CIFAR, Tiny-ImageNet and ImageNet-1K at minimal extra computational cost.

MKDT · ICLR 2025 · Trajectory matchingnotable

MKDT shows that applying supervised trajectory matching directly to a self-supervised objective fails because the SSL gradient has high variance, and instead trains a small student to match the representations of a larger SSL-trained teacher (knowledge distillation), then distills a synthetic dataset by matching the training trajectories of these lower-variance student models. The paper reports up to 13 percentage points higher downstream accuracy than prior work when labeled data is limited, across a variety of transfer tasks, as the first effective dataset distillation method for self-supervised pre-training.

LPLD · NeurIPS 2024 · Label distillation and soft labelsnotable

Traces the need for large stored soft-label sets in ImageNet-scale distillation to high within-class similarity, itself caused by batch-normalization matching that mixes samples from different classes in one batch; batching within classes during synthesis instead raises within-class diversity, letting soft labels be compressed with simple random pruning; compresses required soft labels from 113GB to 2.8GB (40x) at IPC=200 on ImageNet-1K with a 2.6-point accuracy gain.

Offline Behavior Distillation · NeurIPS 2024 · Meta-learning through unrolled trainingnotable

This paper formulates offline behavior distillation (OBD), synthesizing a small set of expert-like behavioral data from suboptimal offline RL data so that a policy can be learned quickly, and shows that naive decision-difference objectives are hard to optimize through the intractable bi-level problem and give only quadratic-discount-complexity guarantees. It introduces action-value weighted PBC (Av-PBC), proving an equivalence between policy performance and the action-value weighted decision difference, and reports faster convergence, better distillation performance and robust cross-architecture/optimizer generalization than the naive objectives on multiple D4RL datasets.

FairDD · NeurIPS 2025 · Optimization and training recipesnotable

Shows that dataset distillation does not fix, and typically worsens, unfairness toward protected-attribute minority groups because matching-based objectives align synthetic data to the whole distribution, which majority groups dominate; FairDD instead synchronously matches synthetic data to each protected-attribute group separately, without changing the underlying matching architecture. Reported to significantly improve fairness with a favorable accuracy trade-off across both distribution-matching and gradient-matching DD methods.

DELT · CVPR 2025 · Decoupled teacher-driven synthesisnotable

Diagnoses a diversity problem in batch-to-global (decoupled) matching methods such as SRe2L, G-VBSM, WMDD and CDA, where images optimized independently under the same reused global supervision end up homogeneous; DELT partitions the images-per-class budget into subtasks distilled at distinct early and late optimization phases, then generalizes back to the full task, reporting 2-5% average accuracy gains, 5%+ higher per-class diversity, and up to 39.3% less synthesis time on CIFAR, Tiny-ImageNet, ImageNet-1K and its sub-datasets.

Provable KRR DD · NeurIPS 2024 · Kernel and closed-form inner solversnotable

Proves that for kernel ridge regression one data point per class is necessary and sufficient to exactly recover the original model in many settings, derives necessary and sufficient conditions for exact recovery with linear and surjective-feature-map kernels, and shows k+1 points suffice for deep linear networks with k classes; the resulting closed-form construction outperforms KIP while reported up to 15,840x faster on CIFAR-100.

DDM · WACV 2025 · Distribution and feature matchingnotable

Decomposes the dataset distribution that distribution matching targets into content and style, and identifies two shortcomings of the plain distribution-matching objective: a style mismatch between real and condensed data, and limited intra-class diversity in the condensed set. Matches per-layer feature-map statistical moments as a style term and maximizes intra-class KL divergence among synthetic samples as a content-diversity term, reporting accuracy gains of up to 4.1% on CIFAR-10, 4.2% on CIFAR-100, 4.3% on Tiny-ImageNet, 2.0% on ImageNet-1K, 3.3% on ImageWoof, 2.5% on ImageNette, and 5.5% in continual-learning accuracy over the distribution-matching baseline.

Adaptive Dataset Quantization · AAAI 2025 · Dataset quantizationnotable

Replaces Dataset Quantization's uniform bin sampling with an adaptive strategy that scores each generated bin's representativeness (texture level), diversity (contrastive-learning based) and importance, then samples accordingly; reports an average 3% accuracy gain over DQ across benchmarks with better cross-architecture generalization.

FedVCK · AAAI 2025 · Distribution and feature matchingnotable

FedVCK condenses each client's data with latent distribution constraints to improve knowledge quality, and each round selectively condenses only knowledge the current global model has not yet assimilated, cutting redundant communication; on the server, relational supervised contrastive learning gives extra supervision for the global update. Reports outperforming state-of-the-art federated learning methods on medical tasks under non-IID data and limited communication budgets.

UniDD · CVPR 2025 · Diffusion-based synthesisnotable

UniDD extends dataset distillation beyond classification to detection and segmentation with a task-driven diffusion model, first mining task-relevant information by training task-specific proxy models (Universal Task Knowledge Mining), then using those proxies to guide a diffusion process that generates task-specific synthetic images with localized, region-level control rather than the global, single-class updates of prior generation methods (Universal Task-Driven Diffusion). The paper reports consistently outperforming state-of-the-art methods on ImageNet-1K, Pascal VOC and MS COCO across classification, detection and segmentation.

Point Cloud DD · ICML 2025 · The generator as the distilled artifactnotable

This paper adapts dataset distillation to unstructured 3D point clouds by theoretically showing that matching rotation-invariant features between real and synthetic data matters for 3D distillation, introducing a plug-and-play point cloud rotator that aligns point clouds to a canonical orientation, and replacing fixed-size synthetic data with a point-wise generator that produces point clouds at multiple resolutions instead of optimizing one fixed set directly.

IGD · ICLR 2025 · Diffusion-based synthesisnotable

Frames dataset distillation as controlled diffusion generation guided by a trajectory influence function that links generated samples to downstream training effectiveness, requiring no diffusion retraining; reports 60.3% top-1 at IPC=50 on ImageNet-1K, state-of-the-art among diffusion-based methods.

ROME · ICML 2025 · Optimization and training recipesnotable

Applies the Information Bottleneck principle to distillation robustness with two loss terms — a performance-aligned term to preserve clean accuracy and a robustness-aligned term that aligns feature distributions between synthetic and adversarially perturbed images — and introduces an Improved Robustness Ratio (I-RR) metric to evaluate DD robustness more precisely. Reports up to roughly 40% and 35% I-RR improvements over existing DD methods under white-box and black-box attacks respectively on CIFAR-10/100.

RDC · ICCV 2025 · Optimization and training recipesnotable

RDC is an end-to-end dataset-condensation method that mitigates label noise without a separate noise-reduction preprocessing step, using a "golden MixUp contrast" that applies contrastive learning tailored for robust condensation: synthetic samples sharpen class boundaries and reduce noisy representations, while an augmentation strategy pulls clean samples out of the noisy training data to enrich the synthetic images with real-data diversity. Reports outperforming other condensation approaches on CIFAR-10/100 across symmetric, asymmetric and real-world noise.

Pushforward quantization · ICLR 2026 · Diffusion-based synthesisnotable

Shows that decoupled/disentangled dataset-distillation methods with an encoder-decoder structure are instances of classical optimal quantization and Wasserstein-barycenter problems, and uses this connection to propose clustering in a diffusion model's latent space to select prototype points, reporting better performance and cross-model generalization than D4M on ImageNet-1K, plus state-of-the-art results using the distilled noise initializations with a stronger diffusion transformer.

Dark Distillation: Backdooring Distilled Datasets without Accessing Raw Data · AAAI 2026notable

Shows that an attacker who intercepts an already-distilled dataset in transit, with no access to the raw data it came from, can still inject a working backdoor by reconstructing per-class conceptual archetypes from a model trained on the distilled set and updating those archetypes with a hybrid loss that adds backdoor information while preserving the original optimization trajectory. Reports that distilled datasets remain highly vulnerable to this raw-data-free attack across multiple raw datasets and distillation methods.

Liu & Du survey · arXiv 2025core

This survey focuses on post-2023 advances that scale dataset distillation to ImageNet-1K and ImageNet-21K, organizing progress into trajectory matching, gradient matching, distribution matching, scalable generative approaches and decoupled optimization, and highlights SRe2L, soft-label strategies and lossless-distillation techniques as the field's key breakthroughs. It also reviews robustness to adversarial and backdoor attacks, non-IID data, and emerging applications in video, audio, multimodal learning, medical imaging and scientific computing.

GRADMM · ICML 2025 · Gradient matchingnotable

Proposes GRADMM, which uses ADMM to optimize the embeddings of synthetic text examples so their gradient matches the (noisy) gradient of real training or validation data, then decodes the embeddings into low-perplexity token sequences, giving convergence, performance and privacy guarantees for fine-tuning LLMs on human-readable synthetic text; reports effectiveness across several classification tasks.

NCFM · CVPR 2025 · Distribution and feature matchingnotable

Reformulates distribution matching as a minmax problem using a Neural Characteristic Function Discrepancy, where a network optimizes the sampling of characteristic-function frequency arguments to maximize the measured discrepancy while the synthetic data is optimized to minimize it, aligning phase and amplitude of real and synthetic features in the complex plane; reports a 20.5-point gain on ImageSquawk, over 300x less GPU memory, and the first lossless CIFAR-100 compression on a single 2080 Ti.

DDiF · ICLR 2025 · Synthetic-set parameterizationcore

Parameterizes each synthetic instance as a neural field mapping coordinates to output quantities instead of storing pixels directly, theoretically shown to be more expressive than prior parameterizations at the same per-instance budget; reports superior performance on several benchmarks and extends beyond images to video, audio and 3D voxel data.

Condensing Action Segmentation Datasets via Generative Network Inversion · CVPR 2025 · Optimization in a generative latent spacenotable

Condenses procedural video datasets for temporal action segmentation into compact latent codes using a generative prior learned from the dataset and network inversion, reducing storage across both the temporal and channel dimensions, plus a diversity-driven sampling step that selects representative action sequences to cut video-wise redundancy; on the Breakfast dataset it reports over 500x storage reduction while retaining 83% of full-dataset segmentation performance, and improves downstream incremental-learning performance.

SADM · NeurIPS 2025 · Distribution and feature matchingnotable

Extends distribution matching to 3D point-cloud dataset distillation, jointly optimizing synthetic point geometry and orientation; introduces a Semantically Aligned Distribution Matching loss computed on per-channel sorted features to correct the semantic misalignment caused by unordered point indexing, and jointly learns rotation angles to correct orientation variation, reporting consistent gains over prior distillation methods with strong cross-architecture generalization.

UniDetox · ICLR 2025notable

UniDetox distills a small synthetic text dataset that carries detoxifying signal, generated from one source model (GPT-2) via contrastive decoding, and fine-tunes any target LLM on it to reduce toxicity without per-model hyperparameter tuning. The paper reports that text distilled from GPT-2 successfully detoxifies larger, unrelated models (OPT, Falcon, LLaMA-2) with a single shared hyperparameter configuration, and that the distilled text also reduces politically biased content.

RepBlend · NeurIPS 2025 · Trajectory matchingnotable

RepBlend identifies "modality collapse" in multimodal dataset distillation, over-concentrated intra-modal representations and an enlarged cross-modal gap, as a conflict between dataset distillation's over-compression and contrastive cross-modal supervision, and addresses it by weakening overdominant cross-modal supervision through representation blending to restore intra-modal diversity, plus symmetric projection trajectory matching that synchronizes optimization dynamics across modality-specific projection heads to correct asymmetric supervision. On Flickr30K and MS-COCO the paper reports consistent gains over prior multimodal distillation methods (e.g. +9.4 IR@10, +6.3 TR@10 at the 100-pair setting) with up to 6.7x faster distillation.

DD-Ranking · arXiv 2025core

Shows that reported accuracy gains from decoupled dataset-distillation pipelines largely come from soft labels and stronger post-hoc augmentation rather than the intrinsic quality of the distilled images themselves, since even randomly sampled images can match state-of-the-art distilled sets under the same labeling and augmentation recipe, and proposes DD-Ranking, a unified evaluation framework and set of metrics that isolate the images' own contribution.

D3HR · ICML 2025 · Diffusion-based synthesisnotable

Identifies inaccurate distribution matching, deviation from the random-noise domain, and separate per-image sampling as flaws in diffusion-based distillation, and fixes them with DDIM inversion that maps the full dataset's latents into a high-normality Gaussian domain, plus an efficient sampling scheme that aligns representative latents to that domain; reports higher cross-architecture accuracy than state-of-the-art diffusion-based baselines.

MGD³ · ICML 2025 · Diffusion-based synthesisnotable

Uses a pretrained diffusion model with no distillation-loss fine-tuning, addressing diversity in three stages: Mode Discovery finds distinct data modes, Mode Guidance steers sampling toward intra-class diversity, and Stop Guidance halts guidance early enough to avoid artifacts; reports accuracy gains of 4.4%, 2.9%, 1.6% and 1.6% on ImageNette, ImageIDC, ImageNet-100 and ImageNet-1K over the state of the art, at much lower computational cost than fine-tuning-based methods.

PRISM · CVPR 2026 · Gradient matchingnotable

PRISM treats a condensed video as one fully coupled spatiotemporal structure rather than disentangling it into separate static and dynamic components, starting from minimal temporal anchors and progressively inserting key-frames only where linear interpolation between them fails to capture non-linear motion, guided by gradient misalignments between synthetic and real data. The paper reports competitive accuracy with state-of-the-art storage efficiency on standard video-distillation benchmarks by allocating representational capacity only where the motion needs it.

HDD · NeurIPS 2025 · Distribution and feature matchingnotable

Embeds pretrained-network features into Lorentz hyperbolic space and matches the hyperbolic (geodesic) distance between synthetic and real class centroids instead of Euclidean distribution matching, explicitly encoding the data's hierarchical structure into the distilled set; pruning in hyperbolic space is shown to retain model performance with only 20% of the distilled set.

OD3 · ICLR 2026 · Decoupled teacher-driven synthesisnotable

OD3 distills an object-detection dataset without any bi-level optimization, in two stages it iteratively places real object instances at suitable locations in synthesized images (candidate selection), then screens out low-confidence placements with a pretrained observer model (candidate screening). On MS COCO and PASCAL VOC at compression ratios from 0.25% to 5%, the paper reports new state-of-the-art results, surpassing the prior detection-distillation method by more than 14 points of mAP50 at a 1.0% compression ratio.

CaO2 · ICCV 2025 · Diffusion-based synthesisnotable

Identifies objective inconsistency (the distillation process diverges from the evaluation objective) and condition inconsistency (mismatches between generated images and their conditions) in diffusion-based distillation, and fixes both with a two-stage Condition-aware Optimization with Objective-guided Sampling: probability-informed sample selection, then latent refinement for higher conditional likelihood; reports a 2.3% average accuracy gain over the best baselines on ImageNet and its subsets.

VLCP · ICCV 2025 · Diffusion-based synthesisnotable

Adds text prototypes -- derived from descriptions an open-source large language model generates for each class -- alongside image prototypes to guide diffusion-based synthesis, so distilled images incorporate semantic and contextual information image-only distillation overlooks; works even on datasets with no pre-existing text descriptions, and reports state-of-the-art validation performance with logically coherent images that better preserve target objects.

FADRM · NeurIPS 2025 · Decoupled teacher-driven synthesisnotable

Introduces data-level residual (skip) connections -- Data Residual Matching -- to counter information vanishing during pixel-space synthesis, balancing newly optimized signal against core information already present in the raw data, while optimization-level refinements cut training time and peak GPU memory by half; reports 47.7% (single-model) and 50.0% (multi-model) top-1 on ImageNet-1K with ResNet-18 at 0.8% compression, ahead of RDED by 5.7 points and of EDC/CV-DD by 1.4/4.0 points.

D2C · CVPR 2026 · Coreset selectionnotable

Applies a condensation-based perspective to training diffusion models efficiently rather than to classification: a Select phase scores real images with a diffusion difficulty score combined with interval sampling to pick a compact, informative subset, and an Attach phase enriches each selected image with additional semantic and visual conditioning signals. Reports an FID of 4.3 after 40k steps on ImageNet 256x256 with SiT-XL/2 using only 0.8% of the training images, about 233x and 100x faster than training vanilla SiT-XL/2 and SiT-XL/2 + REPA respectively.

Rate-utility DD · ICCV 2025 · Synthetic-set parameterizationnotable

Frames dataset distillation as a joint rate-utility optimization: synthetic samples are optimizable latent codes decoded by extremely lightweight networks, the Shannon entropy of the quantized latents is the rate term, and any existing distillation loss serves as the utility term, traded off via a Lagrange multiplier; introduces bits-per-class (bpc), a storage metric accounting for sample, label and decoder-parameter costs, and reports up to 170x greater compression than standard distillation at comparable accuracy on CIFAR-10, CIFAR-100 and ImageNet-128.

Dosser · ICCV 2025 · Distribution and feature matchingnotable

Identifies that prior differentially-private dataset distillation wastes privacy budget because sampling and optimization are synchronized and training signals come from noisy, randomly-initialized networks; decouples sampling from optimization for better convergence and matches in an informative subspace to reduce the impact of DP noise. Reports a 10.0% accuracy improvement at 50 images per class on CIFAR-10 and an 8.3% gain using only one-fifth the distilled-set size of prior state-of-the-art private DD methods.

EDGE · NeurIPS 2025 · Diffusion-based synthesiscore

Proposes EDGE, which fine-tunes a pretrained Stable Diffusion v1.5 model for image-text dataset distillation by replacing its denoising loss with a bidirectional contrastive loss that aligns generated image and text embeddings and a minimax diversity loss that pushes different pairs' embeddings apart, then generates synthetic pairs by sampling from the fine-tuned model; evaluated with a pretrained NFNet image encoder and a frozen BERT-base text encoder, it reports on Flickr30K at 500 pairs 21.0/30.5% image-retrieval Recall@5/10 and 35.6/47.5% text-retrieval Recall@5/10 (Table 1), running about 18x faster than the trajectory-matching baseline LoRS.

RD3 · ICLR 2026notable

RD3 systematically studies how post-evaluation choices (augmentation, epoch-wise soft labels, and other protocol details introduced after decoupled distillation methods such as SRe2L) affect reported test accuracy, and finds that much of the performance variation between existing decoupled-distillation methods reflects inconsistent evaluation protocols rather than differences in the intrinsic quality of the synthetic data. The paper proposes a standardized benchmark and evaluation protocol, and identifies general strategies that improve distilled-dataset effectiveness across settings.

DAP · ICLR 2026 · Diffusion-based synthesisnotable

Formalizes the representativeness already implicit in a pretrained diffusion model by quantifying feature-space similarity between synthetic and real data with a Mercer kernel, then uses it as training-free guidance for the reverse diffusion process instead of adding external constraints; reports state-of-the-art fidelity and cross-architecture generalization on ImageNet-1K and its subsets.

CovMatch · NeurIPS 2025 · Distribution and feature matchingnotable

Proposes CovMatch for image-text dataset distillation, which fixes the encoders at each distillation step and matches the cross-covariance between real and synthetic image and text feature matrices, together with per-modality feature-distribution regularization, instead of unrolling trajectories; unlike prior methods that freeze the text tower, CovMatch keeps a BERT-base text encoder trainable end to end (only its embedding layer is frozen) alongside a pretrained NFNet image encoder. On Flickr30K at 500 pairs it reports 38.4% mean recall (Table 2), a 6.8-point absolute gain over the LoRS baseline (31.6%), and 19.6% mean recall on COCO at 500 pairs, a 6.1-point gain over LoRS.

ImageBindDC · AAAI 2026 · Distribution and feature matchingnotable

ImageBindDC condenses omnimodal data inside ImageBind's shared embedding space using a Characteristic Function loss that performs exact infinite-moment matching in the Fourier domain, enforcing uni-modal, cross-modal (real-synthetic pair) and joint-modal distributional consistency simultaneously rather than matching each modality separately. On NYU-v2 the paper reports lossless performance versus full-data training at 5 condensed datapoints per class, an 8.2 percentage-point absolute improvement over the previous best method, and over 4x less condensation time.

DP-GENG · AAAI 2026 · Distribution and feature matchingnotable

Initializes the distilled dataset with DP-generated data for realism, then refines it under a small privacy budget with a DP-feature-matching (distribution-matching) objective and aligns examples to their class distribution with an expert model, alongside a privacy-budget allocation strategy with a theoretical DP guarantee. Reports outperforming prior differentially-private dataset-distillation methods in both utility and robustness to membership-inference attacks.

Linear Gradient Matching · NeurIPS 2025 · Gradient matchingnotable

Linear Gradient Matching distills a dataset for training linear probes on top of a frozen, pretrained self-supervised vision model, optimizing synthetic images so that gradients they induce in the linear classifier match those from real data. The paper reports that the resulting synthetic sets outperform all real-image baselines, generalize across pretrained backbones (e.g. a set distilled via DINO trains a competitive linear CLIP probe), and are effective for fine-grained classification and for interpretability probes such as measuring embedding-space similarity between models.

ADSA · NeurIPS 2025 · Label distillation and soft labelsnotable

Derives an imbalance-aware generalization bound for models trained on distilled data and traces long-tailed dataset distillation's performance loss to soft-label bias entangled from two sources, the distillation model and the distilled images themselves, identified through systematic perturbation of imbalance levels; proposes ADSA, a lightweight Adaptive Soft-label Alignment module that calibrates this bias and plugs into existing distillation pipelines. Reports up to 11.8% tail-class accuracy improvement and 41.4% overall accuracy on ImageNet-1k-LT with EDC at IPC=50.

RLDD · AAAI 2026 · Decoupled teacher-driven synthesisnotable

Moves long-tailed dataset distillation away from trajectory matching to a statistical-alignment perspective, recalibrating Batch Normalization statistics via a full forward pass with dynamically adjusted momentum, using separate observer and teacher expert models for reliable statistics and soft-label generation, and initializing synthetic images by incrementally selecting high-confidence, diverse augmentations. Reports 15.6% top-1 accuracy improvement on CIFAR-100-LT and 11.8% on Tiny-ImageNet-LT at IPC=10, IF=10, over prior state-of-the-art long-tailed distillation methods.

Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation · NeurIPS 2025 · Diffusion-based synthesisnotable

Reformulates generative dataset distillation as Optimal Transport distance minimization to capture instance-level and intra-class geometry that global mean/variance matching misses, with three components: OT-guided diffusion sampling that aligns latent distributions, label-image-aligned soft relabeling that adapts label distributions to the complexity of the distilled images, and OT-based logit matching between student outputs and soft-label distributions; reports at least a 4% accuracy gain over the state of the art at IPC=10 on ImageNet-1K across architectures.

Algorithmic Guarantees for Distilling Supervised and Offline RL Datasets · ICLR 2026 · Distribution and feature matchingnotable

Gives a dataset-distillation algorithm for linear regression that matches training and synthetic losses with respect to a fixed set of randomly sampled regressors, with no model training in the loop, proving it needs only O(d^2) sampled regressors for any bounded linear model's synthetic-data loss to track its real-data loss (with a matching Omega(d^2) lower bound), then extends the algorithm to offline reinforcement learning by matching Bellman loss instead of behavioral cloning, using both rewards and next-state information.

Text dataset distillation report · arXiv 2025notable

A review of dataset distillation for text, tracing its path from adaptations of vision methods on embeddings, through transformer-based methods and the generation of discrete synthetic text, to distillation for decoder-only models above one billion parameters, and naming benchmark standardization, the discreteness of text, and complex tasks as the main open problems.

CoDA · ICLR 2026 · Diffusion-based synthesisnotable

Uses an off-the-shelf text-to-image diffusion model, with no target-dataset fine-tuning, by first discovering the target dataset's 'intrinsic core distribution' with a density-based mechanism and then steering sampling to align with it; reports 60.4% top-1 at IPC 50 on ImageNet-1K, matching or beating methods that require a dataset-specific diffusion model.

HALD · ICML 2026 · Label distillation and soft labelsnotable

Shows theoretically and empirically that limiting soft-label crops per image to save storage causes local semantic drift -- visually ambiguous crops pull soft supervision away from the image's true label -- and that hybridizing hard labels back in as a content-invariant semantic anchor corrects it; HALD uses hard labels as an intermediate corrective signal alongside sparse soft labels, reaching 42.7% top-1 on ImageNet-1K with only 285MB of soft-label storage (a 100x reduction) and a 9.0-point gain over LPLD.

3DDP · ICLR 2026 · Synthetic-set parameterizationnotable

This method extends parameterization-based dataset distillation to 3D point clouds by storing a small set of coarser-resolution anchor point clouds plus learnable per-sample morphing weights, so that morphing the anchors' shapes generates a more diverse synthetic set than the same memory budget would allow if samples were stored directly; a uniformity-aware matching loss keeps the morphed samples structurally consistent with the originals. The paper reports outperforming existing dataset distillation methods on ModelNet10, ModelNet40, ShapeNet, ScanObjectNN and OmniObject3D.

OGM · CVPR 2026 · Decoupled teacher-driven synthesisnotable

Shows theoretically that batch-norm statistic matching in decoupled synthesis mainly aligns the scale of real and synthetic gradients but overlooks their direction, though experiments show direction, not scale, is what drives training; Orthogonal Gradient Matching (OGM) orthogonalizes real and synthetic gradients by fixing all singular values to one and matches their singular vectors with a forward-pass-only least-squares loss, reporting 47.0% top-1 with soft labels and 16.7% with hard labels at IPC=10 on ImageNet-1K, ahead of RDED.

MMDD · ICML 2026 · Kernel and closed-form inner solversnotable

MMDD replaces the bi-trajectory distillation used by prior multimodal dataset distillation methods (aligning expert and student trajectories per modality, which needs many stored checkpoints and double backpropagation) with analytic parameter matching, directly matching the closed-form analytic parameters of the modal projectors, which the paper connects theoretically to matrix whitening. It caches only two matrices instead of trajectory checkpoints and computes the match in a single forward pass, reporting up to 65x storage reduction and 9.6x distillation speedup while scaling to 1000 synthetic pairs on Flickr30k and MS-COCO.

AMD · ICLR 2026 · Trajectory matchingnotable

Proposes AMD for image-text dataset distillation, performing asynchronous trajectory matching that decouples the starting points of image and text expert trajectories to accommodate their different learning speeds, and replacing random prototype initialization with a Semantics-Aware Prototype Mining module that clusters the feature space into representative prototypes; reported Recall@1/5/10 gains are 4.5, 9.6 and 10.9 points on Flickr30K at 200 distilled pairs.

Grounding and Enhancing Informativeness and Utility in Dataset Distillation · ICLR 2026 · Decoupled teacher-driven synthesisnotable

Defines Informativeness (information within one sample) and Utility (how essential a sample is to the training set) to mathematically characterize optimal dataset distillation, replacing the field's heuristic balancing of efficiency and quality; InfoUtil instantiates both with game-theoretic Shapley-value attribution to extract key per-sample information and gradient-norm-based selection of globally influential samples, reporting a 6.1% accuracy improvement over the prior state of the art on ImageNet-1K with ResNet-18.

PDS · ICLR 2026 · Diffusion-based synthesiscore

PDS is a learning-free multimodal distillation framework that uses CLIP to extract aligned image-text embeddings, derives prototypes from them, and synthesizes images with an unCLIP decoder, avoiding the full-dataset training and joint pixel/text optimization that architecture-dependent distillation methods require. The paper reports consistently outperforming optimization-based dataset distillation and subset-selection baselines and achieving state-of-the-art cross-architecture generalization.

ManifoldGD · CVPR 2026 · Diffusion-based synthesisnotable

Argues that existing training-free diffusion guidance for distillation relies on simple mode-based centroids that are suboptimal, and instead computes a multi-scale coreset of prototypes via hierarchical divisive clustering of VAE latent features, then at every denoising step projects the mode-alignment vector onto the local tangent space of a manifold built from nearby prototypes, keeping generation manifold-faithful while staying semantically consistent; the first geometry-aware training-free distillation framework, improving FID, real-synthetic embedding distance and classification accuracy over prior training-free and training-based baselines.

Fixed Anchors Are Not Enough: Dynamic Retrieval and Persistent Homology for Dataset Distillation · CVPR 2026 · Decoupled teacher-driven synthesisnotable

Identifies a 'pull-to-anchor' effect in residual-matching decoupled pipelines that rely on static real patches, which reduces intra-class diversity; RETA's Dynamic Retrieval Connection selects a real patch from a prebuilt pool by minimizing a fit-complexity score in teacher feature space before injecting it via a residual connection, and Persistent Topology Alignment penalizes topological discrepancies between real and synthetic sets via persistence images over a mutual k-NN feature graph; reports 64.3% top-1 on ImageNet-1K with ResNet-18 at 50 images per class, +3.1 points over the prior best.

Harmonic Dataset Distillation for Time Series Forecasting · AAAI 2026 · Distribution and feature matchingnotable

HDT (Harmonic Dataset Distillation) decomposes each time series into its sinusoidal basis via the FFT and condenses by Harmonic Matching, aligning the core periodic structure of the synthetic and real series in the frequency domain so that updates never disturb temporal dependencies. The paper reports strong cross-architecture generalization and scalability for time series forecasting compared to conventional, image-centric condensation methods adapted to time series.

Post Training Quantization for Efficient Dataset Condensation · AAAI 2026 · Synthetic-set parameterizationnotable

Applies post-training quantization to already-condensed images to cut storage further: a patch-based scheme keeps quantization localized to avoid the severe degradation plain quantization causes at extreme bit-widths (e.g. 2-bit), quantization-aware clustering groups similar patches to reduce parameter overhead, and a refinement module aligns dequantized images back toward the originals; plugs into any dataset-condensation method's output, nearly doubling test accuracy at 2-bit compression on some methods (e.g. 26.0% to 54.1% for DM at IPC=1).

EVLF · CVPR 2026 · Diffusion-based synthesisnotable

Observes that late-stage cross-attention fusion in diffusion-based distillation lets text prompts dominate generation, over-correcting samples toward prompt patterns at the expense of visual detail; Early Vision-Language Fusion instead aligns text and visual embeddings with a lightweight cross-attention module at the encoder-to-backbone transition, and plugs into any diffusion-based distillation pipeline with an encoder regardless of denoiser architecture or sampling schedule.

IMS3 · CVPR 2026 · Diffusion-based synthesisnotable

Identifies a goal misalignment in diffusion-based distillation -- diffusion models optimize generative likelihood, over-concentrating on high-density regions and under-covering the boundary samples classification needs -- and addresses it with Inversion-Matching, an inversion-guided fine-tuning process that aligns denoising trajectories with their inversion counterparts to broaden distributional coverage, plus Selective Subgroup Sampling, a training-free step that picks representative yet distinctive synthetic subsets for better inter-class separability.

PTM-ST · ICLR 2026 · Trajectory matchingnotable

PTM-ST distills vision-language pairs by matching a stage-aware model of the teacher's training dynamics rather than a single trajectory, using a shortcut-based trajectory construction strategy to fit distinct training phases and stabilize the match, addressing performance gaps between phases and unstable teacher trajectories seen in prior multimodal trajectory-matching methods. On Flickr30K and MS-COCO the paper reports surpassing prior state-of-the-art methods by up to 13.5 percentage points absolute, with an average gain of 9.53 points on Flickr30K, while also reducing storage overhead.

FD2 · ECCV 2026 · Decoupled teacher-driven synthesisnotable

Improves decoupled dataset distillation for fine-grained image datasets, where coarse class-label supervision otherwise yields distilled samples with large intra-class variation and limited inter-class separation; FD2 uses counterfactual attention learning during pretraining to build discriminative class prototypes, then applies a fine-grained characteristic constraint that aligns each sample with its prototype while repelling others, plus a similarity constraint that diversifies attention across same-class samples; reports improvements when combined with existing decoupled methods on fine-grained and general benchmarks.

Sneakdoor · NeurIPS 2025 · Distribution and feature matchingnotable

Targets distribution-matching-based dataset condensation with a backdoor attack designed for stealth, exploiting the vulnerability of class decision boundaries and using a generative module that builds input-aware triggers aligned to local feature geometry to hide artifacts in both the condensed data and the triggered inference samples. Reports a better balance of attack success rate, clean accuracy and stealthiness than prior condensation backdoor attacks, evading both human inspection and statistical detection.

Learnability-guided diffusion · CVPR 2026 · Diffusion-based synthesisnotable

Finds that prior diffusion-based distillation methods produce redundant samples -- disjoint subsets of a distilled set share 80-90% overlapping signal -- because they optimize visual diversity or average training dynamics without accounting for sample-to-sample similarity; Learnability-Guided Diffusion instead builds the synthetic set incrementally, training a model on what exists so far and generating new samples guided by learnability scores balanced against validity under a reference model, cutting redundancy by 39.1% and reaching 60.1% on ImageNet-1K, 87.2% on ImageNette and 72.9% on ImageWoof.

Soft Label Pruning and Quantization for Large-Scale Dataset Distillation · TPAMI 2026 · Label distillation and soft labelsnotable

Extends the within-class-batching idea for reducing soft-label storage with two further mechanisms: Label Pruning with Dynamic Knowledge Reuse improves label diversity per augmentation, and Label Quantization with Calibrated Student-Teacher Alignment improves diversity per image; reduces soft-label storage by 78x on ImageNet-1K and 500x on ImageNet-21K while improving accuracy by up to 7.2 and 2.8 points respectively.

Hard truths about soft labels · CVPR 2026notable

Shows that with abundant or fixed soft labels from a pretrained teacher, coresets fail to convincingly beat random subsets and performance saturates near the full-dataset level regardless of subset quality — unlike the hard-label coreset literature — and that in the hard-label setting only RDED reliably beats random baselines on ImageNet-1K among nine distillation methods tested; introduces a compute-aware pruning metric (CAD-Prune) and a resulting method (CA2D) that outperforms existing distillation methods on ImageNet-1K under a fixed compute budget.

COBRA · ICML 2026 · Optimization and training recipesnotable

Argues that fairness gaps in distilled data come from mismatches between demographic subgroups' predictive patterns, not just group-size imbalance, and so persist even after correcting for imbalance; proposes distilling toward a group-imbalance-agnostic barycenter of the predictive information so subgroups induce similar representations. The plug-in is compatible with existing distillation methods and reported to substantially reduce the bias dataset distillation introduces.

DMGD · CVPR 2026 · Diffusion-based synthesisnotable

Proposes training-free Dual Matching Guided Diffusion: Semantic Matching via conditional-likelihood optimization removes the need for an auxiliary classifier, a dynamic guidance mechanism balances diversity against semantic alignment, and an optimal-transport-based Distribution Matching term aligns generated samples with the target distribution's structure, with two efficiency strategies (Distribution Approximate Matching, Greedy Progressive Matching); reports average accuracy gains of 2.1%, 5.4% and 2.4% on ImageNet-Woof, -Nette and -1K over fine-tuning-based diffusion methods.

Graph condensation needs a reset · arXiv 2026notable

A position paper arguing that mainstream graph condensation contradicts its own goal: gradient-matching methods must train on the full graph to produce the condensed one, generalize poorly across GNN architectures, and are judged by node compression ratios that ignore real resource savings and condensation overhead. It calls for lightweight, architecture-agnostic methods and resource-based evaluation.

C^2R · ICML 2026 · Optimization and training recipesnotable

Argues prior robust dataset distillation trades off accuracy for robustness poorly because it treats all adversarial examples uniformly and does not explicitly widen inter-class decision-boundary separation; C^2R instead derives a robust-margin perturbation score to build an attack-aware curriculum that prioritizes the smallest-margin adversaries driving robust error, paired with a class-balanced contrastive robustness loss that widens boundary separation. Reports the best robust accuracy across six attacks on CIFAR-10/100, Tiny-ImageNet and ImageNet-1K subsets, beating prior robust DD by 2.8% on average.

MDM · CVPR 2026 · Distribution and feature matchingnotable

MDM distills vision-language pairs by initializing synthetic image-text pairs from clusters in the joint embedding space, forming a mixed teacher by interpolating independently fine-tuned models in weight space, and matching real and synthetic joint distributions on the unit hypersphere with a geometry-aware objective that exploits cross-modal agreement and discrepancy directions alongside symmetric contrastive learning. The paper reports compact synthetic sets that preserve multimodal semantics and remain robust under cross-architecture evaluation on image-text retrieval benchmarks.

Do distilled sets beat coresets? · arXiv 2026notable

Benchmarks seven state-of-the-art dataset-distillation methods against three coreset-selection strategies on ImageNet-1K, ImageNet-100 and ImageNette under three standardized evaluation protocols, finding that some distillation methods fail to beat random subsets and that the strongest ones are comparable to or worse than coresets on these large-scale datasets while costing substantially more to construct, with coresets giving better distributional coverage.

DO-ALL · ECCV 2026notable

Before deployment, DO-ALL distills the source data once into a small set of synthetic anchors; during continual test-time adaptation, each target sample is matched to its closest anchor, which supplies a stable reference for source replay, representation alignment and manifold-smoothing regularization without retaining raw source data. As a plug-in, it is reported to consistently improve long-term robustness of existing CTTA algorithms on CIFAR100-C, ImageNet-C and CCC.

RAHA · ECCV 2026 · Distribution and feature matchingnotable

RAHA argues that full-dimensional Euclidean alignment is too restrictive for the rank-deficient image-text correlation typical of vision-language distillation (shared semantics concentrate in a low-dimensional range, with the remainder in a weakly correlated residual subspace), and instead lifts representations to hyperbolic space, optimizing distilled pairs with asymmetric objectives that enforce geodesic alignment on the shared range while regularizing the residual subspace to keep modality-private diversity. The paper reports competitive cross-modal retrieval and improved transfer robustness under fixed budgets versus prior similarity-level low-rank alignment (LoRS).

CIM · ECCV 2026 · Decoupled teacher-driven synthesisnotable

Argues that the squeeze-recover-relabel pipeline's implicit dual compression (data to model, model back to images) causes a distribution shift that makes the pretrained model an unreliable relabeler; CIM instead directly quantifies and minimizes a metric-driven information gap between real and synthetic distributions, reporting 48.7% top-1 on ImageNet-1K at IPC=10 in 80 minutes on one RTX-4090, ahead of NRR-DD and DELT by 2.6 and 2.9 points respectively.