Gradient matching

DC · ICLR 2021

Formulates condensation as matching the gradients of network weights computed on synthetic and on real data, and demonstrates the condensed sets for continual learning and architecture search.

DSA · ICML 2021

Applies the same differentiable augmentation to real and synthetic batches during matching, so that synthetic images stay informative under the augmentation used to train on them.

DCC · ICML 2022

Proves that class-wise gradient matching can do worse than random selection when task-irrelevant information dominates the data, and restores contrastive signal between classes; the fix matters most for fine-grained classification.

GCond — Graph Condensation for Graph Neural Networks

Wei Jin, Lingxiao Zhao, Shichang Zhang et al. · ICLR 2022landmarkGraphspaper ↗code ↗

Introduces GCond, which condenses a large graph into a small synthetic graph and features by imitating the GNN training trajectory on the original graph through a gradient-matching loss, jointly synthesizing node features and structure so that GNNs trained on the condensed graph transfer across architectures; reports approximating original test accuracy by 95.3% on Reddit, 99.8% on Flickr and 99.0% on Citeseer while shrinking graph size by over 99.9%.

Static-dynamic video DD — Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglement

Ziyu Wang, Yue Xu, Cewu Lu et al. · CVPR 2024coreVideopaper ↗code ↗

This paper gives the first systematic study of video distillation, finding that the temporal dimension of synthetic clips is usually not well learned and contributes little, and responds by disentangling static and dynamic information: a static learning stage distills videos into still images via gradient matching, and a dynamic fine-tuning stage compensates motion with a separately stored, learnable dynamic memory block rather than distilling every frame. The paper reports state-of-the-art results on video datasets at multiple scales with a notably smaller memory storage budget than prior methods.

Linear Gradient Matching — Dataset Distillation for Pre-Trained Self-Supervised Vision Models

George Cazenavette, Antonio Torralba, Vincent Sitzmann · NeurIPS 2025notablePre-training & transferpaper ↗code ↗

Linear Gradient Matching distills a dataset for training linear probes on top of a frozen, pretrained self-supervised vision model, optimizing synthetic images so that gradients they induce in the linear classifier match those from real data. The paper reports that the resulting synthetic sets outperform all real-image baselines, generalize across pretrained backbones (e.g. a set distilled via DINO trains a competitive linear CLIP probe), and are effective for fine-grained classification and for interpretability probes such as measuring embedding-space similarity between models.

PRISM — PRISM: Video Dataset Condensation with Progressive Refinement and Insertion for Sparse Motion

Jaehyun Choi, Jiwan Hur, Gyojin Han et al. · CVPR 2026notableVideopaper ↗

PRISM treats a condensed video as one fully coupled spatiotemporal structure rather than disentangling it into separate static and dynamic components, starting from minimal temporal anchors and progressively inserting key-frames only where linear interpolation between them fails to capture non-linear motion, guided by gradient misalignments between synthetic and real data. The paper reports competitive accuracy with state-of-the-art storage efficiency on standard video-distillation benchmarks by allocating representational capacity only where the motion needs it.

GRADMM — Synthetic Text Generation for Training Large Language Models via Gradient Matching

Dang Nguyen, Zeman Li, Mohammadhossein Bateni et al. · ICML 2025notableTextpaper ↗code ↗

Proposes GRADMM, which uses ADMM to optimize the embeddings of synthetic text examples so their gradient matches the (noisy) gradient of real training or validation data, then decodes the embeddings into low-perplexity token sequences, giving convergence, performance and privacy guarantees for fine-tuning LLMs on human-readable synthetic text; reports effectiveness across several classification tasks.

Distilled Datamodel with Reverse Gradient Matching

Jingwen Ye, Ruonan Yu, Songhua Liu et al. · CVPR 2024notablepaper ↗

Approximates each training sample's influence on a pretrained model without leave-one-out retraining, by first distilling the training set into a synset via reverse gradient matching (offline) and then using that synset to speed up leave-one-out evaluation and compute an attribution matrix (online). Reports training-data-attribution and data-quality evaluations comparable to direct retraining at substantially lower cost.

SSD — Summarizing Stream Data for Memory-Constrained Online Continual Learning

Jianyang Gu, Kai Wang, Wei Jiang et al. · AAAI 2024notablepaper ↗code ↗

SSD summarizes the incoming data stream into more informative replay samples by distilling the training characteristics of real images, maintaining consistency of training gradients and relationship to past tasks so summarized samples represent the stream better than raw exemplars at the same buffer size. Reports more than a 3% accuracy boost on sequential CIFAR-100 under an extremely restricted memory buffer, with limited extra computational overhead.

SR-GM — Decoupling and Damping: Structurally-Regularized Gradient Matching for Multimodal Graph Condensation

Lian Shen, Zhendan Chen, Meijia Song et al. · arXiv 2025Graphspaper ↗

Condenses multimodal graphs, where node features integrate sources such as vision and text, by decoupling per-modality gradients before matching so that semantic misalignment between modalities does not create gradient conflicts, and adds a structural damping regularizer that suppresses gradient noise the message-passing mechanism would otherwise amplify through the topology. Reports state-of-the-art results and cross-architecture generalization on four multimodal graph datasets.

Low-Level Dataset Distillation for Medical Image Enhancement

Fengzhi Xu, Ziyuan Yang, Mengyu Sun et al. · arXiv 2025Dense predictionpaper ↗

Argues that low-level tasks (denoising, enhancement) make dataset distillation underdetermined because the many-to-many pixel-level mapping cannot be constrained by a small distilled set the way many-to-one classification labels can; proposes the first low-level DD method, which builds a shared anatomical prior from one representative patient, personalizes it per patient with a Structure-Preserving Personalized Generation module, and injects patient-specific knowledge by matching gradients between networks trained on distilled pairs and on the patient's raw data. Downstream users only receive the abstracted distilled pairs, not raw patient data.

HCDC — Calibrated Dataset Condensation for Faster Hyperparameter Search

Mucong Ding, Yuancheng Xu, Tahseen Rabbani et al. · arXiv 2024paper ↗

Rather than matching gradients to preserve accuracy, HCDC condenses a synthetic validation set so that the ranking of models by hyperparameter/architecture choice is preserved between the condensed and real validation sets, matching hyperparameter gradients via implicit differentiation and an approximate inverse Hessian. The paper reports the condensed validation set keeps model rankings stable and speeds up hyperparameter and architecture search on both image and graph tasks.

DiLM — DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation

Aru Maekawa, Satoshi Kosugi, Kotaro Funakoshi et al. · NAACL 2024Textpaper ↗code ↗

Proposes DiLM, which trains a language model to generate readable synthetic text examples for dataset distillation instead of optimizing word-embedding sequences directly, so the distilled data is not tied to one model's embedding table; reports it outperforms coreset-selection baselines on text classification and generalizes to training different model types and to in-context learning with large language models.

Graph Data Condensation via Self-expressive Graph Structure Reconstruction

Zhanyu Liu, Chaolv Zeng, Guanjie Zheng · KDD 2024Graphspaper ↗code ↗

GCSR (GCSR) condenses a graph by reconstructing an interpretable, closed-form self-expressive adjacency for the condensed nodes (each node reconstructed from a sparse combination of the others in its feature subspace) instead of learning a structure generator or dropping structure entirely, while node features are refined by multi-step gradient matching against the full graph's training trajectories. The paper reports the best average cross-architecture GNN accuracy among graph condensation baselines, attributing the gain to keeping explicit structural information.

CTRL — Two Trades is not Baffled: Condensing Graph via Crafting Rational Gradient Matching

Tianle Zhang, Yuchen Zhang, Kun Wang et al. · arXiv 2024Graphspaper ↗code ↗

CTRL observes that prior graph condensation methods match only the direction of gradients, which lets the condensed graph's training trajectory drift from the real graph's and accumulates error (worsened by condensation/evaluation-phase mismatch); it corrects this with an optimized starting point closer to the real feature distribution and a more refined gradient-matching strategy. The paper reports that CTRL neutralizes the impact of accumulated errors on condensed-graph performance, with experiments across multiple graph datasets and downstream tasks.

IADD — Importance-Aware Adaptive Dataset Distillation

Guang Li, Ren Togo, Takahiro Ogawa et al. · NN 2024paper ↗

Observes that parameter-/gradient-matching distillation treats every network parameter as equally important and proposes IADD, which automatically assigns importance weights to different parameters during distillation to synthesize more robust distilled sets. Reports outperforming other parameter-matching SOTA methods on multiple benchmarks and in cross-architecture generalization, and validates the method on real-world COVID-19 detection.

Differentially Private Dataset Condensation

Tianhang Zheng, Baochun Li · NDSS 2024 Workshoppaper ↗

Proposes two differentially-private dataset condensation algorithms: LDPDC, a linear DC method that runs on a low-end CPU, and NDPDC, which uses a neural network for representation extraction under a DP feature/gradient-matching objective. Reports LDPDC performs comparably to recent privacy-preserving generative methods, while NDPDC gives acceptable DP guarantees with only mild utility loss relative to plain distribution matching.

LCMat — Loss-Curvature Matching for Dataset Selection and Condensation

Seungjae Shin, Heesun Bae, Donghyeok Shin et al. · AISTATS 2023paper ↗code ↗

Matches the loss curvature — not just the loss value or a first-order gradient — of the original and reduced datasets over a neighborhood of the model-parameter space, deriving an implementable upper bound on the worst-case curvature gap, and applies the same objective to both coreset selection and dataset condensation, improving generalization over existing baselines in both settings.

FedLGD — Federated Learning on Virtual Heterogeneous Data with Local-global Distillation

Chun-Yin Huang, Ruinan Jin, Can Zhao et al. · TMLR 2024paper ↗code ↗

Observes that distilling each client's local data independently can worsen federated heterogeneity, so FedLGD instead uses iterative distribution matching to inpaint global information into local synthetic ("virtual") data and federated gradient matching to distill global virtual anchor points that rectify heterogeneous local training. Reports outperforming prior heterogeneous-FL methods on benchmark and real-world datasets, including at large numbers of class-imbalanced clients.

FedLAP-DP — FedLAP-DP: Federated Learning by Sharing Differentially Private Loss Approximations

Hui-Po Wang, Dingfan Chen, Raouf Kerkouche et al. · arXiv 2023paper ↗code ↗

Clients synthesize a small set of samples that approximate their local loss landscape by simulating the gradients of real images within a local region; these loss surrogates, rather than model updates, are aggregated on the server to recover the global loss landscape, which the paper shows lets record-level differential privacy be enforced at the same privacy cost as gradient-sharing FL but with a better privacy-utility trade-off and faster convergence.

Fair Graph Distillation

Qizhang Feng, Zhimeng Jiang, Ruiquan Li et al. · NeurIPS 2023Graphspaper ↗

Shows that GNNs trained on graphs condensed by existing graph-distillation methods can exhibit more severe group-fairness problems than GNNs trained on the original graph, then proposes a fair graph-distillation approach that generates small distilled graphs that are both fair and informative.

CGM — Gradient Matching for Categorical Data Distillation in CTR Prediction

Cheng Wang, Jiacheng Sun, Zhenhua Dong et al. · RecSys 2023Other datapaper ↗

Proposes CGM (Categorical data distillation with Gradient Matching), which extends gradient-matching dataset distillation to the high-dimensional, sparse categorical features of click-through-rate prediction data, addressing the blocked gradient flow through categorical embeddings and the cost of the resulting bi-level optimization; distills a small synthetic dataset that trains CTR models from scratch toward performance close to training on the full data.

Data-efficient Neural Network Training with Dataset Condensation

Bo Zhao · The University of Edinburgh 2023paper ↗

PhD thesis presenting the author's own gradient-matching (DC), differentiable-Siamese-augmentation (DSA) and distribution-matching (DM) approaches to dataset condensation as one body of work on data-efficient neural network training.

DosCond — Condensing Graphs via One-Step Gradient Matching

Wei Jin, Xianfeng Tang, Haoming Jiang et al. · KDD 2022Graphspaper ↗code ↗

Proposes DosCond, a one-step gradient-matching scheme for graph condensation that models the discrete graph structure probabilistically and matches gradients for a single step instead of unrolling multi-step training; reports 90% graph-size reduction while retaining up to 98% of original performance, about 15x faster than multi-step gradient matching.

Learning to Generate Synthetic Training Data using Gradient Matching and Implicit Differentiation

Dmitry Medvedev, Alexander D'yakonov · AIST 2021paper ↗code ↗

Combines ideas from Generative Teaching Networks, gradient matching and the Implicit Function Theorem into new data distillation techniques aimed at reducing training-data requirements. Reports the new methods are more computationally efficient than the prior techniques they combine and improve the performance of models trained on the distilled MNIST data.