Method · Gradient matching
Learning to Generate Synthetic Training Data using Gradient Matching and Implicit Differentiation
Dmitry Medvedev, Alexander D'yakonov
AIST 2021 · first public 2022-03-16 · arXiv 2203.08559
In one paragraph
Combines ideas from Generative Teaching Networks, gradient matching and the Implicit Function Theorem into new data distillation techniques aimed at reducing training-data requirements. Reports the new methods are more computationally efficient than the prior techniques they combine and improve the performance of models trained on the distilled MNIST data.
Where it sits
- Gradient matching (Surrogate matching)
- Setting: Image classification
Abstract (verbatim from arXiv)
Using huge training datasets can be costly and inconvenient. This article explores various data distillation techniques that can reduce the amount of data required to successfully train deep networks. Inspired by recent ideas, we suggest new data distillation techniques based on generative teaching networks, gradient matching, and the Implicit Function Theorem. Experiments with the MNIST image classification problem show that the new methods are computationally more efficient than previous ones and allow to increase the performance of models trained on distilled data.
BibTeX (generated; prefer the venue's official entry)
@article{medvedev2022learning,
title = {Learning to Generate Synthetic Training Data using Gradient Matching and Implicit Differentiation},
author = {Dmitry Medvedev and Alexander D'yakonov},
journal = {AIST 2021},
year = {2022}
}Nearby in Gradient matching
Linear Gradient Matching — Dataset Distillation for Pre-Trained Self-Supervised Vision Models
George Cazenavette, Antonio Torralba, Vincent Sitzmann · NeurIPS 2025notablePre-training & transferpaper ↗code ↗
Distilled Datamodel with Reverse Gradient Matching
Jingwen Ye, Ruonan Yu, Songhua Liu et al. · CVPR 2024notablepaper ↗
Static-dynamic video DD — Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglement
Ziyu Wang, Yue Xu, Cewu Lu et al. · CVPR 2024coreVideopaper ↗code ↗