Analysis & theory
Can pre-trained models assist in dataset distillation?
Yao Lu, Xuguang Chen, Yuchen Zhang, Jianyang Gu, Tianle Zhang, Yifan Zhang, Xiaoniu Yang, Qi Xuan, Kai Wang, Yang You
arXiv 2023 · first public 2023-10-05 · arXiv 2310.03295
In one paragraph
Empirically studies which properties of a pretrained model used to guide dataset distillation matter, finding that greater model diversity helps, that sub-optimal (not fully converged) models can outperform well-trained ones, and that only a reasonable domain match — not an exact one — is needed, then uses these findings to improve cross-architecture generalization over baseline distillation methods.
Where it sits
- Setting: Image classification
Abstract (verbatim from arXiv)
Dataset Distillation (DD) is a prominent technique that encapsulates knowledge from a large-scale original dataset into a small synthetic dataset for efficient training. Meanwhile, Pre-trained Models (PTMs) function as knowledge repositories, containing extensive information from the original dataset. This naturally raises a question: Can PTMs effectively transfer knowledge to synthetic datasets, guiding DD accurately? To this end, we conduct preliminary experiments, confirming the contribution of PTMs to DD. Afterwards, we systematically study different options in PTMs, including initialization parameters, model architecture, training epoch and domain knowledge, revealing that: 1) Increasing model diversity enhances the performance of synthetic datasets; 2) Sub-optimal models can also assist in DD and outperform well-trained ones in certain cases; 3) Domain-specific PTMs are not mandatory for DD, but a reasonable domain match is crucial. Finally, by selecting optimal options, we significantly improve the cross-architecture generalization over baseline DD methods. We hope our work will facilitate researchers to develop better DD techniques. Our code is available at https://github.com/yaolu-zjut/DDInterpreter.
BibTeX (generated; prefer the venue's official entry)
@article{lu2023trained,
title = {Can pre-trained models assist in dataset distillation?},
author = {Yao Lu and Xuguang Chen and Yuchen Zhang and Jianyang Gu and Tianle Zhang and Yifan Zhang and Xiaoniu Yang and Qi Xuan and Kai Wang and Yang You},
journal = {arXiv preprint arXiv:2310.03295},
year = {2023}
}