Analysis & theory
ELF
Boosting the Cross-Architecture Generalization of Dataset Distillation through an Empirical Study
Lirui Zhao, Yuxin Zhang, Fei Chao, Rongrong Ji
arXiv 2023 · first public 2023-12-09 · arXiv 2312.05598
In one paragraph
Attributes the poor cross-architecture transfer of distilled datasets to an inductive bias the synthetic images inherit from the network used to distill them, then proposes ELF, an evaluation procedure that trains the evaluation network against intermediate features of the distillation model instead of only labels. Reports that ELF improves cross-architecture accuracy for several existing distillation methods without changing the distilled images themselves.
Where it sits
- Setting: Image classification
Abstract (verbatim from arXiv)
The poor cross-architecture generalization of dataset distillation greatly weakens its practical significance. This paper attempts to mitigate this issue through an empirical study, which suggests that the synthetic datasets undergo an inductive bias towards the distillation model. Therefore, the evaluation model is strictly confined to having similar architectures of the distillation model. We propose a novel method of EvaLuation with distillation Feature (ELF), which utilizes features from intermediate layers of the distillation model for the cross-architecture evaluation. In this manner, the evaluation model learns from bias-free knowledge therefore its architecture becomes unfettered while retaining performance. By performing extensive experiments, we successfully prove that ELF can well enhance the cross-architecture generalization of current DD methods. Code of this project is at \url{https://github.com/Lirui-Zhao/ELF}.
BibTeX (generated; prefer the venue's official entry)
@article{zhao2023boosting,
title = {Boosting the Cross-Architecture Generalization of Dataset Distillation through an Empirical Study},
author = {Lirui Zhao and Yuxin Zhang and Fei Chao and Rongrong Ji},
journal = {arXiv preprint arXiv:2312.05598},
year = {2023}
}