Method
Data Distillation for Text Classification
Yongqi Li, Wenjie Li
arXiv 2021 · first public 2021-04-17 · arXiv 2104.08448
In one paragraph
Proposes an early data-distillation method for text classification that learns a small set of synthetic training examples; reports on eight benchmark datasets that a distilled set 0.1% the size of the original text data recovers about 90% of full-data performance.
Where it sits
- Setting: Text and language models
Abstract (verbatim from arXiv)
Deep learning techniques have achieved great success in many fields, while at the same time deep learning models are getting more complex and expensive to compute. It severely hinders the wide applications of these models. In order to alleviate this problem, model distillation emerges as an effective means to compress a large model into a smaller one without a significant drop in accuracy. In this paper, we study a related but orthogonal issue, data distillation, which aims to distill the knowledge from a large training dataset down to a smaller and synthetic one. It has the potential to address the large and growing neural network training problem based on the small dataset. We develop a novel data distillation method for text classification. We evaluate our method on eight benchmark datasets. The results that the distilled data with the size of 0.1% of the original text data achieves approximately 90% performance of the original is rather impressive.
BibTeX (generated; prefer the venue's official entry)
@article{li2021data,
title = {Data Distillation for Text Classification},
author = {Yongqi Li and Wenjie Li},
journal = {arXiv preprint arXiv:2104.08448},
year = {2021}
}