Method · Meta-learning through unrolled training
Dataset Distillation with Attention Labels for Fine-tuning BERT
Aru Maekawa, Naoki Kobayashi, Kotaro Funakoshi, Manabu Okumura
ACL 2023 · first public 2023-01
In one paragraph
Proposes an attention-label distillation method for fine-tuning BERT on condensed text data, transferring attention-probability supervision from a teacher alongside a small number of learned synthetic examples in a bi-level, few-gradient-step distillation setup; reports 98.5% of full-dataset performance from a single example per class and a single gradient step, including 93.2% accuracy on AG News.
Where it sits
- Meta-learning through unrolled training (Bi-level performance matching)
- Label distillation and soft labels (Orthogonal design choices)
- Setting: Text and language models
BibTeX (generated; prefer the venue's official entry)
@article{maekawa2023dataset,
title = {Dataset Distillation with Attention Labels for Fine-tuning BERT},
author = {Aru Maekawa and Naoki Kobayashi and Kotaro Funakoshi and Manabu Okumura},
journal = {ACL 2023},
year = {2023}
}Nearby in Meta-learning through unrolled training
2024-10
Shiye Lei, Sen Zhang, Dacheng Tao · NeurIPS 2024notableOther datapaper ↗code ↗
2023-11
2019-12