Settings · other modalities
Text and language models
Condense text datasets for classifiers or language-model fine-tuning, where the discreteness of tokens breaks the gradient-based synthesis that works for pixels.
One synthetic sample
A token sequence, a sequence of embeddings, or soft token distributions.
Standard evaluation
Fine-tune a text classifier or language model on the synthetic set and evaluate on the real test set.
Budget
examples per class
Benchmarks
SST-2 · AG News · other GLUE-style classification tasks · instruction / fine-tuning corpora
What is different here
- Tokens are discrete, so methods either distill in embedding space (efficient but not human readable), attach distilled attention labels, or distill into a generator language model that emits readable text.
- In the LLM era, condensation meets synthetic-data generation for fine-tuning.
Comparison pitfalls
- Embedding-space synthetic sets only work with the embedding table they were distilled for.
Start with
Soft-label DD · DiLM · GRADMM
Methods by family 9
A paper appears under every family it belongs to. Oldest first within a family.
Bi-level performance matching
Meta-learning through unrolled training 1
Dataset Distillation with Attention Labels for Fine-tuning BERT
Aru Maekawa, Naoki Kobayashi, Kotaro Funakoshi et al. · ACL 2023Textpaper ↗code ↗
Proposes an attention-label distillation method for fine-tuning BERT on condensed text data, transferring attention-probability supervision from a teacher alongside a small number of learned synthetic examples in a bi-level, few-gradient-step distillation setup; reports 98.5% of full-dataset performance from a single example per class and a single gradient step, including 93.2% accuracy on AG News.
Surrogate matching
Gradient matching 2
DiLM — DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation
Aru Maekawa, Satoshi Kosugi, Kotaro Funakoshi et al. · NAACL 2024Textpaper ↗code ↗
Proposes DiLM, which trains a language model to generate readable synthetic text examples for dataset distillation instead of optimizing word-embedding sequences directly, so the distilled data is not tied to one model's embedding table; reports it outperforms coreset-selection baselines on text classification and generalizes to training different model types and to in-context learning with large language models.
GRADMM — Synthetic Text Generation for Training Large Language Models via Gradient Matching
Dang Nguyen, Zeman Li, Mohammadhossein Bateni et al. · ICML 2025notableTextpaper ↗code ↗
Proposes GRADMM, which uses ADMM to optimize the embeddings of synthetic text examples so their gradient matches the (noisy) gradient of real training or validation data, then decodes the embeddings into low-perplexity token sequences, giving convergence, performance and privacy guarantees for fine-tuning LLMs on human-readable synthetic text; reports effectiveness across several classification tasks.
Generative priors
The generator as the distilled artifact 1
DiLM — DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation
Aru Maekawa, Satoshi Kosugi, Kotaro Funakoshi et al. · NAACL 2024Textpaper ↗code ↗
Proposes DiLM, which trains a language model to generate readable synthetic text examples for dataset distillation instead of optimizing word-embedding sequences directly, so the distilled data is not tied to one model's embedding table; reports it outperforms coreset-selection baselines on text classification and generalizes to training different model types and to in-context learning with large language models.
Orthogonal design choices
Label distillation and soft labels 4
Soft-label DD — Soft-Label Dataset Distillation and Text Dataset Distillation
Ilia Sucholutsky, Matthias Schonlau · IJCNN 2021Textpaper ↗code ↗
Extends dataset distillation with soft (distributional) labels learned jointly with the synthetic images instead of one hard label per sample, letting a single synthetic sample encode information for multiple classes; reports 2-4% accuracy gains over hard-label distillation, over 96% MNIST accuracy from 10 distilled images and about 92% from just 5, and extends the same approach to distilling sequential text data, retaining near-original accuracy on IMDB sentiment analysis from 20 distilled sentences.
Learned labels — Flexible Dataset Distillation: Learn Labels Instead of Images
Ondrej Bohdal, Yongxin Yang, Timothy Hospedales · NeurIPS 2020 WorkshopTextpaper ↗code ↗
Studies label distillation -- learning synthetic soft labels for a small, fixed set of real images -- as an alternative to distilling images, using a more robust meta-learning algorithm and a first-order strategy based on convex optimization layers; shows label distillation improves on prior image-based distillation, generalizes across optimizers and architectures more readily, and can even transfer across datasets, e.g. learning Japanese character recognition from synthetically labeled English letters.
Dataset Distillation with Attention Labels for Fine-tuning BERT
Aru Maekawa, Naoki Kobayashi, Kotaro Funakoshi et al. · ACL 2023Textpaper ↗code ↗
Proposes an attention-label distillation method for fine-tuning BERT on condensed text data, transferring attention-probability supervision from a teacher alongside a small number of learned synthetic examples in a bi-level, few-gradient-step distillation setup; reports 98.5% of full-dataset performance from a single example per class and a single gradient step, including 93.2% accuracy on AG News.
VQAE — Vector-Quantized Soft Label Compression for Dataset Distillation
Ali Abbasi, Ashkan Shahbazi, Hamed Pirsiavash et al. · arXiv 2026Textpaper ↗
Quantifies the bit cost of soft teacher labels across augmentations in dataset distillation, showing they dominate total storage in large-class settings such as ImageNet-1K, and introduces a vector-quantized autoencoder (VQAE) to compress them. Reports 30-40x additional compression over RDED, LPLD, SRe2L and CDA on ImageNet-1K while retaining over 90% of their original accuracy, and validates the approach on both vision and language distillation benchmarks.
Setting-specific methods
CondenseLM — CondenseLM: LLMs-driven Text Dataset Condensation via Reward Matching
Cheng Shen, Yew-Soon Ong, Joey Tianyi Zhou · EMNLP 2025Textpaper ↗code ↗
Proposes CondenseLM, which uses an LLM-driven pipeline guided by reward matching to generate condensed text-classification datasets directly rather than optimizing embedding-level synthetic samples, aiming to improve representability and coverage over prior text condensation methods; reports it substantially outperforms coreset-selection and dataset-condensation baselines on SST-2, MNLI, AG News and IMDB while reducing computational cost.
Textual Dataset Distillation via Language Model Embedding
Yefan Tao, Luyang Kong, Andrey Kan et al. · EMNLP 2024Textpaper ↗
This method distills text classification datasets into language-model embeddings rather than into embeddings tied to one specific model, aiming for a model-agnostic, data-efficient distilled representation with better transferability than prior embedding-space text distillation. On IMDB and AG-News the paper reports results comparable to or better than parameter-efficient fine-tuning (LoRA) and than prior model-dependent text distillation methods, with faster processing.
Data Distillation for Text Classification
Yongqi Li, Wenjie Li · arXiv 2021Textpaper ↗
Proposes an early data-distillation method for text classification that learns a small set of synthetic training examples; reports on eight benchmark datasets that a distilled set 0.1% the size of the original text data recovers about 90% of full-data performance.
Analyses, benchmarks, surveys and applications
Attention Hijacking — Attention Hijacking: Backdooring Text Dataset Distillation via Semantic Anchors
Hang Ren · ICML 2026Textpaper ↗
Proposes a backdoor attack on text dataset distillation built on a "Semantic Anchoring Hypothesis": the attack reshapes gradients into input embeddings so the synthetic data evolves to turn a trigger word into an adversarial feature, integrated into the bi-level distillation loop so the attack satisfies both the clean-task and backdoor objectives at once.
Text dataset distillation report — Technical Report on Text Dataset Distillation
Keith Ando Ogawa, Bruno Lopes Yamamoto, Lucas Lauton de Alcantara et al. · arXiv 2025notableTextpaper ↗
A review of dataset distillation for text, tracing its path from adaptations of vision methods on embeddings, through transformer-based methods and the generation of discrete synthetic text, to distillation for decoder-only models above one billion parameters, and naming benchmark standardization, the discreteness of text, and complex tasks as the main open problems.
UniDetox — UniDetox: Universal Detoxification of Large Language Models via Dataset Distillation
Huimin Lu, Masaru Isonuma, Junichiro Mori et al. · ICLR 2025notableTextpaper ↗code ↗
UniDetox distills a small synthetic text dataset that carries detoxifying signal, generated from one source model (GPT-2) via contrastive decoding, and fine-tunes any target LLM on it to reduce toxicity without per-model hyperparameter tuning. The paper reports that text distilled from GPT-2 successfully detoxifies larger, unrelated models (OPT, Falcon, LLaMA-2) with a single shared hyperparameter configuration, and that the distilled text also reduces politically biased content.
Luyang Fang, Xiaowei Yu, Jiazhang Cai et al. · arXiv 2025Textpaper ↗
Surveys knowledge distillation (KD) and dataset distillation (DD) as complementary strategies for compressing large language models. Covers KD methodologies (task-specific alignment, rationale-based training, multi-teacher frameworks) alongside DD techniques that synthesize compact fine-tuning data via optimization-based gradient matching, latent-space regularization, and generative synthesis, discusses combining the two, surveys applications in healthcare and education, and identifies open challenges in preserving emergent reasoning and linguistic diversity and in establishing evaluation protocols.