Method · Label distillation and soft labels
HeLlO
Heavy Labels Out! Dataset Distillation with Label Space Lightening
Ruonan Yu, Songhua Liu, Zigeng Chen, Jingwen Ye, Xinchao Wang
ICCV 2025 · first public 2024-08-15 · arXiv 2408.08201
In one paragraph
Replaces stored per-crop soft labels with an image-to-label projector distilled from a foundation model (e.g. CLIP) via LoRA-style fine-tuning, so synthetic labels are generated online from synthetic images instead of stored, plus an image-optimization step that narrows the gap between original and distilled label generators; reports comparable accuracy to state-of-the-art distillation with only about 0.003% of the storage a full soft-label set would need on large-scale datasets.
Explained
What came before
Decoupled ImageNet-scale methods (SRe2L, G-VBSM, RDED) all relabel every augmented crop at every training epoch with a pretrained teacher, and RDED's own numbers show why: on ImageNet-1K at IPC10, hard labels give 15.2% while soft labels give 42.1%. But storing those labels is expensive: the paper reports ImageNet-1K soft-label storage of 572 MB at IPC1 versus ~15 MB for the images (38x), rising to 110 GB at IPC200, comparable to the original dataset's size.
The problem
Soft-label storage, not image storage, becomes the bottleneck of decoupled distillation at scale, yet no prior method addresses this — they treat the size of the teacher-generated label set as a fixed cost of the relabel phase.
The idea
Replace stored labels with a small, fixed function that generates labels on demand: a CLIP-based image-to-label projector, fine-tuned cheaply with LoRA-style low-rank updates from an easily obtained "weak teacher" trajectory, can stand in for a fully pretrained teacher's soft labels at a tiny fraction of the storage, since only the low-rank adaptation matrices (not the labels themselves) need to be kept.
How it works
Builds a linear probe on frozen CLIP image features as the label projector; initializes the linear layer with CLIP's zero-shot text embeddings for the class names (proven equivalent to zero-shot classification, Proposition 1) so no extra parameters need to be stored for initialization. Fine-tunes the projector (CLIP image-encoder convolutions plus the linear head) with LoRA-style low-rank matrices (rank 8 for the image encoder, 64/128 for the linear head) using an MSE-plus-CE loss against soft labels from an ensemble of "weak teacher" checkpoints (9 checkpoints from a single ResNet-18 training trajectory, not a fully trained teacher). Synthetic images are initialized via RDED's patch-selection method, then further optimized to minimize the gap between the true teacher's and the projector's outputs on them. At evaluation, labels are generated online from the projector every step rather than stored; only the low-rank matrices (about 0.6-0.8M parameters) are kept, not any label tensor.
Evidence
ImageNet-1K, ResNet-18 eval, all methods under teacher relabel-with-KD-style regimes (Table 1): HeLlO reaches 12.9% (IPC1, +6.3 over RDED's 6.6%), 43.7% (IPC10, +1.7 over RDED's 42.0%), and 52.2% (IPC50, versus RDED's 56.5% — a regression at the largest budget tested). Label storage for RDED/SRe2L-style relabeling on ImageNet-1K is 572.2 MB / 5722.0 MB / 28610.2 MB at IPC 1/10/50; HeLlO's projector needs roughly 1e-4x/1e-5x/3e-6x of that (its low-rank matrices, ~0.8M params, are fixed-size regardless of IPC), consistent with the abstract's "0.003%" headline figure at large scale. On ImageNet-100 HeLlO also improves accuracy at IPC1/10/50 (+4.4/+12.9/+7.8) while using #Params only 0.06x of RDED's teacher. Cross-architecture (Table 2, IPC10): HeLlO beats RDED on all 5 evaluation architectures, most on Swin-V2-Tiny (+11.7 on ImageNet-1K, +6.2 on ImageNet-100), suggesting the CLIP-based projector generalizes label quality better than a single-architecture teacher. Ablation (Table 3): starting from plain linear-probe CLIP (28.2% accuracy, ImageNet-100 IPC10) and adding multi-weak-teacher guidance (+1.9%), LoRA-like transfer (+13.4%, the largest single gain), text-embedding init (+0.1% but storage falls 1.5M->0.8M params), and image update (+0.1%) reaches 43.7%. Teacher-stage ablation (Table 4, right) shows earlier-stage (less converged) teachers help more at small IPC, later-stage teachers help more as IPC grows — echoing label-worth-2024's early-stopping finding in a different framework.
Limitations
At the largest tested budget (ImageNet-1K IPC50) HeLlO's accuracy is lower than RDED's despite the huge storage saving, so the storage-accuracy trade-off is not uniformly favorable as scale grows. The method depends on a pretrained multimodal foundation model (CLIP) as the label-generation prior, which the atlas's facets record as an added prior not present in SRe2L/RDED; this ties accuracy to CLIP's own class-name/text-embedding coverage and may not transfer to domains poorly covered by CLIP's pretraining. Weak-teacher checkpoints are still needed for fine-tuning (though cheaper than a fully trained teacher). No wall-clock or GPU-hour cost of projector fine-tuning is reported, only storage and parameter counts. Evaluated only on ImageNet-100/1K with the RDED-style patch-selection initialization; no small-scale (CIFAR/TinyImageNet) or non-vision setting is tested.
Written by the atlas from the paper's full text. Check the paper for exact numbers.
Where it sits
- Label distillation and soft labels (Orthogonal design choices)
- Setting: Image classification
Design choices
| Labels | soft-relabel |
| Optimization regime | decoupled |
| Prior / networks used | vision-language-model |
| Largest scale evaluated | imagenet-1k, imagenet-subsets |
Abstract (verbatim from arXiv)
Dataset distillation or condensation aims to condense a large-scale training dataset into a much smaller synthetic one such that the training performance of distilled and original sets on neural networks are similar. Although the number of training samples can be reduced substantially, current state-of-the-art methods heavily rely on enormous soft labels to achieve satisfactory performance. As a result, the required storage can be comparable even to original datasets, especially for large-scale ones. To solve this problem, instead of storing these heavy labels, we propose a novel label-lightening framework termed HeLlO aiming at effective image-to-label projectors, with which synthetic labels can be directly generated online from synthetic images. Specifically, to construct such projectors, we leverage prior knowledge in open-source foundation models, e.g., CLIP, and introduce a LoRA-like fine-tuning strategy to mitigate the gap between pre-trained and target distributions, so that original models for soft-label generation can be distilled into a group of low-rank matrices. Moreover, an effective image optimization method is proposed to further mitigate the potential error between the original and distilled label generators. Extensive experiments demonstrate that with only about 0.003% of the original storage required for a complete set of soft labels, we achieve comparable performance to current state-of-the-art dataset distillation methods on large-scale datasets. Our code will be available.
BibTeX (generated; prefer the venue's official entry)
@article{yu2024heavy,
title = {Heavy Labels Out! Dataset Distillation with Label Space Lightening},
author = {Ruonan Yu and Songhua Liu and Zigeng Chen and Jingwen Ye and Xinchao Wang},
journal = {ICCV 2025},
year = {2024}
}Nearby in Label distillation and soft labels
Xiao Cui, Yulei Qin, Wengang Zhou et al. · NeurIPS 2025notablepaper ↗