Method · Decoupled teacher-driven synthesis
DELT
DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation
Zhiqiang Shen, Ammar Sherif, Zeyuan Yin, Shitong Shao
CVPR 2025 · first public 2024-11-29 · arXiv 2411.19946
In one paragraph
Diagnoses a diversity problem in batch-to-global (decoupled) matching methods such as SRe2L, G-VBSM, WMDD and CDA, where images optimized independently under the same reused global supervision end up homogeneous; DELT partitions the images-per-class budget into subtasks distilled at distinct early and late optimization phases, then generalizes back to the full task, reporting 2-5% average accuracy gains, 5%+ higher per-class diversity, and up to 39.3% less synthesis time on CIFAR, Tiny-ImageNet, ImageNet-1K and its sub-datasets.
Explained
What came before
Decoupled methods (SRe2L, CDA, G-VBSM, WMDD) recover every synthetic image independently against the same global teacher statistics (batch-norm running means/variances and class logits). Bi-level matching methods (FRePo, RCIG, RaT-BPTT, gradient/distribution/trajectory matching) supervise with local real batches but do not scale to ImageNet-1K. G-VBSM increased diversity by adding backbones and statistics, at a large cost in synthesis time.
The problem
Batch-to-global matching reuses one supervision signal for all images of a class, so independently optimized images converge to similar solutions; the paper measures this as high intra-class cosine similarity of ResNet-18 features on ImageNet-1K.
The idea
Split each class's budget into sequential sub-batches that join one shared recovery loop at staggered times. Early-joining batches are optimized for many iterations and drift far from their real-image initialization; late-joining batches stay close to it. The concatenated set therefore spans a range of optimization depths, which adds diversity and saves compute without extra models.
How it works
SRe2L-style recovery (teacher cross-entropy plus batch-norm statistic matching) with a pretrained ResNet-18 teacher, initialized from real images. Two hyperparameters: a maximum iteration count for the first sub-batch (e.g. 4,000) and a round interval after which the next sub-batch joins (e.g. 500), all under one Adam schedule. Students are trained with SRe2L/FKD-style per-crop soft labels from the teacher: AdamW, 300 epochs, RandAugment, RandomResizedCrop and flips.
Evidence
ImageNet-1K with ResNet-18 students and teacher soft labels (Table 1): 46.1% at IPC 10 and 59.2% at IPC 50, against RDED's 42.0% and 56.5%; 66.1% at IPC 50 with ResNet-101. Recovery time at IPC 50 falls from 29.0 h (SRe2L/CDA) to 17.6 h, a 39.3% reduction (Table 6). Adding the staggered schedule to real-image initialization raises IPC-10 accuracy from 43.5% to 46.1% on ImageNet-1K and from 39.4% to 43.0% on CIFAR-10. No hard-label results are reported. Note: Table 1 quotes SRe2L at 21.3% (IPC 10, ResNet-18), while the paper's own cross-architecture table re-runs SRe2L under CDA's configuration at 41.9%. That gap shows how strongly the evaluation recipe moves decoupled-method numbers.
Limitations
The authors note a remaining gap to full-data training and possible privacy leakage from images kept close to real initializations. In the experiments the staggered schedule helps little or hurts at IPC 1, where RDED is stronger (e.g. ImageNette, 24.1% vs 35.8%), and RDED is clearly better on ImageNet-100 at IPC 10 across backbones. All results depend on teacher soft labels, and the paper does not report label storage.
Written by the atlas from the paper's full text. Check the paper for exact numbers.
Where it sits
- Decoupled teacher-driven synthesis (Decoupled synthesis)
- Setting: Image classification
Abstract (verbatim from arXiv)
Recent advances in dataset distillation have led to solutions in two main directions. The conventional batch-to-batch matching mechanism is ideal for small-scale datasets and includes bi-level optimization methods on models and syntheses, such as FRePo, RCIG, and RaT-BPTT, as well as other methods like distribution matching, gradient matching, and weight trajectory matching. Conversely, batch-to-global matching typifies decoupled methods, which are particularly advantageous for large-scale datasets. This approach has garnered substantial interest within the community, as seen in SRe$^2$L, G-VBSM, WMDD, and CDA. A primary challenge with the second approach is the lack of diversity among syntheses within each class since samples are optimized independently and the same global supervision signals are reused across different synthetic images. In this study, we propose a new Diversity-driven EarlyLate Training (DELT) scheme to enhance the diversity of images in batch-to-global matching with less computation. Our approach is conceptually simple yet effective, it partitions predefined IPC samples into smaller subtasks and employs local optimizations to distill each subset into distributions from distinct phases, reducing the uniformity induced by the unified optimization process. These distilled images from the subtasks demonstrate effective generalization when applied to the entire task. We conduct extensive experiments on CIFAR, Tiny-ImageNet, ImageNet-1K, and its sub-datasets. Our approach outperforms the previous state-of-the-art by 2$\sim$5% on average across different datasets and IPCs (images per class), increasing diversity per class by more than 5% while reducing synthesis time by up to 39.3% for enhancing the training efficiency. Code is available at: https://github.com/VILA-Lab/DELT.
BibTeX (generated; prefer the venue's official entry)
@article{shen2024delt,
title = {DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation},
author = {Zhiqiang Shen and Ammar Sherif and Zeyuan Yin and Shitong Shao},
journal = {CVPR 2025},
year = {2024}
}Nearby in Decoupled teacher-driven synthesis
Fixed Anchors Are Not Enough: Dynamic Retrieval and Persistent Homology for Dataset Distillation
Muquan Li, Hang Gou, Yingyi Ma et al. · CVPR 2026notablepaper ↗
Grounding and Enhancing Informativeness and Utility in Dataset Distillation
Shaobo Wang, Yantai Yang, Guo Chen et al. · ICLR 2026notablepaper ↗