Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets?

Trisha Mittal, Akshay Mehra, Joshua Kimball

arXiv 2026 · first public 2026-06-16 · arXiv 2606.18209

paper ↗notable✓ full text read

In one paragraph

Benchmarks seven state-of-the-art dataset-distillation methods against three coreset-selection strategies on ImageNet-1K, ImageNet-100 and ImageNette under three standardized evaluation protocols, finding that some distillation methods fail to beat random subsets and that the strongest ones are comparable to or worse than coresets on these large-scale datasets while costing substantially more to construct, with coresets giving better distributional coverage.

Explained

What came before

label-worth-2024 (Qin et al.) and DD-Ranking showed soft labels and training tricks inflate reported DD accuracy relative to random images; RD3 showed the same for post-evaluation protocol variance among decoupled methods. But no prior work directly benchmarked current diffusion-guided and decoupled DD methods (VLCP, D3HR, Minimax, DiT-Distillation, MGD3, ManifoldGD, FADRM+) against strong coreset-selection baselines (AUM, concept-guided CCS) under one standardized protocol at ImageNet scale; most DD papers either omit CS baselines or compare only to random selection or weak coresets.

The problem

DD papers routinely claim superiority over coreset selection on the assumption that a subset of real images is fundamentally less expressive than synthesized data, but this claim is rarely tested against strong CS methods under the same evaluation pipeline, and DD accuracy is known to be sensitive to which of several training protocols (hard-label, single-teacher, multi-teacher) is used.

The idea

Hold the dataset, architecture, IPC and training protocol fixed and vary only whether the condensed set was built by distillation or by coreset selection; if strong coresets match or beat distilled sets under every protocol while costing far less to build and covering the data distribution better, DD's claimed intrinsic advantage over selection does not hold at ImageNet scale.

How it works

Benchmarks seven DD methods (three generative training-based: VLCP, D3HR, Minimax; three generative training-free/guided: DiT-Distillation, MGD3, ManifoldGD; one non-generative decoupled: FADRM+) against three CS methods (uniform random, AUM margin-based scoring, concept-bottleneck AUM) on ImageNet-1K, ImageNet-100 and ImageNette at IPC 10/50, training ResNet-18/ResNet-50 (plus ResNet-AP-10 on the smaller sets) from scratch under three evaluation protocols: hard label (CutMix + random-resize-crop, no teacher), single-teacher soft label (RDED's per-crop relabeling with one ResNet-18 teacher), and multi-teacher soft label (EDC's protocol: four teachers' MSE loss plus hard-label cross-entropy). CS coresets use a per-class-adapted Coverage-centric Coreset Selection (stratified sampling over an importance score after discarding the least reliable hardest samples, Alg. 1/App. B) rather than picking pure-hardest examples, since prior work shows hardest-only selection fails catastrophically at small IPC. AUM scores use one ResNet-18 trained 60 epochs on the full dataset; concept scores use a frozen CLIP/LLaVA-derived concept bottleneck trained for 100 epochs, avoiding a full-scale training run. Also measures representativeness (min-max cosine coverage of a validation set) and diversity (1 minus max intra-set cosine similarity) in a frozen ResNet-18 feature space, FID, and construction wall-clock time.

Evidence

ImageNet-1K, ResNet-18/50, same IPC and dataset, three label regimes on the same condensed sets (Table 2) — hard label: AUM beats the best DD method by +3.3 points at IPC10 (18.9% vs. 15.6% for ManifoldGD) and by >4.2 at IPC50 (43.5% vs. 39.3%) on ResNet-18; on ResNet-50, AUM beats the best DD method (Minimax, 12.3%) by +5.0 at IPC10 (17.3%) and beats the best DD method (44.8% MGD3) by +4.1 at IPC50 (48.9%); several DD methods (VLCP, D3HR, DiT) underperform even Random coreset selection under hard labels. Single-teacher soft label (RDED protocol): gaps narrow but CS stays ahead or ties — ResNet-18 IPC50, Concepts 61.0% edges out the best DD method ManifoldGD 60.4%; ResNet-50 IPC50, AUM and Concepts both reach 66.0% vs. best DD (ManifoldGD) 65.6%. Multi-teacher soft label (EDC protocol): CS consistently ahead — ResNet-18 IPC50, AUM/Concepts 60.5-60.6% vs. best DD (ManifoldGD) 59.8%. On the small, easily-separable ImageNette (10 classes), the pattern reverses: DD methods (e.g. ManifoldGD, VLCP) beat all three CS methods under hard labels (e.g. ResNet-18 IPC10: ManifoldGD 60.0% vs. AUM 57.0% vs. Random 44.9%) — the paper's evidence that DD's advantage, where real, is confined to small/easy datasets, not ImageNet scale. Construction cost (Table 5, ImageNet-1K, excludes any teacher/generative-model pretraining): DD methods scale linearly with IPC (ManifoldGD 12.8h at IPC10 -> 51.7h at IPC50; FADRM+ 2.5h -> 12.5h) while CS is IPC-independent (AUM 6.0h at both IPC10 and IPC50; Concepts 3.0h at both). Representativeness- diversity scatter (Fig. 1b) shows AUM in the high-representativeness/high-diversity corner while DD methods match CS on representativeness but trail on diversity, and a qualitative comparison (Fig. 4, "Bald Eagle" class) shows AUM covering varied pose/background/lighting while VLCP/ManifoldGD repeat near-identical canonical views and FADRM+ shows visible synthesis artifacts.

Limitations

Restricted to image classification. Restricted to ImageNet-1K and its subsets because most DD methods only release distilled sets for that dataset; several generative DD methods require training a generative model on the full dataset, which the authors note makes them impractical to extend to new domains within this evaluation. Does not include gradient/trajectory/distribution- matching or kernel-based bi-level DD methods (only decoupled and diffusion-guided/generative methods are tested) or hard-label-native coreset baselines like forgetting/k-center directly (uses AUM and concept-CCS as its two scored CS methods). AUM's own scoring pass still requires one 60-epoch ResNet-18 training run on the full dataset, an upfront cost the paper's time comparison (Table 5) does not include. The hard-label protocol applies CutMix/random-resize-crop uniformly, which the paper does not ablate against a no-augmentation control (unlike DC-BENCH).

Written by the atlas from the paper's full text. Check the paper for exact numbers.

Where it sits

Setting: Image classification

Design choices

Labels	mixed
Optimization regime	decoupled
Prior / networks used	pretrained-classifier, diffusion
Largest scale evaluated	imagenet-1k, imagenet-subsets

Abstract (verbatim from arXiv)

Dataset distillation (DD) has emerged as a prominent approach in data centric machine learning, aiming to synthesize compact training sets for efficient training by compressing the information in large datasets into a small number of synthetic samples. However, DD methods are often evaluated under inconsistent evaluation protocols, ranging from standard ERM to single/multi-teacher supervision, making it difficult to isolate the effectiveness of distilled data from evaluation. Moreover, many prior methods claim that DD outperforms data pruning approaches such as coreset selection (CS), based on the assumption that restricting condensed datasets to subsets of real samples fundamentally limits their expressiveness. In this work, we critically evaluate DD methods through large-scale experiments using standardized datasets and evaluation protocols to assess their intrinsic effectiveness. We benchmark seven state-of-the-art (SOTA) DD methods on ImageNet-1K, ImageNet100, and ImageNette, using three widely adopted training protocols against three CS strategies. Our results show that while some DD methods fail to outperform even simple random subsets, the SOTA DD approaches are comparable to or worse than coresets on large-scale datasets and incur a substantially higher cost for construction. Beyond accuracy, we also evaluate the representativeness, diversity, and quality of condensed sets, and find that coresets consistently achieve better coverage of the original data distribution. These findings highlight the limited practical advantages of current DD methods and show that coresets remain competitive and are often a more computationally efficient alternative for data-centric learning.

BibTeX (generated; prefer the venue's official entry)

@article{mittal2026rethinking,
  title   = {Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets?},
  author  = {Trisha Mittal and Akshay Mehra and Joshua Kimball},
  journal = {arXiv preprint arXiv:2606.18209},
  year    = {2026}
}