Rectified Decoupled Dataset Distillation: A Closer Look for Fair and Comprehensive Evaluation

Xinhao Zhong, Shuoyang Sun, Xulin Gu, Chenyang Zhu, Bin Chen, Yaowei Wang

ICLR 2026 · first public 2025-09-24 · arXiv 2509.19743

In one paragraph

RD3 systematically studies how post-evaluation choices (augmentation, epoch-wise soft labels, and other protocol details introduced after decoupled distillation methods such as SRe2L) affect reported test accuracy, and finds that much of the performance variation between existing decoupled-distillation methods reflects inconsistent evaluation protocols rather than differences in the intrinsic quality of the synthetic data. The paper proposes a standardized benchmark and evaluation protocol, and identifies general strategies that improve distilled-dataset effectiveness across settings.

Explained

What came before

Decoupled methods split into three sub-paradigms the paper explicitly separates for the first time: optimization-based (SRe2L, CDA, G-VBSM, DWA, EDC — pixel optimization against a pretrained classifier's cross-entropy and BN statistics), selection-based (RDED — crop/score/stitch real patches), and generation-based (D4M, Minimax — fine-tuned generative models or optimized visual-textual embeddings). Each paper's own post-evaluation recipe (batch size, LR schedule, epochs) differs; DC-BENCH (2022) and DD-Ranking (2025) had flagged augmentation and soft labels as confounds but had not isolated batch size/LR-decay-style protocol differences specifically within the decoupled family, nor quantified how much of the SRe2L-to-newer-method accuracy climb survives a fully unified protocol.

The problem

On ImageNet-1K IPC10 with ResNet-18, the accuracy gap reported across published decoupled methods exceeds 27 percentage points; the paper's diagnosis is that this gap is inflated by inconsistent post-evaluation settings (batch size, LR decay, and other hyperparameters), not by differences in the intrinsic quality of the synthetic images.

The idea

Re-run every representative decoupled method (across all three sub-paradigms) under one unified, standardized post-evaluation protocol and measure how much of the reported inter-method gap survives; separately, test whether a random-image-plus-soft-label baseline (in the spirit of label-worth-2024) still beats the "state of the art" once evaluation is unified.

How it works

No new synthesis method (despite the RD3 name, the contribution is the standardized evaluation framework, not a new distillation algorithm). Re-implements/re-evaluates SRe2L, CDA, G-VBSM, DWA, EDC (optimization-based), RDED (selection-based), D4M and Minimax (generation-based) under a single fixed post-evaluation recipe across CIFAR-10/100, TinyImageNet, ImageNette, ImageWoof and ImageNet-1K at multiple IPCs; also cross-evaluates on ResNet-18/50/101, MobileNet-V2, EfficientNet-B0, Swin-V2-T and ViT-B-16, and measures per-method synthesis wall-clock time (IPC=100, single RTX-3090) alongside accuracy. Separately reruns every method with hard labels and with a random-real-image-plus-soft-label control under the same unified protocol.

Evidence

ImageNet-1K, ResNet-18, IPC10, relabel+KD regime (Fig. 1 / Table 2): before unification, published numbers span a 27.3-point range across methods; after RD3's unified protocol, the gap shrinks to 6.7 points, and every method's own accuracy rises substantially once evaluated under RD3's setting (Table 2 deltas: SRe2L +8.4, CDA +3.2, G-VBSM +3.9, DWA +4.0, EDC +2.1, RDED +2.4, D4M +5.0 at IPC50 — i.e. the *original* papers were themselves under-tuned relative to what a good but fixed protocol can achieve). Cross-architecture generalization at IPC50 (Table 3, single ResNet-18 teacher, relabel+KD): performance variance stays within ~5% across the ResNet family but diverges sharply on MobileNet-V2 and ViT-B-16, where SRe2L/CDA/DWA degrade substantially (attributed to low image diversity) while D4M (generation-based, high diversity) reaches SOTA across most architectures. Efficiency (Fig. 3, IPC100, single RTX-3090, excluding classifier-training time): synthesis time varies by up to 100x across methods despite near-identical accuracy — the paper argues efficiency, not accuracy, is now the more informative axis of comparison. Overlooked baseline (Section 5, Table 7): under the soft-label (relabel+KD) regime, simple random real-image sampling *beats every tested decoupled method* on CIFAR-10/100, ImageNette and ImageNet-1K (coarse-grained datasets, where random sampling maximizes diversity while staying consistent with the teacher's knowledge), though decoupled methods still win on fine-grained datasets (TinyImageNet, ImageWoof) where representative image selection matters more. Under hard labels, only Minimax and RDED (both producing realistic, class-representative images) beat random sampling across most datasets/ratios; optimization-based methods and D4M, which rely on soft-label alignment rather than hard-label-correlated images, fall far below random sampling with hard labels.

Limitations

Confirms rather than resolves the underlying tension: the paper shows evaluation protocol explains most of the *inter-method* gap, but does not fully explain why *any* decoupled method beats hard- label random sampling in the first place (it attributes this to soft-label/teacher effects, echoing label-worth-2024 and DD-Ranking without adding a new mechanism). Efficiency comparisons explicitly exclude classifier/teacher training time, understating optimization- and selection-based methods' true deployment cost. Only covers the decoupled-synthesis paradigm (not gradient/trajectory/ distribution matching or coreset selection as first-class comparators, only as an appendix hard-label check). Single-GPU (RTX-3090) timing may not reflect cost at other hardware scales. The "unified protocol" is itself one specific choice of batch size/LR schedule among many plausible ones; the paper does not claim this specific choice is optimal, only that fixing *some* consistent choice collapses most of the reported gap.

Written by the atlas from the paper's full text. Check the paper for exact numbers.

Where it sits

Setting: Image classification

Design choices

Largest scale evaluated

imagenet-1k, imagenet-subsets, mnist-cifar, tiny-imagenet

Abstract (verbatim from arXiv)

Dataset distillation aims to generate compact synthetic datasets that enable models trained on them to achieve performance comparable to those trained on full real datasets, while substantially reducing storage and computational costs. Early bi-level optimization methods (e.g., MTT) have shown promising results on small-scale datasets, but their scalability is limited by high computational overhead. To address this limitation, recent decoupled dataset distillation methods (e.g., SRe$^2$L) separate the teacher model pre-training from the synthetic data generation process. These methods also introduce random data augmentation and epoch-wise soft labels during the post-evaluation phase to improve performance and generalization. However, existing decoupled distillation methods suffer from inconsistent post-evaluation protocols, which hinders progress in the field. In this work, we propose Rectified Decoupled Dataset Distillation (RD$^3$), and systematically investigate how different post-evaluation settings affect test accuracy. We further examine whether the reported performance differences across existing methods reflect true methodological advances or stem from discrepancies in evaluation procedures. Our analysis reveals that much of the performance variation can be attributed to inconsistent evaluation rather than differences in the intrinsic quality of the synthetic data. In addition, we identify general strategies that improve the effectiveness of distilled datasets across settings. By establishing a standardized benchmark and rigorous evaluation protocol, RD$^3$ provides a foundation for fair and reproducible comparisons in future dataset distillation research.

BibTeX (generated; prefer the venue's official entry)

@article{zhong2025rectified,
  title   = {Rectified Decoupled Dataset Distillation: A Closer Look for Fair and Comprehensive Evaluation},
  author  = {Xinhao Zhong and Shuoyang Sun and Xulin Gu and Chenyang Zhu and Bin Chen and Yaowei Wang},
  journal = {ICLR 2026},
  year    = {2025}
}