Elucidating the Design Space of Dataset Condensation

What came before

Directly re-evaluates SRe2L, G-VBSM and RDED under one unified protocol and shows each left design choices unexplored or unexplained: noise initialization, coarse global-only BN matching, a sharp loss landscape, and — specifically for RDED — an unablated smoothing learning-rate schedule and small evaluation batch size that the paper shows account for 16.2 of RDED's reported points on their own (RDED alone: 25.8% to 42.0% once those tricks are added).

The problem

Because prior papers each explored only a slice of the possible design space (synthesis initialization, matching granularity, loss-landscape flatness, post-evaluation schedule/batch size), reported gaps between methods conflate genuine methodological advances with unexplained evaluation-recipe differences, making comparisons across SRe2L/G-VBSM/RDED unreliable.

The idea

Systematically ablate every design axis across the synthesis, soft-label-generation and post-evaluation stages, keep only the components with an empirical or theoretical justification, and assemble them into one recipe (EDC) evaluated identically across methods.

How it works

Teacher ensemble for ImageNet-1K synthesis: ResNet-18, MobileNetV2, ShuffleNetV2, EfficientNetV2/B0, AlexNet (torchvision-pretrained); relabeling swaps EfficientNetV2 for AlexNet. Recovery keeps BN-statistics-plus-logit matching but adds "soft category-aware matching": a weighted mix of global statistical matching and per-class (Gaussian-mixture, one component per class) statistical matching, $\alpha{=}0.5$ found best, justified by a KL-divergence argument that per-class matching alone would otherwise leave a nonzero divergence floor. Synthesis is initialized from a training-free, RDED-style real-crop condensed set rather than noise, argued via optimal transport to lower the transport cost to the real distribution and shown to roughly halve the needed recovery iterations. A smoothing cosine learning-rate schedule with deceleration factor $\zeta{=}2$ replaces the standard cosine schedule. Flatness regularization uses an EMA teacher copy of the synthetic batch and restricts the KL term to the observer model's logits, cheaper than SAM but matching or beating it. Relabeling follows SRe2L/G-VBSM-style FKD with RandomResizedCrop+RandomHorizontalFlip (RandAugment on smaller sets); no explicit storage size is reported. Post-eval: ImageNet-1K 300 epochs, AdamW, batch 100, EMA-averaged student (rate 0.99); smaller datasets use 1000 epochs.

Evidence

ImageNet-1K, ResNet-18, IPC=10 (Table 1): EDC 48.6% vs SRe2L 21.3%, G-VBSM 31.4%, RDED 42.0% — the latter three re-run by EDC under its own protocol and matching their own papers' numbers closely (RDED matches its own 42.0% exactly; SRe2L/G-VBSM match their own IPC=10 figures). IPC=50: EDC 58.0% vs SRe2L 46.8%, G-VBSM 51.8%, RDED 56.5%. Compression ratio 0.78% = 10,000 synthetic images / ~1.28M real training images at IPC=10. CIFAR-100 IPC=10: EDC 63.7% vs RDED 42.6%, G-VBSM 59.5%. Tiny- ImageNet IPC=50: EDC 57.2% vs RDED 58.2% — RDED edges out EDC here, an exception the paper does not resolve. Synthesis cost (Table 23, 4xRTX4090): roughly halves synthesis time (about 9.8h down to about 5h) while raising accuracy, mainly from the real-image-initialization reduction in iterations. Key ablation (Table 3): real-image init + smoothing LR + small batch alone raise a RDED-style baseline from 34.4% to 38.7% (ResNet-18) at zero extra cost; Table 5 isolates soft category-aware matching + weak augmentation + EMA-eval together adding roughly +2.4 points on top of that. No hard-label ablation or result is reported.

Limitations

States the design space explored, while broad, may still miss valuable strategies, and that EDC's advantage over RDED shrinks as IPC grows (it loses to RDED at Tiny-ImageNet IPC=50). Applying EDC to a new dataset or task requires substantial re-tuning of its hyperparameters or even redesigning the statistical matching, an open challenge the paper does not solve. Still fully dependent on multiple pretrained teacher/observer networks for both synthesis and relabeling, with no reduction in that dependency relative to SRe2L, G-VBSM or RDED.