On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm

What came before

Groups prior work into bi-level optimization (DC/DSA/MTT), prior-regularized bi-level optimization (GLaD), uni-level/decoupled optimization (SRe2L) and coreset selection (Herding), and evaluates each on realism, diversity and efficiency using qualitative image comparisons (Fig. 2) plus each category's own reported costs.

The problem

Bi-level methods produce noise-like, unrealistic images and overfit to the synthesis architecture; prior-regularized bi-level methods (GLaD) fix realism but inherit bi-level's poor efficiency; SRe2L is efficient and more realistic but, having distilled a pretrained model containing only partial information about the dataset, has limited diversity; coreset selection is realistic but too low in information/diversity. No single method satisfies realism, diversity and efficiency together.

The idea

Treat distillation as maximizing V-information rather than optimizing pixels: score real-image crops for class-informativeness with a pretrained observer model, keep a diverse top-scoring subset, and spatially stitch several crops into one synthetic image with region-level soft labels — no pixel gradients are computed at all.

How it works

An observer model $\varphi_{\theta_{\mathcal T}}$ (ResNet-18 for ImageNet-scale sets, a modified ResNet-18 or ConvNet for CIFAR/Tiny-ImageNet) scores random crops of real images by negative cross-entropy against the true label. Per class, 300 images are pre-selected, each split into $K{=}5$ random patches, the single best-scoring patch per image kept (300 candidate patches/class), then the top $N\times\text{IPC}$ patches by score are retained. $N$ patches (4 for high-resolution images in a 2x2 grid, 1 for CIFAR-scale) are drawn without replacement and concatenated into one synthetic image. No BN-statistics matching and no optimization loop is used; the method cannot even be applied to architectures without batch-norm the way SRe2L can. Relabeling stores an FKD-style soft label per region/patch within a stitched image rather than one label per image. Post-eval augmentation follows SRe2L (RandomResizedCrop + CutMix); students evaluated include ResNet-18/101, EfficientNet-B0, MobileNetV2, VGG-11, Swin-V2-Tiny, and ConvNet-3/4 for CIFAR/Tiny-ImageNet.

Evidence

ImageNet-1K, ResNet-18 (Table 2): IPC=10, RDED 42.0% vs SRe2L 21.3%; IPC=50, 56.5% vs 46.8%. ResNet-101: IPC=10 48.3% vs 30.9%; IPC=50 61.2% vs 60.8%. Tiny-ImageNet, ResNet-18: IPC=10 41.9% vs SRe2L 16.1%; IPC=50 58.2% vs 41.1%. On small ConvNet-distilled CIFAR-10, RDED is weaker than trajectory matching (IPC=10 50.2% vs MTT 65.3%; IPC=50 68.4% vs DATM 76.1%), but stronger on CIFAR-100 (IPC=10 48.1% vs MTT 40.1%). Synthesis cost (Table 3, one RTX-4090, 100 images at once, ResNet-18): 39.89ms/image and 1.57GB peak memory for RDED vs 2113.23ms/image and 9.14GB for SRe2L — the "10 IPC of ImageNet-1K in 7 minutes vs 6 hours, 42% vs 21%" headline claim. No hard-label result is reported; all comparisons use soft/relabeled targets. Key ablation (Table 9): realism-score selection beats Random/Herding/K-Means selection on every one of 6 datasets (e.g. ImageNet-1K 42.0% vs Random 37.9%, Herding 38.4%, K-Means 38.2%), isolating the scoring mechanism itself as the source of gain, independent of the stitching step.

Limitations

States the method struggles at IPC=1 on both high- and low-resolution datasets and its efficacy diminishes on smaller datasets generally. Depends on a pretrained observer model trained on the full real dataset and on that dataset containing enough real, well-scoring patches — a reliance on real data availability and diversity that a later critique (Hard truths about soft labels) ties directly to both RDED's strength under hard labels and its limits. No formal limitations section; these points are embedded in the experiments discussion rather than argued explicitly.