Dataset Distillation via Curriculum Data Synthesis in Large Data Era

What came before

Names SRe2L as its "closest baseline" and keeps its squeeze-recover-relabel pipeline and its cross-entropy-plus-BN-statistics recovery loss unchanged. Also compares against MTT and TESLA (trajectory matching) and DM/DSA/FRePo (surrogate/kernel methods) on small-scale data.

The problem

SRe2L uses a constant RandomResizedCrop range across all recovery iterations even though, the paper argues and visualizes, the first few update steps already fix an image's global outline; a poor early crop distribution permanently bakes a noisy or degenerate layout into the synthetic image, and the later iterations can only refine local detail around that fixed outline.

The idea

Schedule the crop scale used during recovery from global to local: start with large, near-full-image crops so the outline forms correctly, then shrink the minimum crop scale over iterations (a curriculum, step/linear/cosine schedules tested, cosine chosen) so later iterations refine local texture. No new loss term is added — only the augmentation schedule fed into the unchanged SRe2L objective changes.

How it works

Squeeze uses the same off-the-shelf teachers as SRe2L (ResNet-18/DenseNet-121 for ImageNet-1K); for ImageNet-21K a ResNet-18/50 is initialized from ImageNet-1K weights and fine-tuned with label smoothing and stronger augmentation. Recover keeps SRe2L's loss (cross-entropy against the target label plus BN mean/variance matching) but replaces the fixed RandomResizedCrop range with a schedule $\alpha=\beta_l+\gamma\cdot(\beta_u+\cos(\pi s/T))/2$ moving the minimum crop scale from near 1.0 toward 0.08 as iteration $s$ approaches milestone $T$. Relabel is unchanged from SRe2L: FKD-style stored soft labels from the same teacher, no per-epoch recomputation. Post-evaluation uses AdamW, cosine LR, 300 epochs, RandomResizedCrop only (no CutMix/Mixup), and a smaller batch size (32) than SRe2L's, justified via a generalization-gap argument.

Evidence

ImageNet-1K, ResNet-18, IPC=50 (Table 1/4): CDA 53.5% vs SRe2L 46.8% (both soft-labeled); IPC=100 58.0% vs 52.8%; IPC=200 63.3% vs 57.0%. The headline "63.2% at IPC=50" and "36.1% at IPC=20" numbers in the abstract are not the ResNet-18 figures but the best cross-architecture results (RegNet-Y-8GF on ImageNet-1K; ResNet-101 on ImageNet-21K). First ImageNet-21K distillation at 224x224 (Table 4): IPC=20, ResNet-18 26.4% vs SRe2L 21.8% (both replicated by CDA, since SRe2L never reported 21K); ResNet-101 36.1% vs 33.2%. Table 1 marks several SRe2L numbers with an asterisk as CDA's own replication (e.g. ImageNet-1K IPC=10 SRe2L* 21.3%) where SRe2L's own paper did not report that setting, while IPC=50/100/200 SRe2L numbers are taken verbatim from SRe2L's paper. No hard-label result is reported. Convergence claim (Table 17): CDA at 1K recovery iterations (52.9%, ResNet-18) beats SRe2L's full 4K-iteration budget (46.8%). Key ablation (Table 3): fixed, non-curriculum crop ranges degrade sharply at either extreme (large-only or small-only crops) relative to the reproduced SRe2L baseline (44.9%), isolating the global-to-local schedule itself as the mechanism.

Limitations

States synthetic realism is not competitive with generative models by design, since the objective prioritizes informativeness. Reports no advantage at very low IPC (e.g. IPC=1). Not evaluated beyond ImageNet-21K. Still fully dependent on SRe2L's teacher-architecture-specific BN-matching recovery objective — CDA changes only the augmentation schedule, not the underlying loss — and its curriculum hyperparameters ($\beta_l,\beta_u,\gamma$, milestone) are tuned per dataset and architecture, an unaddressed tuning burden.