Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment

What came before

Builds directly on SRe2L's decoupled recovery loss (classification loss plus BN mean/variance matching). Does not name G-VBSM or CDA anywhere in the text; positions itself purely as a fix to SRe2L's per-image independent optimization, and separately criticizes MTT/TESLA (trajectory matching) for compute cost that blocks ImageNet-1K scaling.

The problem

In decoupled synthesis, each synthetic image is optimized independently against the same fixed global teacher BN statistics, so images of one class end up similar. The paper proves the variance-matching term of the BN loss governs almost all of the diversity a synthetic batch can have, that random Gaussian initialization contributes only marginally, and that the mean- and variance-matching gradients can actively conflict, suppressing diversity further.

The idea

Instead of adding random noise for diversity, perturb the teacher's weights in a directed way — chosen to maximize loss on a small random real-data batch, i.e. a sharpness-aware-style ascent step — so each synthetic image is optimized against a slightly different, data-grounded teacher. Because the perturbation is derived from real data rather than noise, it adds meaningful diversity while a first-order argument shows the perturbed teacher still generalizes.

How it works

Teacher: standard pretrained ResNet-18/50/101 (torchvision for ImageNet-1K; standard recipe for CIFAR/Tiny-ImageNet). Baseline recovery loss is SRe2L's, but with the BN coefficient split into separate $\lambda_{mean}$ (fixed 0.01) and $\lambda_{var}$ (tuned, optimum 0.11), since variance matching is shown to dominate diversity. Directed weight adjustment: $\Delta\tilde\theta = \arg\max_{\Delta\theta} L_B(f_{\theta_{\mathcal T}+\Delta\theta})$ over a random real-data subset $B$, solved with $K{=}12$ forward-backward ascent steps and perturbation radius $\rho{=}0.015$; the perturbed teacher is used for that image's recovery loss $\ell(f_{\theta_{\mathcal T}+\Delta\tilde\theta},s_i)+\lambda_{mean}L_{mean}+\lambda_{var}L_{var}$. No change to the relabeling step is described (no storage size or per-crop/stored distinction reported), and label regime is otherwise unchanged from SRe2L. Post-eval: ImageNet-1K 300 epochs, AdamW, cosine LR, RandomResizedCrop+Flip; CIFAR 400 epochs; Tiny-ImageNet 200 epochs, SGD.

Evidence

Own re-run SRe2L baseline throughout (Table 1/2, ResNet-18): CIFAR-10 IPC=10 27.2% to DWA 32.6%, IPC=50 47.5% to 53.1%; CIFAR-100 IPC=10 31.6% to 39.6%, IPC=50 52.2% to 60.9%; Tiny-ImageNet IPC=50 41.1% to 52.8%; ImageNet-1K IPC=10 21.3% to 37.9% (+16.6 points), IPC=50 46.8% to 55.2% (+8.4 points), IPC=100 52.8% to 59.2%. Larger teacher/students at ImageNet-1K IPC=50: ResNet-50 55.6% to 62.3%, ResNet-101 60.8% to 63.3%. No hard-label result. Overhead: the abstract's "<0.1%" claim refers to marginal per-iteration cost, but Table 11 reports a measured 7.3% wall-clock overhead on CIFAR-100 (116.6s/IPC to 125.1s/IPC) — the two overhead figures in the paper are not consistent with each other. Key ablation (Table 3, CIFAR-100, ResNet-18): no adjustment IPC=10 30.6%/IPC=50 56.1%; random (undirected) $\Delta\theta$ IPC=10 collapses to 14.9%/IPC=50 56.2%; directed (DWA) IPC=10 39.6%/IPC=50 60.3% — showing direction, not magnitude of perturbation, is what helps, and that random perturbation actively hurts at low budgets.

Limitations

States its own dependence on sampling a random distribution to select/scale the ascent step is a limitation: too much variance in that distribution can introduce unhelpful noise and bottleneck performance, as the random-perturbation ablation shows directly. No soft-label storage cost is reported despite the atlas-wide concern that relabeling dominates storage in this family. Proposes, but does not implement, sequential/curriculum synthesis for further diversity gains as future work.