Learnability-Guided Diffusion for Dataset Distillation

What came before

Minimax fine-tunes for representativeness/diversity, MGD3 guides toward discovered modes, and IGD (Influence-Guided Diffusion, its closest direct predecessor here) steers sampling to match training gradients from the full dataset -- but all three synthesize every sample of a class under one static objective (a fixed diversity criterion or a fixed average-gradient target).

The problem

Optimizing every sample toward the same static target (visual diversity, or the average training gradient) makes samples converge to similar information content: the paper's own cross-validation experiment shows a model trained on any one 10-IPC increment of a 50-IPC set already gets 80-90% (average 94.7% for raw DiT, 87.1% for IGD) of the accuracy achievable from a disjoint increment, meaning the "extra" samples carry little additional learning signal. The root cause identified: a single sample cannot simultaneously suit early training (which needs strong, coarse-feature gradients) and late training (small, fine-grained gradients), so optimizing for "the average" produces samples useful at no specific stage.

The idea

Stop treating distillation as a single-shot synthesis problem; build the distilled set incrementally, training a model on each increment before generating the next, and condition new samples on a "learnability score" that rewards what the current model still struggles with while penalizing samples a reference model (trained on the full dataset) finds invalid or unrepresentative -- so each new increment targets exactly the current model's learning frontier instead of repeating earlier signal.

How it works

The final set of size $M$ is built as $K$ disjoint increments $\mathcal{I}_1,...,\mathcal{I}_K$; stage $i$ trains a learner $\theta_{i-1}$ on the cumulative set $\mathcal{D}_{i-1}=\bigcup_{k<i}\mathcal{I}_k$ and picks the next increment via $\mathcal{I}_i^*=\arg\max_\mathcal{I}[\mathcal{L}(\theta_{i-1}, \mathcal{I})-\mathcal{L}(\theta^*,\mathcal{I})]$, where $\theta^*$ is a reference model trained on the full dataset (regularizing against degenerate/invalid samples). Per-sample learnability score $\mathcal{S}(x,y)=\mathcal{L}(\theta_{i-1},x,y)-\omega\cdot\mathcal{L}(\theta^*,x,y)$ ($\omega=0.5$) is turned into diffusion guidance by adding its gradient to the predicted noise: $\tilde\epsilon_\phi(x_t,t,y)=\epsilon_\phi(x_t,t,y)+\lambda\rho_t\nabla_{x_t}\mathcal{S}(x_t,y)$ ($\lambda=15$, $\rho_t$ a timestep-dependent norm-matching scale), applied only in the timestep window $t\in[10,45]$ of 50 steps (following IGD/MGD3's practice that full-trajectory guidance hurts). A deviation-guidance term additionally subtracts the gradient of cosine similarity to the nearest already-generated same-class sample in a memory buffer ($\gamma=50$) to push new samples away from existing ones. For each of $N_i$ sample slots per class, $\kappa=3$ candidates are generated and the highest-learnability-scoring one is kept (learnability-based selection) before being added to the memory buffer and the process repeats sequentially. The seed set $\mathcal{D}_1$ (10 IPC) is initialized from IGD-distilled images (or optionally other sources); base generator is a pretrained DiT, unmodified (training-free at the generator level; the "training" is of successive learner/reference classifiers, not the diffusion model).

Evidence

ImageNette hard-label protocol (Table 1), IPC50/100 on ResNet-18: 85.0±0.9 / 86.9±0.6 vs IGD 81.0±0.7/84.4±0.8, MGD3 81.5±3.4/85.6±0.2, Minimax 78.1±0.6/81.3±0.7. ImageWoof, ResNet-18, IPC50/100: 65.1±0.7/72.9±0.6 vs IGD 62.0±1.1/70.6±1.8, MGD3 63.9±0.3/71.3±0.5. ImageNet-1K soft-label protocol (Table 2), IPC50, ResNet-18: 60.1±0.1, essentially matching MGD3 60.2±0.1 and beating IGD 59.8±0.3, Minimax 58.6±0.3, DiT 52.9±0.6, RDED 56.5±0.1, all under one shared protocol. Redundancy diagnostic (Fig. 2, ImageNette, 50 IPC split into five 10-IPC increments): cross-increment accuracy averages 94.7% for DiT and 87.1% for IGD but only 57.65% for LGD, a 39.1% reduction versus DiT, directly evidencing reduced information overlap. Component ablation (Table 7, ImageNette IPC20-50, ResNetAP-10, additive over DiT baseline): +Learnability Guidance alone gives +4.7 to +7.0 points; + Deviation Guidance an additional +1.9 to +3.3; + Learnability Ranking a further +0.3 to +1.2 -- isolating that all three pieces (guidance, diversity repulsion, selection) contribute independently and additively, cumulative gain +8.1 to +10.0 over DiT. A difficulty-distribution analysis (Fig. 5) shows LGD's Jensen-Shannon divergence from the real data's difficulty distribution (0.40) is 5x better than DiT (2.04) and 4x better than IGD (1.57).

Limitations

Stated: none in a dedicated Limitations section; the Conclusion frames the incremental/curriculum idea as a "foundation" for future staged-learning-dynamics methods, implying the specific instantiation is a first step. Observed: requires training a learner model to convergence at every one of $K$ incremental stages plus maintaining a separate reference model trained on the full target dataset, substantially more synthesis-time compute than a single-pass guided-sampling method (MGD3, ManifoldGD, CoDA); no GPU-hour or wall-clock comparison to those training-free alternatives is given; depends on IGD-distilled images to seed the first increment, inheriting IGD's own dependence on a target-trained DiT and gradient computation against the full dataset; the ablation without the reference-model regularizer ("DiT+Loss", Fig. 6) is shown to drift out-of-distribution, indicating the method is sensitive to the weighting $\omega$ between current-model and reference-model loss.