DMGD: Train-Free Dataset Distillation with Semantic-Distribution Matching in Diffusion Models

What came before

Minimax fine-tunes a diffusion model per dataset for representativeness/diversity; D4M and MGD3 guide a frozen generator with clustering-derived prototypes or modes but treat each generated sample independently and (per this paper's critique) can overemphasize invalid modes; IGD adds a classifier trained on gradient trajectories for guidance, at the cost of extra training.

The problem

Diffusion-guided distillation methods either need an added fine-tuning stage (Minimax) or use clustering-based conditioning (D4M, MGD3) that neglects the interrelationships among the samples being generated for the same class, producing diversity deficiencies and, per the paper's mean-matching analysis, samples that over-concentrate near a class mean rather than covering the target distribution.

The idea

Guide an unmodified pretrained diffusion model (DiT-XL) with two decoupled, training-free signals computed at sampling time: a "Semantic Matching" term that staggers classifier-free-style soft-label guidance across the denoising trajectory (stochastic exploration, then dynamic soft-label guidance, then semantic refinement), and a "Distribution Matching" term that pulls the evolving synthetic set toward the target distribution's structure via an optimal-transport objective, made tractable with two approximations (K-means clustering of the target, and greedy progressive matching that freezes already generated samples so later ones are pushed to cover what is missing rather than collapsing to the mean).

How it works

Sampling proceeds in three guidance stages by timestep window (t>=45 stochastic exploration, t in [25,45] dynamic soft-label guidance with Gaussian-noised and cross-class-perturbed labels for diversity, t<=25 semantic refinement). A second guidance term adds the gradient of an optimal-transport distance $\nabla_{z_t}\mathcal{L}_{OT}(P_S^t,P_T)$ between the current synthetic latent distribution $P_S^t$ and the target $P_T$; computing this exactly over the full target set is replaced by (a) Distribution Approximate Matching, clustering the target into K=10 clusters per class to approximate $P_T$ cheaply, and (b) Greedy Progressive Matching, generating samples sequentially and freezing earlier ones so each new sample is guided toward the still-uncovered part of the distribution. Nothing is fine-tuned; the DiT backbone, VAE and any teacher/classifier are frozen. Only pixels (plus labels per the evaluation protocol) are stored as the distilled artifact.

Evidence

Hard-label protocol, ResNet10-AP (Table 1): ImageNet-Woof IPC10/50: 40.8+/-1.1 / 60.1+/-0.8 vs MGD3 40.4+/-1.9 / 56.5+/-0.8; ImageNet-Nette IPC10/50: 68.4+/-0.2 / 80.6+/-0.5 vs MGD3 66.4+/-2.4 / 79.5+/-1.3. Plugged onto Minimax's fine-tuned backbone (Minimax+Ours), Woof reaches 42.4+/-0.5 (IPC10) / 60.8+/-0.2 (IPC50). ImageNet-1K soft-label protocol, ResNet-18 (Table 2): IPC10/50: 46.3+/-0.8 / 61.4+/-0.6, vs RDED 42.0+/-0.2 (IPC10) and MGD3 60.2+/-0.1 (IPC50), Minimax 44.3+/-0.5 (IPC10); ResNet-101 IPC10/50: 50.6+/-1.2 / 68.4+/-0.4 vs RDED 48.3+/-1.0 (IPC10) / MGD3 67.7+/-0.4 (IPC50). D3HR, CaO2 and IGD are not included in the main numerical comparison tables (IGD is discussed only in related work, criticized for needing extra classifier-trajectory training); baselines are re-run under the paper's own hard-label/soft-label protocols on a shared DiT/ResNet10-AP or ResNet-18/101 setup rather than quoted verbatim from each source paper. Cost: Minimax's fine-tuning takes ~0.7 GPU-hours versus DMGD's Distribution Approximate Matching at 0.03s per class; a full ImageNet-Woof IPC50 set generates in 0.26 hours; per-image sampling cost is 1.65s vs the unguided DiT baseline's 1.49s. Ablation (Table 3, Woof): Semantic Matching alone 38.9+/-1.2 (IPC10) / 59.3+/-0.4 (IPC50); Distribution Matching alone 41.6+/-1.1 / 56.8+/-0.2; combined 40.8+/-1.1 / 60.1+/-0.8 -- showing SM dominates at low IPC (where per-sample semantic fidelity matters most) and DM dominates at high IPC (where distribution coverage matters most), and combining them is not simply additive.

Limitations

Stated: the paper's own limitations discussion is brief and mostly frames the per-image sampling overhead (1.65s vs 1.49s baseline) as the main cost of adding two guidance terms. Observed: no comparison to D3HR, CaO2 or IGD with numbers under a shared protocol, so its ranking against the inversion/latent-statistics and influence-guided branches of the family is untested; the OT-based Distribution Matching term still requires access to the full target dataset's latents at synthesis time (via K-means over real data), so it does not remove the family's general dependence on real-data statistics even though it removes fine-tuning; K=10 clusters and the timestep boundaries (25, 45) are fixed hyperparameters not shown to transfer across datasets without retuning.