Influence-Guided Diffusion for Dataset Distillation

What came before

Pixel-level bilevel methods (DM, IDC-1) don't scale to high-resolution ImageNet; SRe2L/G-VBSM/RDED (decoupled synthesis) and Minimax Diffusion/D4M (diffusion sampling/fine-tuning) address ImageNet scale, but Minimax's fine-tuning approach only reshapes the generator's overall distribution toward representativeness/diversity, without any signal tying a specific generated sample to its actual training value; the paper cites core-set selection literature (Killamsetty et al., Iyer et al.) showing even real data sampled from the true distribution contributes unevenly to training, casting doubt on whether representativeness/diversity fine-tuning alone is "optimally tailored for effective training."

The problem

No existing diffusion-based distillation method has a way to steer individual sample generation toward samples that are specifically useful for training a downstream classifier, as opposed to merely representative of and diverse within the real distribution — a gap the paper diagnoses by pointing out that diffusion fine-tuning (Minimax) optimizes distributional properties, not a training-effectiveness objective per sample.

The idea

Reframe dataset distillation as guided diffusion sampling under a "generalized training-effective condition," using the trajectory influence function (TracIn) as the guidance signal: TracIn's formulation happens to reduce to an aggregate over training-loss drops on real data caused by gradient alignment with candidate samples, which the paper shows is (with substitutions) equivalent to the general dataset-distillation objective — so guiding sampling to maximize this influence proxy steers generation toward samples whose training effect approximates that objective, without retraining the diffusion model.

How it works

A pretrained DiT (ImageNet-1K-trained, Stable Diffusion's VAE for the latent encoder/decoder, DDIM sampling with 50 steps) is left frozen; the class-conditional reverse process is modified at each step with two extra guidance gradients (Eq. 9): influence guidance $\mathcal{G}_I(\hat{z}_{0|t})$ (Eq. 7), a cosine similarity between the class-averaged gradient of a surrogate ConvNet-6 (trained on the real target dataset for 50 epochs, with representative checkpoints $\mathcal{R}$ chosen by a gradient- similarity filter rather than fixed intervals) computed on the decoded candidate $D(\hat z_{0|t})$ versus on real class data, approximating TracIn's trajectory-influence sum without needing to retrain any model on synthetic data at each sampling step; and deviation guidance $\mathcal{G}_D(z_t)$ (Eq. 8), which pushes each new latent away from the most-similar previously-generated latent for that class (stored in a per-class memory $\mathcal{M}^c$) to prevent redundancy. Guidance is applied only during an early-to-mid window of denoising steps (e.g. $t\in[30,45]$ of 50 DDIM steps) — "early-stage guidance" — because full-range guidance with a large scale $k$ overfits to the surrogate and produces visible image artifacts. The framework composes with either a raw pretrained DiT ("DiT-IGD") or a Minimax-fine-tuned DiT ("Minimax-IGD"), since it only modifies sampling, not training.

Evidence

ImageNette/ImageWoof, hard labels, cross-architecture (ConvNet-6/ResNetAP-10/ResNet-18), Table 1: at IPC>=50, DiT-IGD improves over raw DiT by 5.8% (Nette) and 6.6% (Woof) on average; Minimax-IGD improves over Minimax by 4.7% (Nette) and 5.1% (Woof); e.g. Woof/ResNetAP-10/IPC=50: Minimax-IGD 65.0% vs. Minimax 59.8% vs. DiT-IGD 62.7% vs. raw DiT 55.1%. ImageNet-1K (Table 2, RDED's protocol, soft-label teacher relabeling with a real-data-trained ResNet-18): IPC=50 Minimax-IGD 60.3% vs. RDED 56.5% vs. G-VBSM 51.8% vs. SRe2L 46.8% vs. raw DiT 52.9% (a +4.0-point gain over RDED, the state-of-the-art image-based baseline, cited in the abstract); IPC=10 DiT-IGD 45.5% vs. DiT 39.6% (+5.9). Cross- architecture on ImageNet-1K (Table 3, unseen ResNet101/MobileNet-V2/EfficientNet-B0/Swin-T): DiT-IGD and Minimax-IGD beat RDED by an average 4.6% and 5.0% at IPC=50. Key ablations: (1) Table 5 — on ImageNette/ResNet-18, adding only influence guidance to raw DiT gives 76.5%/79.1% (IPC50/100) vs. 75.2%/77.8% baseline; only deviation guidance gives 78.2%/80.7%; combining both gives the best 81.0%/84.4% — the two guidance terms are complementary, and deviation guidance alone recovers much of Minimax's diversity benefit without any fine-tuning; for the already-fine-tuned Minimax backbone, influence guidance alone nearly closes the gap to the full method (81.5% vs. 82.0% at IPC50), showing influence guidance matters most when diversity is already handled; (2) Fig. 2b/2c — applying guidance across the entire denoising range with large $k$ reduces the influence loss further than early-stage guidance but *drops* validation accuracy and produces visible artifacts (Fig. 2a), confirming that over-guiding overfits to the ConvNet-6 surrogate rather than improving general training utility; (3) Table 6 — the proposed gradient-similarity checkpoint selection beats regular-interval selection at matched checkpoint counts (e.g. 4 similarity-selected checkpoints beat 10 regularly-spaced ones, 82.0% vs. 81.1%), and the selected indices cluster in early training epochs.

Limitations

The DiT backbone is pretrained on ImageNet-1K, i.e. matched to the target distribution for every reported experiment (ImageNette, ImageWoof, ImageNet-1K are all ImageNet-derived); the influence- guidance mechanism itself additionally requires training a surrogate ConvNet-6 on the real target dataset for 50 epochs purely to compute guidance gradients, which is a real-data-dependent cost the paper does not fold into its "no diffusion retraining" framing; the ImageNet-1K headline numbers use teacher soft-label relabeling (RDED's protocol) while the ImageNette/Woof numbers use hard labels, so the two tables are not on a common label regime; the paper has no explicit stated-limitations section, and does not test IPC=1 or CIFAR/MNIST-scale settings, nor a diffusion model pretrained on non-target data.