DIVER:Diving Deeper into Distilled Data via Expressive Semantic Recovery

What came before

Single-stage distillation methods -- meta/gradient/trajectory matching (MTT), distribution matching (DM, NCFM), latent-optimization (GLaD), decoupled inversion (SRe2L, G-VBSM), and diffusion-guided synthesis (D4M, MGD3) -- each fix the distilled images once, optimized against or generated for one particular architecture's dynamics or statistics.

The problem

Because a single-stage distilled image is shaped by one architecture's gradients, BN statistics, or guidance signal, it overfits architecture-specific patterns and suppresses the transferable, high-level semantics that would let a model trained on it generalize to unseen architectures -- observed as consistent cross-architecture accuracy drops for prior methods' distilled sets.

The idea

Treat any existing distilled image as a rough semantic sketch and use a second, purely post-hoc stage with a frozen pretrained diffusion model to "dive deeper": project the distilled image into the diffusion latent space just far enough to strip architecture-specific noise while keeping its semantics recoverable, then run guided denoising that pulls the trajectory back toward the original image's semantics, restricting that guidance to the middle ("semantic") phase of the reverse process so it neither collapses to the input nor drifts into label-irrelevant artifacts.

How it works

A frozen DiT-XL/2 (256x256) with a frozen VAE (vae-ft-mse) is the only diffusion model; nothing is trained. Semantic Inheritance encodes the distilled image $z_0$ into the VAE latent space and adds forward-process noise for $t_f=25$ steps ($z_{t_f}=\sqrt{\alpha_{t_f}}z_0+\sqrt{1-\alpha_{t_f}}\epsilon$), chosen to balance approaching a Gaussian prior against retaining recoverable features. Semantic Guidance adds a term $G_t=(\hat z_t-z_0)^2\cdot\sigma_t/2$ to the reverse process that penalizes drift of the denoised estimate away from the original distilled latent. Semantic Fusion restricts this class- label-conditioned guidance to a semantic phase window $t\in[t_l,t_h]=[25,40]$ of the 50-step DDIM schedule (classifier-free guidance $\gamma=0.1$), rather than the full trajectory, to avoid the artifacts a full-trajectory guidance signal would introduce. The method takes any prior method's already-distilled images as input and requires no access to the original real dataset.

Evidence

ImageNet-subsets 128x128 cross-architecture (Table 1), MTT baseline vs +DIVER, IPC10 (e.g. ImageSquawk): 21.7+/-2.5 to 33.8+/-1.4; ImageYellow IPC10: 19.1+/-0.9 to 34.8+/-1.6. Full ImageNet-1K 224x224 (Table 3): SRe2L IPC50, RN50: 55.6+/-0.3 to 61.1+/-0.2; G-VBSM IPC50, RN50: 58.7+/-0.3 to 64.2+/-0.4. Diffusion- based methods (Table 4), ResNet-18: MGD3 IPC10 45.8+/-0.3 to MGD3+DIVER 46.4+/-0.3; MGD3+Minimax IPC50 58.6+/-0.3 to MGD3+Minimax+DIVER 61.0+/-0.2. Table 5, ImageNette IPC50: D4M 70.2 to D4M+DIVER 75.1; MGD3 79.5 to MGD3+DIVER 81.2. Baseline numbers for MTT/SRe2L/G-VBSM are taken directly from those methods' public distilled datasets (copied); D4M/MGD3/Minimax numbers are produced by running each method's official code (re-run). No D3HR, CaO2 or IGD comparison. Cost: 2.48s/image on one RTX-4090 at 4GB memory, architecture-agnostic since the guidance stage never touches an evaluation network. Ablation (Table 6, ImageFruit, MTT, IPC1): SI alone 19.5+/-1.4, SG alone 20.4+/-1.7, SI+SG 21.1+/-1.9, SI+SG+SF 22.3+/-1.8, each component adding incrementally; the forward-step count $t_f=25$ and guidance factor $\gamma=0.1$ are each shown optimal via sweep (Fig. 4).

Limitations

Stated: "heavily dependent on the quality of the distilled images" it is given as input, and "extended to a limited range of diffusion-based approaches" (tested on D4M/MGD3, not the training-free or vision-language-conditioned branches). Observed: it is not a standalone generative method but a post-processing stage bolted onto seven different prior pipelines, so its numbers are always framed as deltas and never compared head-to-head against CoDA/DAP/ManifoldGD/VLCP/EVLF as an independent method; applying it to diffusion-based (D4M/MGD3) distilled sets shows smaller and sometimes negative gains for ConvNet-family evaluation (an "acceptable trade-off" the paper accepts for the cross-architecture gain); still depends on a large pretrained DiT-XL/VAE at inference time for every image.