EVLF: Early Vision-Language Fusion for Generative Dataset Distillation

What came before

D4M conditions latent diffusion prototypes on label text, MGD3 adds mode guidance inside the denoiser, and VLCP adds richer text prototypes -- but all inject semantic/textual conditioning the same way LDMs and DiTs normally do: via cross-attention inside the denoiser, after the visual latent has already been formed and noised.

The problem

Because standard cross-attention conditioning happens only during denoising, when the latent already carries mainly visual information, textual prompts act as a late corrective force that "overwrites" rather than co-evolves with the visual latent; the paper documents this as unnatural shapes, text-like textures and over-simplified silhouettes (Fig. 1c-d), i.e. samples that match the label but sacrifice visual coherence.

The idea

Move vision-language fusion earlier, to the exact point where a VAE-encoded visual latent and a text/class embedding first meet -- immediately after encoding, before any denoising step is run -- using a small trained cross-attention module, so that semantics and visual structure "co-evolve" from the start of the diffusion trajectory rather than the text dominating only at the end.

How it works

A lightweight cross-attention module CA fuses the VAE visual latent $z_{img}$ (as queries, preserving its spatial role) with class text embeddings $e_{text}$ (as keys/values) at the encoder-backbone interface, giving $z_{fused}=CA(z_{img},e_{text})$, which becomes the diffusion process's initial condition instead of $z_{img}$ alone. CA (plus a small projector $P$) is trained with a dual loss: $\mathcal{L}_{MSE}=\|z_{fused}-z_{img}\|_2^2$ keeps the fused latent close to the original visual latent (visual preservation), and a class-supervised InfoNCE loss $\mathcal{L}_{InfoNCE}=-\frac1B\sum_i\log\frac{\sum_j M^{ij}\exp(s^{ij})}{\sum_j\exp(s^{ij})}$ (with $M^{ij}=1$ iff same class) pulls $z_{fused}$'s projection toward same-class text embeddings and away from other classes'; total loss $\mathcal{L}_{CA}=\lambda_1\mathcal{L}_{InfoNCE}+\lambda_2\mathcal{L}_{MSE}$. This module is trained once (4 epochs, batch 16, AdamW) and is plug-and-play: it slots into any encoder-equipped diffusion pipeline (demonstrated on D4M's LDM and MGD3's DiT-based pipeline) without changing the base method's denoiser architecture or sampling schedule. Optionally, if the base pipeline's denoiser was never adapted to the fused-latent distribution (as with D4M), the denoiser is additionally fine-tuned on fused latents with the standard diffusion loss (Eq. 13); MGD3, whose denoiser already adapts per dataset, keeps its denoiser frozen and only adds the CA module.

Evidence

ImageWoof (Table 1), IPC10 ResNetAP-10: D4M+EVLF/MGD3+EVLF improve their respective bases by roughly 2.7-3.8 points (e.g. 39.3% vs base, +2.7; MGD3+EVLF beats MGD3 by 3.8 at IPC100). ImageNette/ImageIDC, ResNetAP-10 (Table 2), IPC10/50: Nette D4M+EVLF 65.8±1.2/79.7±0.5 vs D4M 60.9±1.7/77.7±1.1; MGD3+EVLF 66.0±1.6/79.5±0.4 vs MGD3 64.3±1.0/79.2±1.9; IDC D4M+EVLF 57.3±1.5/72.1±0.3 vs D4M 47.7±0.5/67.8±1.0 (+9.6 points at IPC10). CIFAR-10/100 (Table 3), IPC10: D4M+EVLF 45.7±0.5/56.2±0.4 vs D4M 37.6±1.8/53.2 ±0.7 (+8.1 on CIFAR-10). Tiny-ImageNet (Table 4), IPC10: D4M+EVLF 49.2±0.4 vs D4M 42.5±0.4. ImageNet-1K soft-label protocol (Table 5), IPC10/50: MGD3+EVLF 51.3±0.3/61.9±0.1 vs MGD3 50.8±0.6/60.3±0.4 and D4M +EVLF 48.3±0.3/60.6±0.0 vs D4M 47.7±0.6/60.1±0.1, both exceeding Minimax 44.3±0.5/58.6±0.3 under the same protocol. Isolating ablation (Table 7, D4M pipeline on ImageIDC, ResNetAP-10): D4M baseline 47.7/56.3/67.8 (IPC10/20/50); +denoiser-fine-tuning-only 54.1/61.1/70.3; +CrossAttention-only 51.1/57.5/69.1; +both (full EVLF) 57.3/62.0/72.1 -- showing CA and denoiser fine-tuning each help independently and combine super-additively. A coverage-metric parameter sweep on $\lambda_1$ (Fig. 4) shows any nonzero text-injection weight sharply raises both accuracy and a nearest-neighbor coverage score relative to $\lambda_1=0$ (no EVLF), with diminishing sensitivity above that.

Limitations

Stated: the method addresses only class-level conditioning and does not handle instance-level or multi-label/compositional prompts; extending to instance-aware prompts is left to future work. Observed: EVLF is not a standalone generative-DD method but a plug-in trained on top of an existing pipeline (D4M, MGD3), so its numbers are only interpretable as deltas over those bases, and it was not tested against training-free or fine-tuned pipelines outside the D4M/MGD3 family (e.g. Minimax's own DiT, CoDA, D3HR); training the cross-attention module and optionally fine-tuning the denoiser still requires target-dataset access and gradient updates, so it does not reduce the target-trained-generator dependency the family's training-free branch (MGD3, ManifoldGD, CoDA) is moving away from; no GPU-hour or wall-clock cost is reported for the added cross-attention training or optional denoiser fine-tuning stage.