Efficient Multimodal Dataset Distillation via Generative Models

What came before

MTT-VL and LoRS both distill vision-language pairs by matching or unrolling expert training trajectories, which the paper documents as extremely costly at this setting's scale: MTT-VL needs about 4 days and up to 320GB memory for 100 pairs on Flickr30K, and LoRS needs about 150 GPU-hours for 500 pairs, because generating expert trajectories and unrolling the bi-level optimization both scale with the number of pairs.

The problem

The paper argues generative distillation is the obvious way to cut this cost but identifies two failure modes specific to applying an off-the-shelf diffusion model to image-text data: (1) a diffusion model trained with a per-sample denoising objective has no incentive to make its generated images correspond to their conditioning captions, so image-text correlation - the thing image-text contrastive (ITC) training needs - is not guaranteed; and (2) with only a few hundred to a thousand pairs standing in for a dataset, generated-sample diversity is essential but not encouraged by the denoising loss either.

The idea

Fine-tune a pretrained text-to-image diffusion model with two auxiliary losses in place of (part of) its denoising objective - a bidirectional image-text contrastive loss that pulls generated images and their conditioning captions together in embedding space while pushing apart mismatched pairs, and a minimax diversity loss that pushes generated samples' embeddings apart - then sample synthetic pairs once from the fine-tuned model, avoiding trajectory matching entirely.

How it works

A pretrained Stable Diffusion v1.5 is fine-tuned (a small subset of its parameters, following the Minimax-Diffusion recipe) on real captions sampled from the target dataset used as generation conditions. The bidirectional contrastive loss (Eqs. 3-4) is a cross-entropy over image-to-text and text-to-image similarity matrices computed from a batch of generated images and their conditioning captions' embeddings, combined as $\mathcal L_C=\lambda\mathcal L_{I\to T}+\mathcal L_{T\to I}$. The diversity (minimax) loss (Eq. 5) is the mean pairwise cosine similarity between normalized concatenated image-text embeddings across the batch, minimized to spread samples apart: $\mathcal L_D=\frac{2}{N(N-1)}\sum_{i<j}\hat z_i\cdot\hat z_j$. After fine-tuning, pairs are produced once by sampling captions from the real dataset and generating an image per caption via reverse diffusion (no further optimization loop); a post-hoc caption- synthesis step then re-captions each generated image with an MLLM (LLaVA or GPT) to add richer text supervision. At evaluation, a pretrained NFNet image encoder is trained on the synthetic pairs while a pretrained BERT-base text encoder is kept frozen (only its projection trained), matching the encoder-freezing protocol MTT-VL/LoRS use.

Evidence

Flickr30K@500 (Table 1, NFNet+frozen-BERT): IR@1/5/10 6.7/21.0/30.5, TR@1/5/10 13.3/35.6/47.5, vs LoRS (re-run/cited) 10.0/28.9/41.6 and 15.5/39.8/53.7 (EDGE trails LoRS here) and vs an unfine-tuned SD v1.5 baseline 3.1/11.5/18.5 and 4.6/15.1/22.2. @1000 (Table 2): IR@1/5/10 9.9/28.2/40.5, TR@1/5/10 14.5/38.3/51.7, vs LoRS 11.2/31.3/42.9 and 14.4/37.5/50.5 (EDGE now slightly ahead on TR@1 but behind on IR). COCO@500 (Table 1): IR@1 1.8, TR@1 2.9, vs LoRS IR@1 2.1/2.8 - EDGE trails LoRS on IR@1 here too. CC3M@1000 (Table 3): IR@1 0.2, TR@1 0.1, both near the random-sampling floor (0.1/0.0). Cost (Table 6, total GPU-hours for the whole distillation pipeline): COCO@500 LoRS 151.6h vs EDGE 7.7h (about 19.7x, rounded to the paper's headline "18x" figure); Flickr30K@500 LoRS 54.2h vs EDGE 13.6h. The key ablation (Table 8, COCO@500) is additive: unfine-tuned SD v1.5 gives IR@1/TR@1 1.2/2.2; adding the contrastive loss alone barely moves it (1.3/2.3); adding the diversity loss too reaches 1.4/2.3; adding caption synthesis on top reaches the full method's 1.8/2.9 - caption synthesis contributes the largest single increment, with the contrastive and diversity losses each contributing only marginally on their own. A caption-source ablation (Table 12) shows the MLLM choice matters: LLaVA-based captions give IR@1 2.8 versus 1.2 for a Llama-based captioner.

Limitations

On several dataset/budget/metric combinations (notably COCO@500 IR@1, both budgets' IR@1 on Flickr30K in places) EDGE does not beat LoRS despite the large cost reduction - it trades some accuracy for roughly 18-20x less compute rather than improving on both axes. Absolute retrieval on CC3M stays near the random-sampling floor even at 1000 pairs. Results depend heavily on the external MLLM used for caption synthesis (a weak captioner collapses performance). The paper's own limitations section flags information loss from distillation, potential security vulnerabilities, and intellectual-property complications from using MLLM-generated captions. The CC3M evaluation uses a partial, URL-decayed subset rather than the full dataset. The text encoder remains frozen at evaluation, so, as in MTT-VL/LoRS, part of the reported number reflects that fixed encoder choice rather than the distillation method alone.