Multimodal Dataset Distillation Made Simple by Prototype-Guided Data Synthesis

What came before

Trajectory-matching methods (MTT-VL, LoRS) and the generative method EDGE all require either full-dataset training and bi-level optimization through the encoders (MTT-VL, LoRS) or per-dataset fine-tuning of a diffusion model (EDGE), which the paper argues makes the synthesized set architecture-dependent: because pixels are optimized adversarially against a specific backbone's gradients, the result generalizes poorly to a different evaluation architecture. Learning-free unimodal methods (D4M, MGD3) exist for image classification but do not transfer to paired data because the VAE embeddings they use are not aligned with a separate text encoder's embeddings.

The problem

The paper diagnoses that optimization-based multimodal distillation effectively bakes architecture-specific adversarial perturbations into the synthetic images (since the optimization process differentiates through one particular image/text encoder pair), and that its parameter count and hyperparameter surface (learning rates for both pixels and text, at minimum) grow linearly with the number of pairs, making the method costlier and less transferable as budgets grow. Naively adapting unimodal learning-free synthesis fails because there is no off-the-shelf decoder that is both trained on an aligned image-text embedding space and requires no per-dataset training.

The idea

Do dataset distillation with no gradient-based optimization of any kind: extract CLIP's already cross-modally aligned image and text embeddings for the real data, form per-modality clusters, match clusters across modalities, and average within matched, cross-modally consistent clusters to get prototype embeddings; then decode prototype image embeddings into pixels using a frozen unCLIP decoder (conditioned directly on CLIP image embeddings, with a retrieved real caption as auxiliary conditioning), so the whole pipeline touches no trainable parameters at distillation time.

How it works

Real image and text CLIP embeddings $\{(z^{img}_n,z^{txt}_n)\}$ are extracted once, frozen. Each modality is separately clustered into $M$ clusters via mini-batch k-means, where $M$ is the target pair budget. A linear assignment (Hungarian algorithm) matches image clusters to text clusters using a cost that counts shared image-text pairs between them; within each matched cluster pair, embeddings whose partner does not fall in the matched cluster on the other side are discarded, and the remaining embeddings are averaged to form an image prototype and a text prototype. Each image prototype is decoded to pixels by a frozen unCLIP decoder conditioned on the CLIP image embedding, with classifier-free guidance (scale 5.0, 100 sampling steps) and an additional real caption (retrieved as the training-set caption closest to the text prototype) as conditioning; the text prototype embedding itself is stored as the distilled text. No network (CLIP, unCLIP, or any evaluation encoder) is trained during distillation. At evaluation, the frozen CLIP text encoder is paired with a trainable image backbone (ResNet-50 or ViT-Ti/16, distinct from the CLIP image encoder used to build prototypes) fine-tuned on the distilled set.

Evidence

Flickr30K@100 pairs, ResNet backbone (Table 1): PDS IR@1/5/10 7.9/25.8/37.3, TR@1/5/10 10.2/28.2/39.0, vs LoRS (re-implemented with a CLIP text encoder for fair comparison) IR 6.3/18.6/28.0, TR 9.1/24.3/34.5, vs TESLA-VL IR 4.1/14.7/22.9, TR 6.5/17.8/27.3. @300 pairs, ResNet: IR 14.4/38.1/51.4, TR 18.7/45.0/57.8. COCO@100 pairs, ResNet: IR 2.8/10.0/17.3, TR 4.5/14.0/21.4; @300: IR 5.3/17.2/27.2, TR 7.4/20.7/30.2. Against subset-selection baselines (Table 2, Flickr30K@100, ResNet), the best coreset method (Herding) reaches IR@10 20.1 and TR@10 28.2 versus PDS's 37.3/39.0. Against other learning-free (unimodal, adapted) methods (Table 3, same setting): MGD3 reaches IR@10 17.2, D4M reaches IR@10 9.8, versus PDS's 37.3. Cost (Tables 4-5): PDS synthesizes one image in 9.7s at 4.34GB, versus 1,477.7s at 6.13GB for a pixel-space CLIP- inversion baseline the paper constructs (roughly 150x slower). The key ablations (Table 8: cluster filtering; Table 5: role of image prototypes; Table 6: clustering algorithm) show: discarding embeddings whose modality partner falls outside the matched cluster (filtering) raises IR@10 from 28.1 to 37.3 and TR@10 from 30.6 to 39.0 at 100 pairs; using only the text prototype to retrieve or condition generation (no image-prototype conditioning) drops IR@10 to 26.7-28.7 depending on the ablation variant, an ~9-11 point loss versus using both prototypes; and swapping mini-batch k-means for a Gaussian mixture model gives comparable accuracy (IR@1 8.3 vs 7.9) but 150x more clustering time (Table 7), so k-means is kept for efficiency, not accuracy.

Limitations

The method depends on a generative model that can be conditioned on the same embedding space as the alignment encoder (unCLIP conditioned on CLIP); the paper notes that stronger alignment encoders such as SigLIP have no equivalent embedding-conditioned decoder available, so PDS cannot currently use them. Prototypes formed by clustering are more influenced by dominant/frequent concepts, so long-tail classes risk under-representation (a limitation the paper argues is shared with subset selection and other distillation baselines, with PDS claimed more robust than those in an appendix study). Both CLIP and unCLIP are trained on natural images, so distilling out-of-domain data (e.g. medical imaging) is stated to fail without fine-tuning those models first, reintroducing a training cost the method otherwise avoids. Evaluation protocol (CLIP text tower, non-CLIP image backbones) differs from the NFNet+BERT protocol used by MTT-VL/LoRS/CovMatch/EDGE, so absolute numbers are not directly comparable across that protocol boundary despite reporting against LoRS and TESLA-VL numbers.