Vision-Language Dataset Distillation

What came before

Before this paper, dataset distillation (trajectory matching in particular, MTT) had only been applied to single-modal image classification, where a fixed set of classes gives synthesis a natural unit (match per-class statistics or trajectories). No prior method existed for paired image-text data.

The problem

Vision-language datasets have no discrete classes to condition synthesis on, so the class-wise machinery of prior trajectory matching does not transfer, and no baseline for this setting existed at all (the paper builds its own adapted coreset baselines to compare against).

The idea

Treat the whole image-text corpus as one unlabeled pool and jointly distill synthetic images and synthetic text embeddings so that a student contrastively trained on them for a few steps lands close to where an expert trained on real pairs would land after many more steps; efficiency at the scale of modern vision-language backbones is recovered by matching only low-rank LoRA updates of a frozen pretrained encoder rather than its full trajectory.

How it works

Images (pixels) and text (768-d BERT embeddings) are both directly optimized parameters. A pretrained NFNet (or ViT) image encoder plus a trainable linear projection, and a frozen pretrained BERT text encoder plus a trainable linear projection, produce embeddings trained with a bidirectional (image-to-text and text-to-image) contrastive loss. Expert trajectories are pairs of (image-tower, text-tower) parameter checkpoints obtained by training on real data for several epochs; the distillation loss matches the student's trajectory after $N$ synthetic-data steps against the expert's trajectory after $M$ real-data steps, normalized by the expert's total movement: $\|\hat\theta_{t+N}-\theta^*_{t+M}\|_2^2/\|\theta^*_t-\theta^*_{t+M}\|_2^2$, summed over the image and text towers. For ViT backbones, LoRA matrices ($A\in\mathbb{R}^{d\times r}$, $B\in\mathbb{R}^{r\times d}$) are matched instead of full weights, cutting the trajectory's trainable parameter count by about 79% (86M to 18M on ViT-Base/16).

Evidence

With an NFNet image encoder and frozen BERT text encoder, on Flickr30K at 100 pairs the method reports TR@1/5/10 = 9.9/28.3/39.1 and IR@1/5/10 = 4.7/15.7/24.6 (Table 6); at 1000 pairs, TR@1/5/10 = 13.3/34.8/45.9 and IR@1/5/10 = 7.9/24.1/33.8, versus the best coreset baseline (K-center, 1000 pairs) at TR@1 = 5.6, IR@1 = 4.4 (Table 1), and versus full-dataset training at TR@1 = 33.9, IR@1 = 27.3 (Table 3). On COCO at 100/1000 pairs, TR@1 = 2.5/6.8, IR@1 = 1.3/3.3 (Table 6). Expert trajectories take about 6-15 GPU-hours on an 8-GPU A6000 node (expert training itself: one RTX 3090, 10 epochs at roughly 40 min/epoch). The key ablation is co-distillation versus unimodal distillation at 100 pairs (Table 5): distilling only text reaches TR@1 = 1.3, distilling only images reaches TR@1 = 3.5, while joint co-distillation reaches TR@1 = 9.9 - the cross-modal contrastive coupling, not either modality alone, drives the gain. A separate ablation (Table 10) shows sensitivity to the text backbone: swapping frozen BERT for frozen CLIP text features raises both the full-data expert ceiling (TR@1 61.2 vs 33.9) and the 100-pair distilled result (TR@1 31.4 vs 9.9) by a similar margin, so much of the reported number is a property of the chosen encoder pair rather than the distillation method alone.

Limitations

The paper explicitly invokes the "no free lunch" theorem: results depend heavily on which image/text backbone and optimizer are used during distillation, and cross-architecture transfer is weak (distilling on NFNet and evaluating on ViT or NF-ResNet50 at 100 pairs collapses TR@1 from 9.9 to 3.1-5.2, Table 4). Absolute recall remains far below full-dataset training even at 1000 pairs. Text is distilled only as continuous BERT embeddings, not as readable tokens. No coreset-selection literature existed for this setting, so the authors constructed their own adapted baselines, which likely understate what a stronger coreset method could achieve.