Low-Rank Similarity Mining for Multimodal Dataset Distillation

What came before

MTT-VL was the first vision-language distillation method, but it (implicitly) treats each synthetic image as matching exactly one synthetic caption one-to-one, the same assumption single-modal trajectory matching makes about image-label pairs.

The problem

Real image-text corpora are not one-to-one: a caption can describe many images and an image can match several captions, and image-text contrastive (ITC) training depends on the whole similarity structure between a batch's images and texts, not just matched pairs. Distilling only paired samples throws away that structure.

The idea

Distill a full image-text similarity matrix alongside the synthetic pairs, so the synthetic set carries its own soft ground-truth affinities rather than an implicit identity matrix; make this affordable by factorizing the matrix into a diagonal plus a low-rank term instead of storing it densely.

How it works

Synthetic image pixels $\tilde{\mathcal X}$, synthetic text embeddings $\tilde{\mathcal Y}$, and a learnable similarity matrix $\tilde S=\omega I+\frac{\alpha}{r}LR^\top$ (diagonal $\omega$ plus rank-$r$ factors $L,R\in\mathbb{R}^{N\times r}$) are jointly optimized. An NFNet image encoder (pretrained, trained during distillation) and a BERT-base text encoder (pretrained, frozen for efficiency) plus a trainable projection produce embeddings. The outer objective is standard multi-step trajectory matching (MTT): $\|\tilde\theta_t-\theta_T^*\|^2/\|\theta_0^*-\theta_T^*\|^2$ on the image tower; the inner student loss on the synthetic batch is a weighted binary cross-entropy (wBCE) over $\tilde S$ rather than the usual InfoNCE with an implicit identity target, so the student is trained to reproduce the learned soft affinities, not just match its own pairs.

Evidence

With NFNet+BERT on Flickr30K, LoRS$_{\text{wBCE}}$ reports IR@1/5/10 = 8.3/24.1/35.1 and TR@1/5/10 = 11.8/35.8/49.2 at 100 pairs, rising to IR@1/5/10 = 10.0/28.9/41.6 and TR@1/5/10 = 15.5/39.8/53.7 at 500 pairs (Table 2). On COCO at 100/500 pairs, IR@1 = 1.8/2.8 and TR@1 = 3.3/5.3 (Table 3). Baselines (coreset methods, MTT-VL, and an author-adapted TESLA$_{\text{wBCE}}$) are re-run by the authors under the same protocol rather than copied. The key ablation (Table 5, 200 pairs) isolates the loss and the low-rank structure: replacing wBCE with plain BCE collapses IR@1 from 8.6 to 0.6; removing the similarity matrix entirely (identity matrix, i.e. classic one-to-one matching) drops performance further to IR@1 = 0.2; removing only the low-rank $L,R$ term (keeping diagonal $\omega$ only) gives IR@1 = 7.4, still above the identity-matrix ablation but below the full factorization (8.6), showing both the learned similarity structure and its low-rank correction matter. Storage and time overhead are small: with $r=20$-$50$ the similarity matrix adds about 0.07-0.3% extra storage and about 0.8% extra training time (6.35s/iteration vs 6.47s baseline at 100 pairs, Table 10), measured on one RTX 4090.

Limitations

The paper only distills text as continuous embeddings, not tokens, and states that direct distillation of text tokens is left to future work. It notes an explicit storage/performance trade-off in the rank $r$, and that the wBCE loss was chosen empirically and would need to be redesigned for other contrastive objectives. Absolute recall is still far below full-data training (e.g. IR@1 = 10.0 vs a full-dataset ceiling around 23-27 reported by later papers using the same protocol), and evaluation is limited to the NFNet+frozen-BERT encoder pair used during distillation.