Multimodal Distribution Matching for Vision-Language Dataset Distillation

What came before

MTT-VL and LoRS both distill vision-language pairs via bi-level trajectory matching, repeatedly generating and matching teacher/student parameter trajectories from real and synthetic data.

The problem

The paper diagnoses two costs of trajectory matching: it requires "substantial computational and memory overhead" from repeated bi-level optimization with nested gradients over stored expert trajectories, and it biases the synthetic data toward the specific architecture whose training dynamics are being matched, which the paper connects to trajectory-matching methods' weak cross-architecture generalization; the paper also argues prior methods do not explicitly preserve the joint image-text distributional structure (both modality-shared and modality-specific information) under tight pair budgets.

The idea

Replace trajectory replay with direct distribution matching in a joint embedding space: initialize synthetic pairs by clustering the joint (concatenated image+text) embedding space of the real data rather than randomly, build a single "mixed teacher" by interpolating independently fine-tuned experts in weight space (weighted by how much they agree, so disagreement falls back toward the pretrained anchor), and match real versus synthetic distributions using a geometry-aware kernel that operates on the unit hypersphere along both a cross-modal agreement direction and a discrepancy direction, instead of matching means as in plain distribution matching (DM).

How it works

Synthetic image pixels ($3\times224\times224$) and 768-d text embeddings are optimized. A frozen encoder $\Psi$ (NFNet image + BERT text + projection heads) provides features but is re-initialized per iteration by weight-space interpolation of $N{=}2$ fine-tuned experts and a pretrained anchor $\theta_0$ (Eq. 5): per layer, the merge coefficient $t_\ell$ is set by the cosine-similarity-derived angular agreement between the two experts' displacements from the anchor, so more agreement gives more weight to the merged expert direction and more disagreement falls back toward the shared pretrained anchor. Synthetic pairs are initialized (Eq. 4) by k-means clustering the real data's joint embeddings $[z_v;z_t]$ into $K=|\mathcal D_{syn}|$ clusters and picking, per cluster, the real pair closest to the centroid. The matching objective (Eq. 6-11) forms an agreement direction $u=\text{normalize}(z_v+z_t)$ and a discrepancy direction $g=\text{normalize}(z_v-z_t)$ per pair, then computes a Geodesic Kernel Energy (an MMD-style two-sample statistic using an angular/geodesic Gaussian kernel $k_{geo}(a,b)=\exp(-\phi(a,b)^2/2 \sigma^2)$ with $\phi$ the arccos angular distance) between the real and synthetic sets' agreement distributions ($\mathcal L_{agr}$) and discrepancy distributions ($\mathcal L_{dis}$) separately; the total loss $\mathcal L_{MDM}=\mathcal L_{InfoNCE}+\lambda_{agr}\mathcal L_{agr}+\lambda_{dis}\mathcal L_{dis}$ adds a standard bidirectional InfoNCE term to keep synthetic pairs themselves cross-modally aligned. Both encoders are used only as fixed (per-iteration) feature extractors during distillation - there is no expert-trajectory replay. At evaluation, a freshly initialized model is trained on the synthetic pairs for 100 epochs.

Evidence

NFNet+BERT protocol, Table 1. Flickr30K@100: MDM (Ours) IR@1/5/10 8.1/24.7/36.2, TR@1/5/10 11.5/32.6/45.0, vs LoRS 8.3/24.1/35.1, 11.8/35.8/49.2 (roughly matched on IR@1/TR@1, ahead on IR@5/10, behind on TR@5/10) and vs MTT-VL 4.7/15.7/24.6, 9.9/28.3/39.1. @200: MDM IR 9.1/26.7/39.1, TR 13.0/33.7/47.4 vs LoRS IR 8.6/25.3/36.6, TR 14.5/38.7/53.4. @500: MDM IR 10.0/29.3/42.0, TR 13.7/37.0/51.5 vs LoRS IR 10.0/28.9/41.6, TR 15.5/29.8/53.7 (near-tied on IR, LoRS ahead on TR@1/10). COCO@100: MDM IR 1.9/7.6/13.2, TR 3.6/13.7/21.6 vs LoRS IR 1.8/7.1/12.2, TR 3.3/12.2/19.6. @500: MDM IR 3.7/13.6/22.2, TR 5.6/18.4/28.2 vs LoRS IR 2.8/9.9/16.5, TR 5.3/18.3/27.9. Cross-architecture generalization (Table 2, distilled on NFNet+BERT, evaluated on other encoders, Flickr30K@100 mean recall): MDM 16.1 vs LoRS 9.9; COCO@100: MDM 6.1 vs LoRS 2.5. Cost (Table 3, Flickr8K): distillation time falls from LoRS's 76.9/137.5/206.4 minutes (100/200/500 pairs) to MDM's 5.7/7.97/3.68 minutes (93-98% less), and MDM converges in roughly 50-200 iterations versus LoRS's 850-2350 (Fig. 4). The key ablations (Table 4a-b, Table 5, Flickr8K@100) isolate three mechanisms: joint-space k-means initialization beats random-sample, image-only, or text-only clustering by about 1.1-1.3 mean-recall points (21.9 vs 20.6-20.8); angle-guided weight-space expert merging beats a naive random weighted sum by 2.3 points (21.9 vs 19.6) and beats using only a single fine-tuned expert (17.8) or only the pretrained anchor (6.8) by much larger margins; and adding the discrepancy loss $\mathcal L_{dis}$ contributes more than the agreement loss $\mathcal L_{agr}$ alone (21.53 vs 21.02 over the InfoNCE-only baseline's 20.98), with both together giving the best result (21.94).

Limitations

The paper states MDM still assumes access to pretrained image and text encoders (and, for the mixed-teacher mechanism, multiple independently fine-tuned experts), so it does not remove that dependency, only the trajectory-storage cost built on top of it. At the 200-pair Flickr30K budget MDM's mean text-retrieval numbers trail LoRS's (e.g. TR@10 47.4 vs 53.4), a result the paper acknowledges without fully explaining. Distilled text remains 768-d embeddings rather than readable captions. Distilled images show the same noise-like high-frequency artifacts reported in prior distillation work. The method's teacher-interpolation mechanism depends on the quality and diversity of the fine-tuned experts used, a dependency the paper does not quantify.