Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation

What came before

LoRS and MTT-VL established trajectory matching for image-text pairs, with LoRS adding a distilled similarity matrix (via weighted BCE) so synthetic pairs carry soft ground-truth affinities instead of an assumed one-to-one match. Both keep separate, asymmetric treatment of the two modalities: the image tower's full encoder is matched while the text side is largely a frozen backbone plus a light projection.

The problem

The paper diagnoses "modality collapse": as distillation compresses data, intra-modal representations become increasingly concentrated (cosine similarity among synthetic embeddings of the same modality rises over training) while the cross-modal gap between image and text embeddings widens. It traces this to a structural conflict between dataset distillation's over-compression (which pulls a few synthetic points to represent many real ones) and contrastive cross-modal supervision (whose positive-pair gradients pull embeddings toward each other's modality, over-concentrating each modality's own spread) - formalized with a gradient analysis (their Eq. 3) and measured empirically (concentration ratio rising over 3000 iterations).

The idea

Counteract the contrastive objective's concentrating pressure by blending synthetic representations across instances (representation blending), which perturbs optimization away from collapse while preserving semantic structure; and replace prior methods' asymmetric trajectory matching (full image encoder vs. text projection only) with symmetric projection trajectory matching, matching lightweight projection heads on both modalities so neither is over- or under-supervised, which also removes the need to match a full encoder trajectory.

How it works

Synthetic image pixels, synthetic 768-d text embeddings, and soft pairwise labels are directly optimized parameters. Both an image encoder (NFNet/ResNet-50/ViT) and a text encoder (BERT/DistilBERT) are pretrained and kept frozen during distillation; only a linear projection head per modality is trained/matched. The base objective is the weighted binary cross-entropy (wBCE) of LoRS between predicted cosine similarities and soft labels. Representation blending (Eq. 5) linearly interpolates a synthetic embedding with a randomly shuffled instance's embedding ($\lambda\sim\text{Beta}(\alpha,\alpha)$) before the projection head, done independently per modality, to reduce intra-modal concentration. Symmetric projection trajectory matching (Eq. 6) matches the image-projection and text-projection parameter trajectories jointly and symmetrically after $T$ synthetic steps against $M$ real steps, normalized by the expert's total movement in both projections combined, replacing the asymmetric image-encoder/text-projection matching of MTT-VL and LoRS.

Evidence

With NFNet image encoder and BERT text encoder, Flickr30K at 100 pairs: IR@1/5/10 = 11.5/32.0/44.5, TR@1/5/10 = 16.2/41.7/55.5 (Table 1), versus LoRS's re-run 8.3/24.1/35.1 and 11.8/35.8/49.2 at the same budget; at 500 pairs IR@1/5/10 = 17.0/42.5/55.9, TR@1/5/10 = 22.5/53.2/66.7 versus LoRS 10.0/28.9/41.6 and 15.5/39.8/53.7. On COCO at 100/500 pairs (Table 2): IR@1 = 4.1/6.2, TR@1 = 5.2/7.0, versus LoRS 1.8/2.8 and 3.3/5.3. Baselines (coreset methods, MTT-VL, TESLA-VL, LoRS) are re-run under the same 20-expert-trajectory buffer and hyperparameters (Tables 5-6). Cost (Table 4, single comparison against LoRS): buffer generation 40 vs 70 min/trajectory (1.75x), buffer memory 0.73 vs 1.63 GB (2.23x), distillation step time 1.71 vs 11.5 s/iteration (6.7x), peak VRAM 10.17 vs 21.78 GB (2.14x). The key ablation (Figure 5) removes representation blending or symmetric projection matching individually; both individually degrade IR/TR at every budget on Flickr30K relative to the full method, with the combination needed for the best numbers. A cross-architecture ablation (Table 3, distilled on NFNet+BERT, evaluated on ResNet-50+BERT and RegNet+BERT at 500 pairs) shows RepBlend transfers better than LoRS, e.g. ResNet-50 IR@10 23.6% vs 20.4%.

Limitations

The paper states current MDD frameworks (including this one) remain pair-level: they do not model fine-grained token-to-object correspondence within a pair or cross-instance interactions beyond the blending step. All experiments use frozen, ImageNet/BERT-pretrained encoders, so results are a property of that backbone pair as much as of the method. Evaluation is centered on Flickr30K and MS-COCO; an audio-text (AudioCaps) extension is included but only briefly. An appendix comparison to a unimodal decoupled method (SRe2L) on ImageNet-100 shows a very large gap that the authors note is not a fair comparison since the setups differ. The claimed 6.7x speedup is for the distillation step only, not buffer/expert-trajectory generation, which is cut by a smaller 1.75x.