CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder

What came before

MTT-VL and LoRS both apply trajectory matching to image-text pairs but freeze the pretrained text encoder during distillation, training only a projection layer and (in LoRS) the image encoder, following the multimodal-DD convention set by MTT-VL that a full text-encoder trajectory is too costly to match.

The problem

The paper diagnoses that a frozen text encoder plus a linear projection lacks the expressiveness to achieve real cross-modal semantic alignment: captions of the same image do not cluster tightly in the resulting embedding space (their Figure 2), and retrieval performance saturates and even degrades below random sampling as the pair budget grows past about 1,000 pairs under this constraint.

The idea

Make both encoders trainable, but sidestep the cost of unrolled trajectory matching (which made a trainable text encoder computationally prohibitive, ~120GB/132 GPU-hours for multimodal MTT) by fixing both encoders within each distillation step and matching the cross-covariance of image and text features (plus per-modality feature regularization) instead of matching multi-step parameter trajectories; under a high-temperature linearization of the contrastive loss the per-step objective becomes closed-form, removing the need for unrolled backpropagation through training.

How it works

Synthetic image pixels and text token embeddings are directly optimized. A pretrained NFNet image encoder and a pretrained BERT-base text encoder are held fixed within each distillation step but periodically updated with one gradient step on real data (and reset to pretrained weights every T=50 steps); their projection heads are optimized implicitly through the closed-form per-step solution rather than matched via an unrolled trajectory. The loss (Eq. 13) is $\mathcal L^{\text{CovMatch}}=\mathcal L^{\text{cov}}+\lambda(\mathcal L^{\text{feat}}_v+\mathcal L^{\text{feat}}_l)$: a cross-covariance alignment term $\|\rho\cdot C^{\mathcal T}-C^{\mathcal S}\|_F^2$ matching the cross-covariance between image and text features of real (T) versus synthetic (S) data (with a scale correction $\rho$ for the size mismatch between the real and synthetic sets), plus an L2 term matching per-modality mean projected features between real and synthetic data. At evaluation, both encoders and their projections are fine-tuned end to end on the synthetic set for 100 epochs.

Evidence

NFNet image encoder, BERT-base text encoder (trainable). Flickr30K@100 (Table 2): IR@1/5/10 10.1/28.6/40.9, TR@1/5/10 14.8/38.0/50.6, mean recall 30.5, vs LoRS re-run 8.3/24.1/35.1, 11.8/35.8/49.2, mean 27.4. @500: IR 14.7/38.4/51.4, TR 19.9/46.7/59.5, mean recall 38.4, vs LoRS re-run 10.0/28.9/41.6, 15.5/39.8/53.7, mean 31.6 (a 6.8-point absolute gain). COCO@500: IR 5.4/18.0/28.2, TR 8.1/23.5/34.6, mean recall 19.6, vs LoRS re-run mean 13.5 (a 6.1-point gain). Baselines (random, herding, k-center, MTT-VL, LoRS) are re-run under the same protocol, averaged over 5 runs. Cost (Table 1): the multimodal-MTT baseline needs 132 GPU-hours of expert training plus 120GB expert-trajectory storage plus 71GB distillation memory; CovMatch needs 0 hours of expert preparation, 0GB storage, 15GB distillation memory, 1.2s/iteration. The key ablations (Table 9, Figure 5c) isolate two mechanisms: freezing the text encoder during distillation (i.e. reverting to the LoRS-style protocol) drops Flickr30K@500 mean recall from 38.4 to 29.4 (about a 29% relative loss); and disabling the "online model update" (periodically refreshing the fixed encoders with a real-data gradient step) costs about 6 points, with updating on synthetic data instead giving worse results than either. A cross-architecture ablation (Table 3, distilled on NFNet+BERT, evaluated on NF-ResNet/NF-RegNet/ViT image encoders and DistilBERT) shows CovMatch reaches 15.5-16.1 average recall on unseen encoders versus 7-9 for MTT-VL/LoRS.

Limitations

The method assumes pretrained image and text encoders are available in the first place and does not address distilling data for from-scratch pretraining. The closed-form step relies on a high-temperature linearization of the contrastive loss, an approximation that may diverge from the low-temperature InfoNCE ($\tau\le0.07$) used in practice. The periodic reinitialization/online update of the "fixed" encoders introduces some instability, since the synthetic data is optimized against a moving target. Absolute retrieval remains far below full-data training even at 500 pairs (COCO IR@1 5.4% vs a full-data ceiling around 25.1% reported in the paper's own table).