What changes when there are no classes
Class-conditional condensation gets structure for free: statistics, gradients and trajectories are matched class by class, and budgets are counted per class. An image–caption corpus has no classes. The distilled set has to approximate a joint image–text distribution, and the model trained on it is contrastive, so the value of a pair depends on the other pairs it is contrasted with. Three consequences shape the whole setting:
- The correspondence between images and captions is itself information that condensation can lose.
- Text is almost always distilled as continuous token embeddings, not readable sentences.
- There is no shared evaluation recipe (see below), so results accumulate across papers far less reliably than in image classification.
Four bets, and what each generation fixed
Trajectory matching. MTT-VL opened the setting in 2023 by matching expert trajectories of an image encoder and a text encoder jointly under a contrastive loss, with hard one-to-one pairs. Each successor fixed the previous failure:
- LoRS observed that a caption can describe many images and learned a low-rank similarity matrix alongside the pairs, for well under 1% extra storage.
- RepBlend then named modality collapse. Under heavy compression, contrastive supervision concentrates each modality’s representations while widening the gap between them. RepBlend countered it with representation blending and symmetric projection matching, and cut distillation time per iteration about 6.7× relative to LoRS.
- PTM-ST replaced a single unstable trajectory with phase-specific teachers and a smoothed shortcut trajectory. Its ablation finds the shortcut trajectory contributes the most.
Distribution matching. The next group argued that the expensive part was trajectory replay itself, not optimization.
- CovMatch matches the cross-covariance of image and text features with encoders fixed per step. It needs no expert trajectories (against 132 GPU-hours and 120 GB of expert storage for MTT-style methods), which makes a trainable text encoder affordable. Its ablation is the cleanest causal result in the setting: freezing BERT during distillation drops Flickr30K mean recall at 500 pairs from 38.4 to 29.4.
- MDM initializes pairs from clusters of the joint embedding space and replaces trajectories with teachers interpolated in weight space, reporting 93–98% less distillation time than LoRS.
- RAHA argues that shared image–text semantics occupy a low-rank subspace and aligns it in hyperbolic space. It is stronger than its CovMatch reproduction at 200 and 500 pairs, weaker at 100, and much slower per iteration at large batch size.
Generative synthesis. EDGE builds correspondence into the generator. It fine-tunes Stable Diffusion with contrastive and diversity losses, then samples pairs. Its total pipeline is about 20× cheaper than LoRS on COCO (7.7 vs 151.6 GPU-hours at 500 pairs), but it does not consistently beat LoRS on recall (Flickr30K, 500 pairs: image→text R@1 13.3 vs 15.5; text→image R@1 6.7 vs 10.0).
Training-free synthesis. PDS optimizes nothing. It clusters CLIP image and text embeddings, matches the two sets of prototypes, and decodes images with unCLIP, at about 10 seconds per synthesized image.
Where each method establishes correspondence
| Where | Methods |
|---|---|
| In the distillation objective | MTT-VL (contrastive loss inside trajectory matching), MDM (InfoNCE plus hypersphere matching) |
| In a stored similarity structure | LoRS, RepBlend, PTM-ST |
| In the feature geometry that is matched | CovMatch (cross-covariance), RAHA (shared low-rank subspace, hyperbolic) |
| In a fine-tuned generator | EDGE |
| After the fact, by matching per-modality prototypes | PDS |
Reading results: three incompatible protocols
The papers fall into at least three evaluation recipes, and numbers should only be compared within one.
| Protocol | Image encoder at evaluation | Text encoder | Papers |
|---|---|---|---|
| NFNet, frozen BERT with a trained projection | pretrained, trained | BERT-base, frozen | LoRS, RepBlend, EDGE, PTM-ST |
| NFNet, trainable BERT | pretrained, trained | BERT-base, trained | CovMatch, RAHA |
| CLIP text tower with other image backbones | ResNet-50 / ViT-Ti, trained | CLIP, frozen | PDS |
Within the frozen-BERT protocol on Flickr30K at 500 pairs, text→image R@1 is 10.0 for LoRS, 17.0 for RepBlend and 16.0 for PTM-ST (each paper’s own table). Even this is imperfect: RepBlend’s re-run of LoRS reproduces LoRS’s published numbers exactly, while PTM-ST’s re-run gives different ones (12.7 instead of 10.0), and no paper documents why. PDS compares against LoRS and TESLA-VL re-implemented with a CLIP text tower, so its table is internally consistent but cannot be placed next to the others.
Where the setting stands
- Accuracy. Under shared protocols, trajectory-matching refinements (RepBlend, PTM-ST) and distribution matching with a trainable text tower (CovMatch, RAHA) report the highest recall. Recall remains far below full-data training at every budget reported.
- Cost. Expert-trajectory replay no longer looks necessary. CovMatch and MDM remove most of it while matching or beating LoRS, and EDGE and PDS remove optimization almost entirely, at an accuracy cost that current protocols cannot measure cleanly.
- Scale. No image–text method has condensed beyond roughly a thousand pairs of Flickr30K or COCO in a way that holds up. EDGE’s CC3M result at 1,000 pairs sits near the random floor. Multimodal condensation has not yet had the decoupling moment that took image classification to ImageNet-1K.
The open problems on incompatible protocols and scale follow from this. The other meaning of “multimodal”, where a vision–language model is a tool for condensing ordinary image datasets (VLCP, EVLF, CoDA), is covered under image classification.