Multimodal Dataset Distillation via Phased Teacher Models

What came before

MTT-VL and LoRS both match a single expert trajectory (a fixed sequence of teacher checkpoints trained once on real data) to guide synthesis, LoRS adding a learned similarity matrix on top. The paper reports that simply importing dynamic/expanding-window trajectory matching from unimodal distillation (DATM-style) does not fix multimodal distillation: applying it directly slightly hurts rather than helps (COCO IR@1 goes from 9.0 to 9.1 in the wrong direction in their Table 8 ablation), so the fix has to be specific to the multimodal setting.

The problem

The paper diagnoses two problems with single-trajectory teacher matching specific to multimodal data: (1) a pronounced cross-stage performance gap - although the teacher continuously improves across training epochs, matching a late-stage teacher checkpoint (e.g. epoch 10) causes student performance to collapse (their Table 7: student score drops from 45.6 to 30.1 when moving from an early to a late teacher checkpoint, versus a much smaller 45.6-to-33.8 drop for an early checkpoint), unlike unimodal distillation, where most of the benefit also concentrates early but without such a sharp late-stage collapse; and (2) unstable teacher trajectories - gradient norms grow steadily during teacher training (expected) but gradient *directions* oscillate with "no discernible pattern" across epochs (their Figure 6a), which they attribute to the sparsity of multimodal data and the absence of an explicit semantic constraint forcing the teacher to encode substantially different knowledge at different stages.

The idea

Instead of forcing all synthetic pairs to match one (unstable, non-uniformly-useful) trajectory, split distillation into sequential phases, each with its own teacher-checkpoint range and its own synthetic subset (phased teacher modeling); and within each phase, replace the noisy original trajectory segment with a smoothed "shortcut" trajectory - a straight-line interpolation between the phase's two endpoint checkpoints, weighted by accumulated real-trajectory parameter distance - which provably bounds gradient-direction oscillation and stabilizes the match.

How it works

Per phase $p$, a synthetic image subset, synthetic text embeddings, and a learnable (full, not low-rank) similarity matrix are optimized against a phase-specific range of teacher checkpoints $\{T_p^-,\dots,T_p^+\}$ (Eq. 2-3), rather than one trajectory for the whole distillation. The shortcut trajectory (Eq. 4-5) replaces the original checkpoint sequence $\{\theta_0,\dots,\theta_n\}$ within a phase with an interpolation $\theta_t^p=(1-\beta_p(t))\theta_0+\beta_p(t)\theta_{t_p}$, where $\beta_p(t)$ is the fraction of accumulated real-trajectory movement (computed layer-wise) reached by step $t$; Proposition 1 bounds the gradient difference between the shortcut and the original trajectory linearly in the interpolation step, formalizing the claimed stabilization. An EMA (decay 0.99) further smooths the synthetic data across iterations (Eq. 7). The image encoder (NFNet, pretrained) is trained during both distillation (matched) and evaluation; the text encoder (frozen pretrained BERT) has only its projection head trained in both phases. At evaluation, the student is trained progressively on the concatenation of phase subsets $\tilde D_1\cup\dots\cup\tilde D_P$ in order.

Evidence

NFNet+BERT protocol. Flickr30K@100 (Table 1): IR@1 9.6, TR@10 52.7, vs their own re-run of LoRS at IR@1 7.8, TR@10 44.9 (note: this LoRS re-run differs from lors-2024's own reported IR@1=8.3 at 100 pairs - both numbers are reported in the literature for the same nominal baseline/budget). @500: IR@1 16.0, IR@10 54.0, TR@1 22.2, TR@10 64.6, vs their LoRS re-run IR@1 12.7, IR@10 44.9, TR@1 14.7, TR@10 51.1 (again differing from lors-2024's own Table 2 IR@1=10.0/TR@1=15.5 at 500). Full-data teacher ceiling: IR@1 27.3, TR@10 75.2. COCO@100 (Table 2): IR@5 9.0 vs LoRS re-run 6.9; @500: IR@10 30.7, TR@10 30.0 vs LoRS re-run IR@10 19.2, TR@10 20.2. At 1000 pairs (Table 6), PTM-ST continues improving (Flickr30K IR@1 17.5, TR@10 67.2; COCO IR@1 7.0, TR@10 30.8) while their LoRS re-run saturates or degrades (Flickr30K IR@1 9.8, TR@10 53.5; COCO IR@1 2.5, TR@10 24.1). A large-scale extension to LLaVA-CC3M (595K pairs, Table 3) at 500 pairs shows roughly a 66% relative gain in an averaged 5-metric score over their LoRS re-run. The key ablation (Table 4, 500 pairs, additive) starts from a no-PTM/no-ST baseline (Flickr IR@1 12.2, COCO IR@1 3.4), and shows the shortcut trajectory (ST) alone gives the largest single jump (Flickr IR@1 to 14.2), phased teacher modeling (PTM) adds incrementally on top (to 15.4 combined with ST), and EMA contributes a small final refinement (to 15.5 full method) - ST is reported as the most impactful single component. Downstream transfer (Table 5): COCO-VQA accuracy improves from 10.8% (LoRS) to 16.3% (PTM-ST); zero-shot ImageNet-50 accuracy from 18.6% to 22.1%. Experiments run on a single RTX 3090.

Limitations

The paper states PTM-ST requires manually specifying phase boundaries (interpolation endpoints and matching ranges per stage), calling this an "excessive parameterisation" burden and proposing adaptive phase selection as future work; the number of phases P is set per budget without a principled selection procedure (P=2 for smaller budgets, more for larger ones, per the paper's own description). Proposition 1 only bounds the *gradient difference* between the shortcut and original trajectories, not the optimality of the interpolation weighting itself. Distilled images show the same "DeepDream-like" texture artifacts common to trajectory-matching distillation. Cross-architecture testing covers NFNet/RegNet/DistilBERT but no vision transformer backbone. Directly importing unimodal dynamic trajectory matching (DATM) into this setting was tried and found not to help (Table 8), showing the phased/shortcut mechanism is not a generic drop-in but specific to the diagnosed multimodal instability.