Omnimodal Dataset Distillation via High-order Proxy Alignment

What came before

Bimodal trajectory-matching methods (LoRS, RepBlend) handle exactly two modalities, matching trajectories and a learned pairwise similarity structure. ImageBindDC condenses in a shared (ImageBind) embedding space but via distribution matching, not trajectory matching, and does not analyze how many modalities its objective can scale to.

The problem

The paper proves (their Lemma 1) that the trajectory-matching endpoint-discrepancy bound sums one mismatch term per modality, so going from $k{=}2$ to more modalities adds more terms to the bound without bounding the interaction *between* modalities. Naively extending bimodal pairwise trajectory matching to $k$ modalities by modeling every pair requires $\mathcal O(k^2)$ pairwise objectives and produces similarity tensors of size $\mathcal O(k^2N^2)$, which the paper calls computationally prohibitive and poorly scalable as more modalities are added.

The idea

Instead of modeling every modality pair, extract a single scalar "unified semantic feature" per instance - the leading singular vector of the SVD of the per-instance cross-modality Gram matrix - and use it as a compact proxy for the instance's full high-order cross-modal structure; this proxy slots into trajectory matching with minimal changes to the bimodal recipe (LoRS-style trajectory matching plus a learned similarity matrix) while avoiding the combinatorial pairwise blow-up.

How it works

Synthetic per-modality embeddings and a learnable similarity matrix $S_e$ are optimized via bi-level trajectory matching: $\mathcal L_{omni\text{-}traj}=\sum_{m\in\mathcal M}\|\theta^t_{e,m}-\theta_{T,m}\|_2^2/\|\theta_{0,m}-\theta_{T,m}\|_2^2$, matching expert trajectories (10 epochs of real-data training, 20 independent runs) per modality-specific projection head. A pretrained ImageBind encoder backbone is frozen throughout (only the projection heads are trained/matched). For each instance, a Gram matrix $G=zz^\top$ of its stacked modality embeddings is SVD-decomposed; by the Eckart-Young theorem (their Lemma 2) the rank-1 approximation $\tilde G=\sigma_1^2u_1u_1^\top$ is the optimal single-vector summary, and its leading singular vector $v_1$ (with $\sigma_1v_1=\sum_{m}[u_1]_m z_m$) becomes the "compact proxy" - instance-to-instance similarity is then approximated as $\tilde s(x_i,x_j)=v_1^{(i)\top}v_1^{(j)}$, a single scalar instead of $\mathcal O(k^2)$ pairwise terms. The total loss adds a modality-level contrastive term on the leading singular value ($\mathcal L_M=-\frac1N\sum_i\log\frac{\exp(\sigma_1^{(i)}/\tau)}{\sum_\ell\exp(\sigma_\ell^{(i)}/ \tau)}$, pulling the top singular value above the rest) to a weighted-BCE term enforcing consistency between $S_e$ and the proxy similarities. A spectral-perspective theorem (their Theorem 1) shows a single-mode (rank-1) objective has zero spectral weight on non-principal modes, while a full-spectrum pairwise objective does not, and that the single-mode per-step mismatch bound is dominated by (i.e. no worse than) the full-spectrum bound whenever non-principal modes carry error - the paper's formal argument for why the compact proxy does not lose the alignment that matters.

Evidence

Three omnimodal (video+audio+text) benchmarks: MSR-VTT, VGGSound-S, DiDeMo, evaluated by average R@1/R@5 retrieval. MSR-VTT@500 (Table 1): HoPA R@1/R@5 22.1/42.9 vs LoRS 21.8/42.4, vs full-data 22.6/44.8. VGGSound-S@500 (Table 2): HoPA 13.4/36.4 vs LoRS 13.3/35.6, vs full-data 13.7/37.7. DiDeMo@100 (Table 3): HoPA 18.4/37.2 vs LoRS (at 500, 5x the budget) 17.3/37.6 - HoPA beats LoRS's larger-budget number with a fifth of the pairs, the paper's strongest compression claim. Cost: despite added SVD overhead, HoPA distillation wall-clock is roughly half of LoRS's at comparable budgets (e.g. MSR-VTT@500: HoPA 57.6 min vs LoRS 120.4 min) because it distills all modalities jointly rather than pairwise. Cross-architecture transfer (Table 4): data distilled on ImageBind transfers well to OmniBind (R@1 13.13 vs ImageBind-source 13.15) but poorly to LanguageBind (R@1 5.93), a large drop. The key ablation (Table 5, MSR-VTT@500) compares the omniwise (joint) proxy against three pairwise variants: HoPA 22.25% vs a 3-pair baseline 18.11%, a text-anchored ("T-bind") variant 20.19%, and a vision-anchored ("V-bind") variant 19.85%, with pairwise variants degrading further as the budget grows while HoPA improves. A rank ablation (Table 6) shows rank-1 beats rank-2 (MSR-VTT R@1 22.25% vs 20.48%), supporting the single-leading-singular-vector design choice over a richer multi-mode proxy. An SVD-cost ablation (Table 7) reports the SVD/backprop-through-SVD overhead as 3.5% extra cost during expert-trajectory (buffer) generation but 39.1% extra during the distillation step itself, due to higher-order derivatives through the SVD.

Limitations

The paper states (Appendix F) that the underlying TESLA-style trajectory-matching algorithm has a non-trivial memory footprint that limits scalability to very large synthetic datasets. The 39.1% per-iteration overhead of differentiating through the SVD could become a bottleneck at higher embedding dimensions or larger modality counts $k$. The method assumes frozen pretrained encoders throughout and does not explore end-to-end fine-tuning. All experiments use exactly three modalities (video, audio, text); generalization to four or more modalities simultaneously, which motivates the paper's framing, is not empirically tested. Cross-architecture transfer to a different pretrained joint-embedding backbone (LanguageBind) shows a large accuracy drop relative to transferring to a closely related one (OmniBind), suggesting the proxy's benefit is tied to the specific embedding geometry it was distilled in.