ImagebindDC: Compressing Multi-modal Data with Imagebind-based Condensation

What came before

AVDD (and DAVDD) match feature distributions in separate, modality-specific spaces built from audio/visual-only backbones; LoRS reduces the full image-text relationship to a single learned scalar similarity per pair; RepBlend heuristically blends representations to fight modality collapse without a guarantee of preserving joint-modal semantics. Unimodal methods (DM, DSA, MTT) have no mechanism at all for keeping a specific sample's modalities linked together.

The problem

The paper diagnoses that matching each modality's distribution independently (as unimodal methods and AVDD-style per-modality DM do) "critically fails to capture the cross-modal relationships that link the data together" - it can produce synthetic data that is statistically representative within each modality while breaking the pairing between, say, a specific image and its corresponding sound; and that kernel-based statistics like MMD (used by DM/AVDD) only approximate a finite number of moments, leaving room for distributional mismatch that a higher-fidelity statistic could catch.

The idea

Condense inside ImageBind's single, already-aligned joint embedding space (rather than separate per-modality spaces built from scratch), and replace MMD-style feature-mean matching with a Characteristic Function loss that matches the full (Fourier-domain) characteristic function of real versus synthetic embedding distributions - equivalent to matching infinitely many moments rather than just means/variances - applied simultaneously at three levels: within each modality, between real-synthetic hybrid pairs across modalities, and over the full joint multi-modal distribution.

How it works

Synthetic raw data per modality (image pixels, raw audio waveforms, depth maps, text representations) are directly optimized. A frozen, pretrained ImageBind encoder maps every modality into a shared 1024-d embedding space; no encoder is trained during distillation. The Characteristic Function Discrepancy (CFD, Eq. 3) computes an L2 distance between the empirical characteristic functions (Fourier transforms of the embedding distributions) of real versus synthetic embeddings, capturing magnitude and phase information rather than only a kernel-weighted mean difference as MMD does. Three losses use CFD: uni-modal alignment (Eq. 5-7) applies CFD separately per modality ($\mathcal L_{uni}=CFD(e_a,\tilde e_a)+CFD(e_v,\tilde e_v)$); cross-modal alignment (Eq. 8-11) forms real and synthetic hybrid pair products ($e_a\odot e_v$ vs. $\tilde e_a\odot\tilde e_v$) and matches them by cosine similarity ($\mathcal L_{cross}=1-\rho_{cross}$); joint-modal alignment (Eq. 12-13) forms batch-mean cross-products of the two modalities' means and matches them similarly ($\mathcal L_{joint}=1-\rho_{joint}$). The total loss (Eq. 14) is a weighted sum $\lambda_{uni}\mathcal L_{uni}+\lambda_{cross}\mathcal L_{cross}+\lambda_{joint}\mathcal L_{joint}$ (all weights fixed at 1.0 in the paper's experiments). At evaluation, a classifier is trained from scratch on the condensed data (with ImageBind's encoder still frozen as the feature extractor); no classifier is trained during the condensation phase itself.

Evidence

NYU-v2 depth-text classification (Table 3): ImageBindDC 80.43/97.30/98.73% at 1/5/10 datapoints-per-class (DPC) vs AVDD 72.22/95.92/98.62% and DM 67.97/89.08/96.89%, vs a 98.62% full-data ceiling - lossless performance is reached at 5 DPC (97.30% vs 98.62% full-data), and the 8.2-point headline gain is 98.73% (ImageBindDC@10 DPC) minus DM's 96.89% at the same budget (not versus AVDD, whose gap at 10 DPC is only 0.11 points). VGGS-10K (Table 2, ImageBind-guided condensation) at 10 DPC: ImageBindDC 56.11% vs AVDD 48.89%, DM 45.46%, vs a 60.57% full-data ceiling. AVE (Table 1) at 20 DPC: ImageBindDC 52.20% vs AVDD 38.41%, MTT 32.57%, herding 33.04%. Clotho audio-text retrieval (Table 4, 20 DPC): A2T@1 ImageBindDC 0.0362 vs AVDD 0.0316 vs DM 0.0268 vs full-data 0.0526; T2A@5 ImageBindDC 0.1567 vs AVDD 0.1330 vs DM 0.1148 vs full-data 0.1674. Cost (Table 6, VGGS-10K, single A100): at 20 DPC, ImageBindDC takes 123.74s/13.39GB versus AVDD's 700.1s/14.24GB (about 5.7x faster) and DM's 707.21s/14.24GB; the paper's "4x less condensation time" and "75% memory savings" headline figures are computed against a full-data training-epoch baseline (419.9s) rather than against AVDD specifically. The key ablations show: (Figure 3) the CF loss beats MMD-based DM matching consistently across settings (e.g. 1 DPC audio-only: 32.33% vs 27.87%; 10 DPC audio+video: 70.34% vs 69.26%); (Figure 4, AVE, additive from uni-modal-only 70.34%) adding joint-modal reaches ~71.5%, adding cross-modal reaches ~72.5%, and all three combined reach 73.67% (a 3.33-point gain over uni-modal alone) - the paper explicitly notes the three terms are synergistic rather than simply additive.

Limitations

The paper states no explicit limitations section. The entire framework depends on a frozen, pretrained ImageBind encoder, so its applicability to modalities or domains poorly represented in ImageBind's own pretraining (e.g. specialized medical imagery or rare audio domains) is untested. Despite being framed as "omnimodal," the reported experiments test at most 2-3 modalities jointly per benchmark (audio+video, depth+text, audio+text) - true joint condensation over 4+ modalities simultaneously is not demonstrated. Loss weights ($\lambda_{uni},\lambda_{cross},\lambda_{joint}$) are fixed at 1.0 throughout and not individually ablated. The paper's own cross-architecture generalization table (Table 5) reports a substantially lower number for at least one architecture/budget combination than the same architecture achieves elsewhere in the paper, suggesting a possible overfitting-to-ImageBind's-embedding-structure effect that the paper does not discuss. The joint-modal alignment's specific matrix-multiplication-of-means construction (Eq. 12) is not justified against alternative joint-distribution statistics (e.g. joint CFD, sliced Wasserstein distance). Only qualitative (not quantitative, e.g. no LPIPS/FID) evidence is given for the claim that distilled images preserve better visual/semantic coherence than AVDD's.