Decoupled Audio-Visual Dataset Distillation

What came before

AVDD extended distribution matching to audio-visual data by summing per-modality DM losses and adding joint-matching and modality-gap-matching terms computed with randomly initialized visual/audio ConvNets (freshly re-sampled each iteration, never pretrained).

The problem

The paper diagnoses two problems specific to AVDD's random-network recipe: (1) independently and randomly initialized visual and audio encoders leave the two modalities in mismatched embedding spaces to begin with, which the paper argues complicates cross-modal matching before it even starts; and (2) enforcing cross-modal (inter-modal) matching directly on the same features used for intra-modal matching creates conflicting objectives, since a feature that helps align across modalities is not necessarily the feature that best represents that modality alone, and AVDD's single shared representation has no way to serve both roles without compromise - degrading modality-private information.

The idea

Use a bank of diverse, pretrained (not randomly initialized) audio-visual encoder pairs for stable, consistent feature spaces, and attach lightweight decoupler MLPs to each encoder that split its output into a shared subspace (used only for cross-modal matching) and a private subspace (used only for intra-modal matching), so the two objectives never compete for the same features; keep private representations entirely outside the cross-modal loss to protect modality-specific cues.

How it works

Synthetic 128x56 log-mel spectrograms (audio) and 224x224 video frames are directly optimized. A bank of $M{=}40$ pretrained encoder pairs (audio: 3-block ConvNet; visual: 5-block ConvNet), each independently trained on the full real dataset, is held frozen throughout distillation. Each encoder pair has $T{=}4$ decoupler instances (2-layer MLPs $g^a_{m,t}, g^v_{m,t}$ per modality) that are trained during distillation to project frozen-encoder features into a shared space $\mathbb R^{d_c}$ (6272-d). Private-representation matching (Eq. 12-14) applies per-modality DM (mean-feature matching) independently to each modality's frozen-encoder features. Common Intermodal Matching (Eq. 15-19) applies DM-style matching to the decoupled shared representations, both per-modality and jointly summed across modalities (as in AVDD's joint-matching term, but now on decoupled shared features rather than raw encoder features). Sample-Distribution Joint Alignment (Eq. 22-27) adds an intra-sample symmetric NT-Xent contrastive loss treating true audio-visual pairs as positives, plus a distribution-level term that aligns batch-mean shared features to cross-modal EMA class prototypes ($P^A_c,P^V_c$) by cosine distance. The decoupler MLPs themselves are trained with a separate decoupling loss (Eq. 21, 28: classification on shared representations plus inter- and intra-sample contrastive terms) so they learn a meaningful shared/private split rather than an arbitrary one. At evaluation, a fresh ConvNet (or alternative architecture) is trained from scratch on the synthetic data only, as in AVDD.

Evidence

Table 1 (5-run averages): VGGS-10K IPC1/10/20: DAVDD 41.9/56.2/59.1 vs AVDD 40.2/54.0/57.7 vs whole-data 68.2 (IPC10 column). MUSIC-21 IPC1/10/20: DAVDD 45.7/67.8/71.4 vs AVDD 44.2/66.9/69.7 vs whole-data 85.9. AVE IPC1/10/20: DAVDD 23.6/37.3/40.9 vs AVDD 22.7/35.9/40.0 vs whole-data 52.2. Gains over AVDD are consistently positive but modest: +0.9 to +2.2 points depending on dataset/IPC. Against the plain DM baseline the gap is larger (VGGS-10K IPC1: 41.9 vs 36.5, +5.4; MUSIC-21 IPC1: 45.7 vs 38.3, +7.4). Cross-architecture generalization (Table 4, VGGS-10K IPC1): DAVDD beats AVDD on ConvNet (41.9 vs 40.2), VGG11 (34.6 vs 34.0), and ResNet18 (31.0 vs 30.5) - all small margins. The key ablation (Table 2, VGGS-10K IPC10, additive from a DM baseline at 43.9): adding the pretrained encoder bank alone (in place of random encoders) reaches 49.2 (+5.3 over DM); adding the decoupler bank on top reaches 54.9 (+5.7 further, i.e. the largest single increment); adding Common Intermodal Matching reaches the full 56.2 (+12.3 total over DM). The pretrained-bank switch and the decoupling mechanism each contribute comparably large, roughly equal-sized gains. No GPU-hour, wall- clock or memory cost is reported anywhere in the paper (all experiments on one RTX A6000 Ada GPU).

Limitations

The paper states no explicit limitations section. Gains over AVDD are consistently small (0.9-2.2 points) despite the added machinery of 40 pretrained encoder pairs and 160 total decoupler MLPs (4 per pair), and no training-time, memory, or wall-clock comparison against AVDD's cheaper random-network recipe is given, so the added cost of maintaining and running the pretrained encoder bank is not weighed against the accuracy gain. The paper's own t-SNE visualization (their Appendix Figure 8) shows the improvement is more visible for the visual modality than for audio. The number of encoder pairs (40) and decouplers per pair (4) are not ablated or theoretically justified, appearing to be empirical choices. Evaluation covers only audio-visual event recognition (classification), unlike AVDD, which also reports retrieval results - DAVDD's paper does not include a retrieval evaluation for direct comparison on that axis.