Audio-Visual Dataset Distillation

What came before

Distribution matching (DM) minimizes the feature-distance between real and synthetic data using randomly initialized networks, applied so far only within a single modality (image classification). MTT (trajectory matching) is used as a second baseline. No prior method addressed paired audio-visual data.

The problem

A naive extension of DM to audio-visual data trains a visual-only DM loss and an audio-only DM loss independently and sums them ($\mathcal L^{av}_{base}=\mathcal L^a_{base}+\mathcal L^v_{base}$). The paper diagnoses that this cannot capture cross-modal correspondence: because the visual network $\psi_{\theta_v}$ and audio network $\psi_{\theta_a}$ are separate, randomly initialized networks with no shared structure, matching each modality's distribution separately creates a "modality gap" - there is nothing in the objective that ties a synthetic audio sample to its paired synthetic visual sample the way real audio-visual pairs are tied together.

The idea

Add two loss terms on top of per-modality DM that explicitly couple the two modalities: a joint matching loss that matches the *sum* of real audio and visual features against the sum of synthetic audio and visual features (which, when expanded, implicitly cross-matches real audio against synthetic visual features and vice versa), and a modality-gap matching loss that aligns the *difference* between audio and visual features so the audio-visual gap itself is consistent between real and synthetic data - both computed via MMD-style feature-mean matching using randomly initialized visual/audio ConvNets.

How it works

Synthetic video frames ($224\times224$) and audio spectrograms are directly optimized pixels, one set per class. A randomly initialized visual ConvNet $\psi_{\theta_v}$ and randomly initialized audio ConvNet $\psi_{\theta_a}$ (fresh random weights sampled per iteration, as in DM; not trained across iterations) extract features under differentiable Siamese augmentation $\mathcal A_\omega(\cdot)$. The base loss (Eq. 1) sums per-modality DM losses. Joint matching (Eq. 2) matches $\mathcal D^r=\bar R^a+\bar R^v$ against $\mathcal D^s=\bar S^a+\bar S^v$ (sums of mean real/ synthetic audio and visual features), which algebraically bounds $\|(\bar R^a-\bar S^v)\|^2+\|(\bar R^v-\bar S^a)\|^2$ - an implicit cross-modal match. Modality-gap matching (Eq. 3) instead matches $\mathcal D^{av}=\bar R^a+\bar S^v$ against $\mathcal D^{va}=\bar R^v+\bar S^a$, which reduces to matching the real audio-visual gap $(\bar R^a-\bar R^v)$ against the synthetic gap $(\bar S^a-\bar S^v)$. The final loss (Eq. 4) sums base + weighted JM + weighted MGM. Synthetic data is initialized via herding (coreset selection) rather than random real samples, and a "factor" technique increases the number of extracted features per stored sample at no extra storage cost. At evaluation, fresh visual and audio ConvNets are trained from scratch on the synthetic set for 30 epochs (5000 iterations/run, 3 runs), fused by late-fusion ensembling (shown to beat concatenation, summation, and attention fusion).

Evidence

On VGGS-10K (10-class VGGSound subset) at IPC=10 (Table 3): audio-visual ensemble accuracy 54.99$\pm$1.73% versus DM 43.85$\pm$1.75%, MTT 36.79$\pm$1.97%, herding 39.89$\pm$1.64%, random 32.01$\pm$1.64%, and whole-data upper bound 68.24$\pm$0.75%. At IPC=1: 40.41$\pm$1.81% vs DM 36.54$\pm$2.52%; at IPC=20: 58.04$\pm$1.68% vs DM 49.01$\pm$2.44%. On AVE at IPC=10: 36.82$\pm$0.88% vs DM 28.14$\pm$1.80%, whole-data 52.20$\pm$0.38%. Audio-visual retrieval (Table 5, IPC=20, class-wise retrieval on 5-samples-per-class test subsets): A$\to$V R@1 19.33$\pm$2.30% vs DM 8.66$\pm$1.15% vs whole-data 44.00$\pm$2.00%; V$\to$A R@1 27.33$\pm$2.30% vs DM 11.33$\pm$3.05% vs whole-data 45.33$\pm$5.03%. The key ablation (Table 4, IPC=10, additive from a base DM run) shows: random-init 32.01% (VGGS-10K) / 20.00% (AVE); switching to herding init 39.89%/26.86%; adding the factor technique 40.28%/31.80%; adding the base audio-visual DM loss 45.31%/34.80%; adding joint matching (JM) 49.07%/35.13%; adding modality-gap matching (MGM) on top reaches the full 54.99%/ 36.82% - every added component improves both datasets, with MGM and JM together contributing the largest joint increment over the herding+factor+base configuration. No GPU-hour/wall-clock cost is reported in this (4-page workshop) version.

Limitations

This full-text read is the 4-page SightSound 2024 workshop precursor rather than the TMLR 2024 journal version; the journal abstract indicates MUSIC-21 and fuller VGGSound evaluations were added later and are not covered by this reading. Retrieval evaluation is class-wise (5 samples per class) rather than instance-level, since the paper states its DM-based objective targets semantic, not instance-level, alignment. AVE at IPC=20 could not scale up for the MTT baseline ("-" in Table 3), limiting the baseline comparison at that budget. Audio and visual data are distilled and evaluated as one-second clips with a single center frame plus a spectrogram, not full video; both modality networks are randomly initialized at each iteration rather than pretrained, so the method's dependence on that specific (untrained-network) DM recipe is untested against a pretrained-encoder alternative.