Dataset Condensation Atlas

Settings · multimodal

Audio–visual and omnimodal data

Condense datasets whose samples span audio, vision and sometimes further modalities, while preserving the correspondence between them.

One synthetic sample

A labeled (visual frame, audio) pair, or a tuple of modality embeddings in a shared bound space.

Standard evaluation

Train an audio–visual classifier or retrieval model on the synthetic set and evaluate on the real test split.

Budget

samples per class

Benchmarks

VGGSound subsets · AVE · Music-21 · AudioSet subsets

The same problem, compressed into four papers

Audio–visual condensation faces the vision–language problem in a sharper form. Each sample pairs a visual frame with an audio clip, the pairing is what a joint model learns from, and matching each modality’s distribution separately can keep both marginals while losing the correspondence between them. The literature is small, but it has already followed the arc the larger settings took over several years.

  1. Explicit cross-modal terms on random networks. AVDD extends distribution matching to audio and vision with two added losses: one matching the joint distribution, one matching the gap between modalities. Evaluation networks are trained from scratch. On VGGSound-10K at 10 samples per class it reports 55.0%, against 43.9% for plain distribution matching and 68.2% with the full training set.
  2. Pretrained encoders and a shared/private split. DAVDD replaces random networks with a bank of frozen pretrained audio–visual encoders and separates shared from modality-private features. Its ablation credits roughly equal gains to the pretrained bank and to the decoupling.
  3. Condense inside an aligned space. ImageBindDC skips bespoke matching networks and condenses directly in ImageBind’s pretrained joint embedding space, matching characteristic functions. It is the clearest case in multimodal condensation of a later method beating its predecessor on accuracy and speed at once, running about 5.7× faster than AVDD under a shared protocol.
  4. More than two modalities. HoPA carries the LoRS-style recipe (trajectory matching plus a similarity structure) to video, audio and text together. It uses a rank-1 spectral proxy to avoid modeling every pair of modalities, and beats the pairwise variants the authors construct.

What the arc shows

The movement is the same one seen in image classification and image–text condensation: from random networks with hand-designed alignment terms, to pretrained encoders, to condensing inside a representation that is already aligned. Each step improves results and adds a dependency on a large pretrained model.

Open questions

What is different here

  • Matching each modality's distribution separately can destroy the alignment between them, so methods add explicit cross-modal terms.
  • Omnimodal work condenses in a shared embedding space that binds more than two modalities, which turns the problem into preserving higher-order relations, not only pairwise ones.

Comparison pitfalls

  • The literature is small; baselines are often unimodal methods applied per modality, which sets a low bar for a joint method.

Start with

AVDD · DAVDD · ImageBindDC

Methods by family 4

A paper appears under every family it belongs to. Oldest first within a family.

Surrogate matching

Trajectory matching 1

2026-04

HoPA — Omnimodal Dataset Distillation via High-order Proxy Alignment

Yuxuan Gao, Xiaohao Liu, Xiaobo Xia et al. · arXiv 2026Audio–visual & omnipaper ↗

Extends dataset distillation beyond two modalities to omnimodal settings by identifying the factor that bounds endpoint discrepancy as the number of modalities grows, then aligns modalities through a compact proxy that captures high-order cross-modal structure directly instead of modeling every pairwise combination; the proxy is compatible with trajectory matching. Reports better compression-performance trade-offs than bimodal distillation baselines across several omnimodal benchmarks, supported by a spectral-perspective theoretical analysis.

Surrogate matching

Distribution and feature matching 3

2024-01

AVDD — Audio-Visual Dataset Distillation

Saksham Singh Kushwaha, Siva Sai Nagender Vasireddy, Kai Wang et al. · TMLR 2024coreAudio–visual & omnipaper ↗code ↗

Extends distribution matching to audio-visual data, first matching visual-only and audio-only feature distributions with randomly initialized networks separately, then adding a joint matching loss that implicitly cross-matches real audio against synthetic visual features (and vice versa) and a modality-gap matching loss that aligns the audio-visual gap between real and synthetic data, plus herding-based initialization; on VGGS-10K at 10 images-per-class it reports 54.99% recognition accuracy versus 43.85% for plain distribution matching and 68.24% for the full dataset (Table 3), and raises audio-to-visual retrieval Recall@1 from 8.66% (DM) to 19.33% (Table 5).

2025-11

ImageBindDC — ImagebindDC: Compressing Multi-modal Data with Imagebind-based Condensation

Yue Min, Shaobo Wang, Jiaze Li et al. · AAAI 2026notableAudio–visual & omnipaper ↗

ImageBindDC condenses omnimodal data inside ImageBind's shared embedding space using a Characteristic Function loss that performs exact infinite-moment matching in the Fourier domain, enforcing uni-modal, cross-modal (real-synthetic pair) and joint-modal distributional consistency simultaneously rather than matching each modality separately. On NYU-v2 the paper reports lossless performance versus full-data training at 5 condensed datapoints per class, an 8.2 percentage-point absolute improvement over the previous best method, and over 4x less condensation time.

2025-11

DAVDD — Decoupled Audio-Visual Dataset Distillation

Wenyuan Li, Guang Li, Keisuke Maeda et al. · arXiv 2025Audio–visual & omnipaper ↗

Proposes DAVDD, a decoupled audio-visual distillation framework built on distribution matching: a bank of pretrained, frozen audio-visual encoder pairs feeds lightweight decoupler MLPs that split each modality's features into shared and private representations; a Common Intermodal Matching loss aligns the shared representations across modalities while a Sample-Distribution Joint Alignment combines instance-level contrastive alignment with EMA class prototypes, and private representations are kept out of the cross-modal loss entirely to protect modality-specific cues. On VGGS-10K at 10 images-per-class it reports 56.2% accuracy versus 54.0% for AVDD and 68.2% for the full dataset (Table 1).