Method · Distribution and feature matching
M3D
M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy
Hansong Zhang, Shikun Li, Pengju Wang, Dan Zeng, Shiming Ge
AAAI 2024 · first public 2023-12-26 · arXiv 2312.15927
In one paragraph
Argues that prior distribution-matching methods only align first-moment (mean) statistics and so under-match the real and synthetic feature distributions, and proposes minimizing the maximum mean discrepancy in a reproducing kernel Hilbert space to align all orders of moments, surpassing the optimization-oriented method IDC on high-resolution ImageNet.
Explained
What came before
dm-2023-family methods (including idm-2023's class-aware regularization) align only the mean of the feature distribution; optimization-oriented methods such as idc-2022 pay a heavy bi-level cost, reportedly around 4 days for ImageNet-10 at IPC=20.
The problem
Matching only the mean leaves higher-order statistics (variance, skewness, and beyond) unmatched, so real and synthetic feature distributions can still differ substantially even once their means agree.
The idea
Embed features into a reproducing kernel Hilbert space and minimize the Maximum Mean Discrepancy there, which implicitly aligns all orders of moments rather than just the first, without training any network.
How it works
Optimizes synthetic pixel images against a class-wise MMD loss computed in an RKHS (Gaussian RBF kernel by default; polynomial/linear kernels give similar results) using several randomly initialized, untrained encoder networks. No bi-level loop and no classification loss -- the MMD term is the sole objective. For the high-resolution ImageNet experiments, M3D is combined with idc-2022's downsampled-multiformation parameterization (factor l=3) rather than plain pixels.
Evidence
CIFAR-10, IPC=10: 63.5% vs. DM 58.6% (Table 2). CIFAR-10, IPC=50: 69.9%, on par with IDM's 69.9%. ImageNet-10, IPC=10 (ResNetAP-10): 73.4% vs. IDC 72.8%, DM 52.3% (Table 3); IPC=20: 76.8% vs. IDC 76.6%. ImageNet-100, IPC=10: 46.9% vs. IDC 46.7%; IPC=20: 55.5% vs. IDC 53.7% -- the first DM-family result to surpass the optimization-oriented IDC at ImageNet scale. Cost (Table 3): ImageNet-10 IPC=10, M3D 1.1h vs. DM 1.4h vs. IDC 70.1h; IPC=20, M3D 3.1h vs. IDC 92.8h (about 30x faster than IDC, slightly faster than DM). Ablation (Table 1, CIFAR-10, IPC=10): first/second/third-order moment distances shrink from DM's 4.91/7.37/6.76 to M3D's 0.82/1.23/1.64 as accuracy rises from 48.9% to 63.5%, directly linking higher-moment alignment to the accuracy gain.
Limitations
No full ImageNet-1K evaluation is reported, only subsets of up to 100 classes. Kernel choice is ablated only lightly (roughly a 1-point spread across Gaussian/polynomial/linear). The ImageNet-scale results combine M3D's MMD objective with IDC's multi-formation parameterization, so the isolated contribution of higher-moment matching at that scale is entangled with the parameterization gain rather than cleanly separated.
Written by the atlas from the paper's full text. Check the paper for exact numbers.
Where it sits
- Distribution and feature matching (Surrogate matching)
- Setting: Image classification
Design choices
| What is stored | pixels, downsampled-multiformation |
| Labels | hard |
| Prior / networks used | random-networks |
| Optimization regime | single-level |
| Largest scale evaluated | mnist-cifar, imagenet-subsets |
Abstract (verbatim from arXiv)
Training state-of-the-art (SOTA) deep models often requires extensive data, resulting in substantial training and storage costs. To address these challenges, dataset condensation has been developed to learn a small synthetic set that preserves essential information from the original large-scale dataset. Nowadays, optimization-oriented methods have been the primary method in the field of dataset condensation for achieving SOTA results. However, the bi-level optimization process hinders the practical application of such methods to realistic and larger datasets. To enhance condensation efficiency, previous works proposed Distribution-Matching (DM) as an alternative, which significantly reduces the condensation cost. Nonetheless, current DM-based methods still yield less comparable results to SOTA optimization-oriented methods. In this paper, we argue that existing DM-based methods overlook the higher-order alignment of the distributions, which may lead to sub-optimal matching results. Inspired by this, we present a novel DM-based method named M3D for dataset condensation by Minimizing the Maximum Mean Discrepancy between feature representations of the synthetic and real images. By embedding their distributions in a reproducing kernel Hilbert space, we align all orders of moments of the distributions of real and synthetic images, resulting in a more generalized condensed set. Notably, our method even surpasses the SOTA optimization-oriented method IDC on the high-resolution ImageNet dataset. Extensive analysis is conducted to verify the effectiveness of the proposed method. Source codes are available at https://github.com/Hansong-Zhang/M3D.
BibTeX (generated; prefer the venue's official entry)
@article{zhang2023dataset,
title = {M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy},
author = {Hansong Zhang and Shikun Li and Pengju Wang and Dan Zeng and Shiming Ge},
journal = {AAAI 2024},
year = {2023}
}Nearby in Distribution and feature matching
RAHA — Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation
Jongoh Jeong, Sun-Kyung Lee, Kuk-Jin Yoon · ECCV 2026notableVision–languagepaper ↗code ↗
MDM — Multimodal Distribution Matching for Vision-Language Dataset Distillation
Jongoh Jeong, Hoyong Kwon, Minseok Kim et al. · CVPR 2026notableVision–languagepaper ↗code ↗
Harmonic Dataset Distillation for Time Series Forecasting
Seungha Hong, Sanghwan Jang, Wonbin Kweon et al. · AAAI 2026notableTime seriespaper ↗
Algorithmic Guarantees for Distilling Supervised and Offline RL Datasets
Aaryan Gupta, Rishi Saket, Aravindan Raghuveer · ICLR 2026notableOther datapaper ↗