Improved Distribution Matching for Dataset Condensation

What came before

dm-2023's plain distribution matching extracts a single feature (the class mean) per image and draws its embedding networks purely from random initialization.

The problem

Two shortcomings of naive distribution matching: (1) imbalanced feature numbers -- comparing thousands of real features against only tens of synthetic ones makes the empirical MMD statistically unreliable unless the real data is highly redundant; (2) unvalidated embeddings -- purely randomly initialized networks occupy only a small, non-representative slice of the space of useful embedding functions, unlike the implicit sweep over trained networks that gradient-matching methods perform.

The idea

Keep dm-2023's bi-level-free objective but fix its statistics: extract more synthetic features per image via a partition-and-expand augmentation, draw embeddings from a growing pool of partially trained (not just randomly initialized) networks, and add a class-aware regularizer that pulls synthetic features toward correct classification.

How it works

Each synthetic image is partitioned into l x l tiles that are differentiably upsampled back to full size, turning one image into l^2 feature-producing views for the matching loss (not extra stored data). A fixed-size FIFO "model queue" is periodically pushed with freshly initialized networks, trained K steps on real data, and popped when full, so embeddings are drawn from a mix of random-init through lightly trained networks rather than random init alone. An accuracy-weighted cross-entropy term (class-aware distribution regularization) is added on top of the same class-wise mean-matching loss as dm-2023. No bi-level loop; only the synthetic images are updated by gradient descent.

Evidence

CIFAR-10, 10 IPC: IDM 58.6% vs. DM 48.9% (Table 1). CIFAR-100, 10 IPC: 45.1% vs. DM 29.7%, vs. DSA 32.3%. TinyImageNet, 10 IPC: 21.9% vs. DM 12.9%. ImageNet-Subset, 10 IPC: 17.1% vs. DM 11.9%. Cumulative ablation (Table 2, CIFAR-100, 10 IPC): DM 29.7% -> +model sampling 32.0% -> +class-aware regularization 34.3% -> +partition-and-expand augmentation 45.1% -- the augmentation gives by far the largest single gain (+10.8pp). On ImageNet-Subset, model sampling alone slightly hurts (-1.1pp), which the paper attributes to the model queue being undersized for a harder dataset. Experiments run on a single RTX 3090; no wall-clock or memory comparison to DM or to trajectory-matching methods is reported.

Limitations

Introduces extra hyperparameters (queue size N_max, per-model training steps K, regularization weight, tuned differently per IPC) beyond DM's near-hyperparameter-free recipe. The model queue can underperform on harder/larger datasets (ImageNet-Subset). No explicit cost/time comparison is given against DM, bi-level gradient matching, or trajectory matching, so its efficiency advantage over those methods is argued qualitatively rather than measured in this source.