Decomposed Distribution Matching in Dataset Condensation

What came before

dm-2023 matches only class-wise mean features in random networks; cafe-2022 aligns richer multi-scale features but reverts to a bi-level scheme; idm-2023 and datadam-2023 improve the embeddings or add spatial attention supervision but, the paper argues, still leave style un-modeled and diversity unconstrained.

The problem

Diagnoses dm-2023's accuracy gap to bi-level methods (an 8-point drop versus dsa-2021 on CIFAR-100 at IPC=10) by decomposing the feature distribution into content and style: (1) a style discrepancy -- t-SNE of first-layer feature-map moments shows condensed and real images of the same class have visibly different style statistics, and drifting Herding's real-image style toward DM's measurably hurts downstream accuracy -- and (2) limited intra-class diversity, since DM's synthetic samples cluster locally in the final-layer embedding rather than spanning the class.

The idea

Add two loss terms on top of DM's mean-matching objective, in the same random-network, single-level framework: a style-matching term using well-established style indicators (channel-wise feature-map moments and Gram-matrix correlations, as in neural style transfer) to close the style gap, and an intra-class diversity term that pushes each synthetic sample's embedding away from its k-nearest synthetic neighbors of the same class via a KL-divergence penalty.

How it works

For randomly sampled networks (as in dm-2023), computes a Moments-Matching loss (MSE between per-layer channel-wise mean and variance of real vs. synthetic feature maps, following AdaIN) and a Correlation- Matching loss (MSE between per-layer Gram matrices, following Gatys-style texture/style representations), summed as the style loss $L_S = \alpha L_{MM} + L_{CM}$. An Intra-Class Diversity (ICD) loss maximizes the KL divergence between each synthetic sample's softmax feature and the mean feature of its k=0.2*IPC nearest synthetic same-class neighbors. Total objective $\lambda L_S + (\beta L_{ICD} + L_{MMD})$ is optimized by gradient descent on the synthetic pixel images only; no network is trained during synthesis. Hard labels; differentiable Siamese augmentation as in DM.

Evidence

CIFAR-10, IPC=10, ConvNet: DDM 53.0% vs. DM 48.9% (+4.1pp, Table 1); IPC=50: 65.6% vs. 63.0% (+2.6pp). CIFAR-100, IPC=10: 33.9% vs. 29.7% (+4.2pp). Tiny-ImageNet, IPC=50: 27.4% vs. 25.3% (+2.1pp). ImageNet-1K (64x64), IPC=50: 15.6% vs. 11.4% (+4.2pp). ImageWoof/ImageNette (128x128, IPC=10): 34.5%/58.1% vs. 31.2%/55.6% (Table 3). ResNet-18 (Table 2, CIFAR-10, IPC=10): 40.9% vs. DM 32.6% (+8.3pp) -- a larger gain than on ConvNet, suggesting the fix matters more for architectures more sensitive to style. Orthogonality (Table 5): adding the style-matching module alone to DataDAM and IDM improves both (e.g. IDM CIFAR-100 IPC=10: 45.1%->47.2%). Cross-architecture (CIFAR-10, IPC=10, trained on ConvNet): DDM 53.0/48.7/46.2/42.6% on ConvNet/AlexNet/VGG-11/ResNet-18 vs. DM 48.6/38.3/40.8/39.2% (Table 4). Continual learning (CIFAR-100, 5-step, IPC-based memory): 39.9% final accuracy vs. DM 34.4%, DSA 31.7%. Ablation (Fig. 3a, CIFAR-10, IPC=10): DM+MM and DM+CM each improve over DM alone, DM+MM+CM is better than either alone, and adding ICD gives a further gain, with the paper noting ICD is irrelevant at IPC=1 (no intra-class diversity to exploit) where the accuracy gain (Table 1) is attributed entirely to style matching.

Limitations

No GPU-hour, wall-clock or memory comparison is reported anywhere in the paper despite adding two extra loss terms (per-layer moments, per-layer Gram matrices over multiple layers, plus a k-NN search for ICD) on top of DM's single MMD term, so the added compute cost relative to plain DM is not quantified. Hard labels only; no soft-label or teacher-relabeling variant is tested, unlike most other 2024-2025 entries in this group. Evaluated only against DM-family baselines (DM, CAFE, DataDAM, IDM) plus DD/DG/DSA and coreset selection; no comparison to trajectory matching (MTT) or to the richer-statistics distribution-matching methods (M3D, WMDD, NCFM) that were concurrent or slightly earlier. Random networks only (no pretrained or partially trained embeddings), so it does not test whether style matching helps or is redundant once embeddings already carry more structure, as in DANCE or WMDD.