DANCE: Dual-View Distribution Alignment for Dataset Condensation

What came before

dm-2023 and its refinements idm-2023 and cafe-2022 rely on random or lightly-trained embeddings that the paper argues fail in two ways: "Persistent Training" (a single random/briefly-trained snapshot cannot proxy the long trajectory a real classifier goes through) and "Distribution Shift" (unconstrained synthetic features can drift outside the real class region).

The problem

Fixes like IDM's model queue or CAFE's feature alignment require training multiple models from scratch during condensation -- expensive and hyperparameter-heavy -- and still do not correct for synthetic features leaving their class region, which changes the decision boundary a downstream classifier learns.

The idea

Use a handful of pretrained, reusable "expert" encoders at intermediate stages of training to give distribution matching both a pseudo long-term view of feature evolution (inner-class) and a classification-based calibration that keeps synthetic features inside the correct class region (inter-class), while keeping distribution matching's single-level optimization.

How it works

Expert models are pretrained once, offline (60 epochs for low-resolution datasets and TinyImageNet, 80 epochs for ImageNet-1K subsets), and reused across all IPC settings. The inner-class Pseudo Long-Term Distribution Alignment (PLTDA) loss interpolates between a random encoder and an expert encoder with a random mixing coefficient (lambda ~ U(0,1)) and matches class-wise mean features under this "middle" encoder. The inter-class Distribution Calibration loss is a cross-entropy loss from a fully trained expert classifier evaluated on the synthetic images, pulling them back toward the correct decision region. No bi-level loop -- only the synthetic images are updated.

Evidence

CIFAR-10, IPC=10, ConvNet-3: DANCE 70.8% vs. DM 48.9%. CIFAR-100, IPC=50: 52.8% vs. DM 43.6%. TinyImageNet, IPC=10: 26.4% vs. IDM 21.9%. ImageSquawk (128x128), IPC=10: 77.2% vs. DM 50.4%. Cost: per-iteration 0.12s / 3045MB vs. DM's 0.08s / 3455MB on CIFAR-10 -- about 50% slower per step but slightly lower peak memory, since expert pretraining is amortized offline and reused across IPC settings rather than counted against each run. Ablation (CIFAR-10, IPC=10): PLTDA alone 64.8%, calibration alone 65.6%, both 70.8% -- the inner- and inter-class views are complementary. A single expert already gives 69.2%, with only marginal gains up to 15+ experts.

Limitations

Requires dataset-specific pretrained expert models before condensation can even start -- an offline but real cost that is conceptually close to trajectory matching's expert-trajectory precomputation, though cheaper since only a few checkpoints (not full trajectories) are needed. The paper does not quantify or directly compare this offline pretraining cost against MTT-style expert-trajectory generation. New datasets or domains require re-pretraining the experts before DANCE can be applied.