Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching

What came before

Responds to mtt-2022 and ftd-2023, which always match expert-trajectory segments at one fixed (typically early) training stage regardless of synthetic-set size, and shows this is why trajectory-matching accuracy stalls or falls below random selection as IPC grows (kip-2021 and rcig-2023's kernel-closed-form results, and tesla-2023's soft-label ImageNet-1K numbers, are also tabulated as the wider contemporary field).

The problem

Trajectory-matching methods remain effective only at extremely small IPC; as the synthetic-set size grows they plateau or lose to random selection, so no prior method reaches "lossless" distillation (matching full-dataset accuracy). The paper diagnoses the cause as a fixed matching target: always matching early-epoch ("easy pattern") expert-trajectory segments, which is the right amount of information for a handful of images but starves a large synthetic set of the harder, later-training patterns it has room to represent.

The idea

Align which segment of the cached expert trajectory is matched -- early/easy vs. late/hard -- with the size of the synthetic set being distilled: small IPC matches early segments, large IPC matches later segments. Combine this with learned (optimized) soft labels, stabilized by a sequential easy-then-hard generation schedule, since naively optimizing soft labels destabilizes low-IPC training.

How it works

Builds directly on mtt-2022's trajectory-matching loss (expert trajectories $\theta^*$ cached from real-data training; student trained N steps on synthetic images from a sampled start point $\theta^*_t$, loss = normalized distance to $\theta^*_{t+M}$). The novel control is sampling $t$ from a shrinking window $\mathcal{U}(\{\theta^*_{T^-},\ldots,\theta^*_T\})$ whose upper bound $T$ starts small and grows toward a dataset-specific ceiling $T^+$ over the course of distillation (a "sequential generation" curriculum), rather than mtt-2022's window fixed at the start of training. Synthetic labels are optimized jointly as $\hat y_i=\mathrm{softmax}(L_i)$, initialized from a pretrained model's logits rather than one-hot vectors. What is stored: per-class synthetic images (pixels) plus a learned soft-label vector per image; expert trajectories remain a synthesis-time cache.

Evidence

Table 5 (ConvNet, learned soft labels): CIFAR-10 IPC1/10/50/500/1000 = 46.9+/-0.5 / 66.8+/-0.2 / 76.1+/-0.3 / 83.5+/-0.2 / 85.5+/-0.4% vs. full-data 84.8+/-0.1% -- IPC1000 (20% of CIFAR-10) *exceeds* full-dataset accuracy, the paper's "lossless" claim; CIFAR-100 IPC1/10/50/100 = 27.9+/-0.2 / 47.2+/-0.4 / 55.0+/-0.2 / 57.5+/-0.2% vs. full-data 56.2+/-0.3% (IPC100 = 20% also exceeds full data); Tiny-ImageNet IPC1/10/50 = 17.1+/-0.3 / 31.1+/-0.3 / 39.7+/-0.3% vs. full-data 37.6+/-0.4% (IPC50 = 10% exceeds full data). At the same table's matched IPC, DATM beats MTT (e.g. CIFAR-100 IPC50 55.0 vs. 47.7), FTD (55.0 vs. 48.5), TESLA (55.0 vs. 47.9) and the kernel-closed-form field (RCIG CIFAR-100 IPC50 46.7). Cross-architecture (CIFAR-10 IPC50, ConvNet-distilled data evaluated unseen): DATM 55.03 (ConvNet) / 51.71 (ResNet18) / 45.38 (VGG) / 45.7+ (AlexNet) vs. FTD 48.90/46.65/43.24/42.20 and MTT 45.68/42.56/41.22/40.29 -- DATM transfers better than prior trajectory-matching methods at every unseen architecture, not just its training architecture. Cost: distillation cost does not scale linearly with IPC (IPC1000 costs only 1.05x IPC50 because late-trajectory matching needs fewer optimization changes per image); run on 4x NVIDIA A100 (80GB each); reducing synthetic steps N from 80 to 40 halves GPU memory at a cost of ~0.4-0.8 accuracy points. Ablation (Fig. 3a/Table, CIFAR-100 IPC50): neither component alone reaches the full method -- no soft label/no difficulty alignment 48.50%, +soft label only 50.79%, +difficulty alignment only 52.96%, both together 55.03% -- isolating that difficulty alignment and learned soft labels are complementary, additive mechanisms, neither sufficient alone.

Limitations

States explicitly that "lossless" holds only for the distillation backbone network (ConvNet); when evaluated on other unseen architectures a performance drop remains, attributed to different-capacity models needing different amounts of training data. Optimizing soft labels destabilizes low-IPC distillation unless paired with the sequential easy-to-hard generation curriculum, adding a new schedule to tune. Distillation cost, while sub-linear in IPC, still requires 4x 80GB A100 GPUs and the same cached-expert-trajectory infrastructure as mtt-2022; the paper states scaling trajectory matching to larger (e.g. ImageNet-1K-scale) datasets remains hard due to this cost and is left to future work, so "lossless" is demonstrated only at CIFAR-10/100 and Tiny-ImageNet scale.