Method · Trajectory matching
FTD
Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation
Jiawei Du, Yidi Jiang, Vincent Y. F. Tan, Joey Tianyi Zhou, Haizhou Li
CVPR 2023 · first public 2022-11-20 · arXiv 2211.11004
In one paragraph
Identifies the accumulated trajectory error of trajectory-matching distillation — the mismatch between the short synthetic trajectory optimized during distillation and the long trajectory used at evaluation — and regularizes expert trajectories toward flatter minima so that weights trained on the synthetic data are more robust to this error, improving accuracy by up to 4.7 points on a higher-resolution ImageNet subset.
Explained
What came before
Responds to mtt-2022, which matches short synthetic-data student trajectories against precomputed expert trajectories, and to dc-2021's gradient matching, both framed as part of the same family that suffers from "accumulated trajectory error."
The problem
Names and diagnoses a gap specific to trajectory matching: a student trained on the synthetic data starts exactly at an expert checkpoint $\theta^*_{t,0}$, but during evaluation training accumulates a growing deviation $\epsilon_t$ from the real-data trajectory at every subsequent matching step, because the synthetic data can only be optimized to match one short segment at a time. The paper shows this accumulated error, not just short-horizon matching per se, is what caps accuracy; a naive fix (robust learning -- training experts with added weight perturbations so $\mathcal{S}$ must tolerate more starting-point noise) helps but overloads the fixed-size synthetic set with the extra information needed to be robust, degrading convergence at a fixed budget.
The idea
Instead of making the synthetic data robust to perturbed starting weights (which costs capacity inside the small synthetic set), make the expert trajectory itself flat -- train experts with an explicit flatness regularizer so nearby points in weight space have similar loss -- so that whatever accumulated error the student picks up during evaluation lands somewhere the trajectory is insensitive to, at no extra cost to the synthetic images.
How it works
Same trajectory-matching objective and student/expert structure as mtt-2022 (student trained on synthetic images from expert start point $\theta^*_{t,0}$, loss = normalized distance to expert target $\theta^*_{t,m}$), but experts are now trained by minimizing $L_{\mathcal{T}}(f_\theta) + \alpha S(\theta)$ (Eq. 13), where the sharpness term $S(\theta)=\max_{\epsilon\in\Psi}[L_{\mathcal{T}}(f_{ \theta+\epsilon})-L_{\mathcal{T}}(f_\theta)]$ is approximated Sharpness-Aware-Minimization-style via a single ascent step $\hat\epsilon=\rho\, g_L/\|g_L\|_2$ (Eq. 27). An Exponential Moving Average of expert weights during buffer-phase training is used as a separate, minor implementation detail for faster distillation convergence. What is stored: per-class synthetic images (pixels), same footprint as mtt-2022; only the expert-training procedure (a synthesis-time cache) changes.
Evidence
Table 2 (ConvNet, hard labels; MTT value in parens): CIFAR-10 IPC1/10/50 = 46.8+/-0.3 (46.2) / 66.6+/-0.3 (65.4) / 73.8+/-0.2% (71.6) vs. full-data 84.8+/-0.1%; CIFAR-100 IPC1/10/50 = 25.2+/-0.2 (24.3) / 43.4+/-0.3 (39.7) / 50.7+/-0.3% (47.7) vs. full-data 56.2+/-0.3%; Tiny-ImageNet IPC1/10 = 10.4+/-0.3 (8.8) / 24.5+/-0.2% (23.2) vs. full-data 37.6+/-0.4%. Table 3, 128x128 ImageNet subsets (IPC1/10, MTT in parens): ImageNette 52.2+/-1.0 (47.7) / 67.7+/-0.7% (63.0), ImageWoof 30.1+/-1.0 (28.6) / 38.8+/-1.4% (35.8), ImageFruit 29.1+/-0.9 (26.6) / 44.9+/-1.5% (40.3), ImageMeow 33.8+/-1.5 (30.7) / 43.3+/-0.6% (40.4) -- the largest single gain (4.7 points, ImageNette IPC10) is the paper's headlined "more than 4%" ImageNet-subset improvement. Ablation: FTD's own "MTT+Robust Learning" baseline (perturbing expert starting weights directly, without flattening the trajectory) reaches only 45.8/63.2/72.7% on CIFAR-10 IPC1/10/50 and 24.1/39.4/47.9% on CIFAR-100 -- comparable to or below plain MTT and clearly below FTD, isolating flat-trajectory regularization (not just any form of perturbation-robustness) as the source of the gain. EMA ablation (Table, CIFAR-100/Tiny-ImageNet): "FTD (w.o. EMA)" reaches 43.4/49.8% (CIFAR-100 IPC10/50) and 9.8/24.1% (Tiny-ImageNet IPC1/10) vs. full FTD's 43.2/50.7% and 10.0/24.5% -- EMA's contribution is small and mixed (it can even lower CIFAR-100 IPC10 slightly), confirming flatness (not EMA) is the primary mechanism. Cross-architecture (CIFAR-10 IPC50, ConvNet-trained data evaluated unseen): FTD reaches 73.8/65.7/58.4/53.8% on ConvNet/ResNet18/VGG11/AlexNet vs. MTT's 71.6/61.9/55.4/48.2%, and vs. dc-2021's 53.9/20.8/38.8/28.7%. Downstream NAS-correlation (Table, CIFAR-10-derived proxy set of 500 images): FTD's proxy ranking correlates with real-training ranking at 0.87/0.68/0.54 (Top5/10/20) vs. MTT's 0.41/0.36/-0.04, at the same 360-minute distillation time, showing flatness also improves the synthetic set's use as a training-free NAS proxy.
Limitations
The fix addresses robustness to accumulated error but does not remove the underlying short-rollout trajectory-matching approximation itself; still requires the same expensive cached-expert-trajectory infrastructure as mtt-2022, now with an added flatness-regularized expert-training step (run on a mix of RTX3090 and Tesla V100 GPUs, no GPU-hour figure reported for the added cost). The gains are consistent but modest at CIFAR/Tiny-ImageNet scale (roughly 1-4 points) and largest specifically on the higher-resolution ImageNet subsets; no ImageNet-1K-scale or soft-label result, so it is not directly comparable to tesla-2023's ImageNet-1K numbers.
Written by the atlas from the paper's full text. Check the paper for exact numbers.
Where it sits
- Trajectory matching (Surrogate matching)
- Setting: Image classification
Design choices
| What is stored | pixels |
| Labels | hard |
| Prior / networks used | trained-experts |
| Optimization regime | single-level |
| Largest scale evaluated | tiny-imagenet, imagenet-subsets |
Builds on
Abstract (verbatim from arXiv)
Model-based deep learning has achieved astounding successes due in part to the availability of large-scale real-world data. However, processing such massive amounts of data comes at a considerable cost in terms of computations, storage, training and the search for good neural architectures. Dataset distillation has thus recently come to the fore. This paradigm involves distilling information from large real-world datasets into tiny and compact synthetic datasets such that processing the latter ideally yields similar performances as the former. State-of-the-art methods primarily rely on learning the synthetic dataset by matching the gradients obtained during training between the real and synthetic data. However, these gradient-matching methods suffer from the so-called accumulated trajectory error caused by the discrepancy between the distillation and subsequent evaluation. To mitigate the adverse impact of this accumulated trajectory error, we propose a novel approach that encourages the optimization algorithm to seek a flat trajectory. We show that the weights trained on synthetic data are robust against the accumulated errors perturbations with the regularization towards the flat trajectory. Our method, called Flat Trajectory Distillation (FTD), is shown to boost the performance of gradient-matching methods by up to 4.7% on a subset of images of the ImageNet dataset with higher resolution images. We also validate the effectiveness and generalizability of our method with datasets of different resolutions and demonstrate its applicability to neural architecture search. Code is available at https://github.com/AngusDujw/FTD-distillation.
BibTeX (generated; prefer the venue's official entry)
@article{du2022minimizing,
title = {Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation},
author = {Jiawei Du and Yidi Jiang and Vincent Y. F. Tan and Joey Tianyi Zhou and Haizhou Li},
journal = {CVPR 2023},
year = {2022}
}Nearby in Trajectory matching
PTM-ST — Multimodal Dataset Distillation via Phased Teacher Models
Shengbin Guo, Hang Zhao, Senqiao Yang et al. · ICLR 2026notableVision–languagepaper ↗code ↗
AMD — Asynchronous Matching with Dynamic Sampling for Multimodal Dataset Distillation
Ding Qi, Jian Li, Shuguang Dou et al. · ICLR 2026notableVision–languagepaper ↗
RepBlend — Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation
Xin Zhang, Ziruo Zhang, Jiawei Du et al. · NeurIPS 2025notableVision–languagepaper ↗
MKDT — Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networks
Siddharth Joshi, Jiayi Ni, Baharan Mirzasoleiman · ICLR 2025notablePre-training & transferpaper ↗code ↗