Trajectory matching

MTT · CVPR 2022

Trains a network for several steps on the distilled data and minimizes its distance to the parameters that experts trained on real data reached, using expert trajectories precomputed once and stored. Also extends distillation to higher-resolution images.

FTD · CVPR 2023

Attributes part of the gap between distillation and evaluation to accumulated trajectory error, and regularizes the distillation toward flat trajectories.

DATM · ICLR 2024

Early expert trajectories suit small synthetic sets and late ones suit large sets; aligning trajectory difficulty with the set size keeps the method effective as the budget grows, and the paper reports lossless distillation for the first time.

SelMatch · ICML 2024

Shows that trajectory-matching methods can fall below random selection as images per class grow, because rare, hard features are not captured; initializes from selected real images and updates only part of the set.

PAD · arXiv 2024

Filters misaligned information: prunes the target data according to the compression ratio and distills through deep layers only.

TESLA · ICML 2023

Computes the unrolled gradient exactly with constant memory and adds a soft-label assignment for datasets with many classes, reaching 50 images per class on ImageNet-1K on a single GPU.

MTT-VL — Vision-Language Dataset Distillation

Xindi Wu, Byron Zhang, Zhiwei Deng et al. · TMLR 2024landmarkVision–languagepaper ↗code ↗

This is the first vision-language dataset distillation method, extending trajectory matching to image-text pairs by jointly distilling them in a contrastive formulation (since there are no discrete classes to condition on) and using LoRA matching for efficient trajectory matching in large vision-language models. Against adapted vision-language coreset-selection baselines, the paper reports nearly doubling Flickr30K image-to-text recall@1 (5.6% to 9.9%) while using 100 distilled pairs versus 1000 selected ones.

LoRS — Low-Rank Similarity Mining for Multimodal Dataset Distillation

Yue Xu, Zhilin Lin, Yusong Qiu et al. · ICML 2024coreVision–languagepaper ↗code ↗

LoRS distills a ground-truth image-text similarity matrix alongside the synthetic pairs, instead of assuming one caption strictly matches one image, and stores that similarity structure efficiently via low-rank factorization. The paper reports significant improvements over prior vision-language distillation algorithms and proposes LoRS as a foundational synthetic-data setup for image-text dataset distillation.

PTM-ST — Multimodal Dataset Distillation via Phased Teacher Models

Shengbin Guo, Hang Zhao, Senqiao Yang et al. · ICLR 2026notableVision–languagepaper ↗code ↗

PTM-ST distills vision-language pairs by matching a stage-aware model of the teacher's training dynamics rather than a single trajectory, using a shortcut-based trajectory construction strategy to fit distinct training phases and stabilize the match, addressing performance gaps between phases and unstable teacher trajectories seen in prior multimodal trajectory-matching methods. On Flickr30K and MS-COCO the paper reports surpassing prior state-of-the-art methods by up to 13.5 percentage points absolute, with an average gain of 9.53 points on Flickr30K, while also reducing storage overhead.

AMD — Asynchronous Matching with Dynamic Sampling for Multimodal Dataset Distillation

Ding Qi, Jian Li, Shuguang Dou et al. · ICLR 2026notableVision–languagepaper ↗

Proposes AMD for image-text dataset distillation, performing asynchronous trajectory matching that decouples the starting points of image and text expert trajectories to accommodate their different learning speeds, and replacing random prototype initialization with a Semantics-Aware Prototype Mining module that clusters the feature space into representative prototypes; reported Recall@1/5/10 gains are 4.5, 9.6 and 10.9 points on Flickr30K at 200 distilled pairs.

RepBlend — Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation

Xin Zhang, Ziruo Zhang, Jiawei Du et al. · NeurIPS 2025notableVision–languagepaper ↗

RepBlend identifies "modality collapse" in multimodal dataset distillation, over-concentrated intra-modal representations and an enlarged cross-modal gap, as a conflict between dataset distillation's over-compression and contrastive cross-modal supervision, and addresses it by weakening overdominant cross-modal supervision through representation blending to restore intra-modal diversity, plus symmetric projection trajectory matching that synchronizes optimization dynamics across modality-specific projection heads to correct asymmetric supervision. On Flickr30K and MS-COCO the paper reports consistent gains over prior multimodal distillation methods (e.g. +9.4 IR@10, +6.3 TR@10 at the 100-pair setting) with up to 6.7x faster distillation.

MKDT — Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networks

Siddharth Joshi, Jiayi Ni, Baharan Mirzasoleiman · ICLR 2025notablePre-training & transferpaper ↗code ↗

MKDT shows that applying supervised trajectory matching directly to a self-supervised objective fails because the SSL gradient has high variance, and instead trains a small student to match the representations of a larger SSL-trained teacher (knowledge distillation), then distills a synthetic dataset by matching the training trajectories of these lower-variance student models. The paper reports up to 13 percentage points higher downstream accuracy than prior work when labeled data is limited, across a variety of transfer tasks, as the first effective dataset distillation method for self-supervised pre-training.

LTDD — Distilling Long-tailed Datasets

Zhenghao Zhao, Haoxuan Wang, Yuzhang Shang et al. · CVPR 2025notablepaper ↗code ↗

Identifies why trajectory-matching distillation fails on long-tailed data — biased expert trajectories from imbalanced training propagate into biased synthetic sets, and experts trained on tail classes give poor guidance and soft labels — then proposes Distribution-agnostic Matching, which keeps the student trajectory away from the biased expert path, and Expert Decoupling, which matches backbone and classifier separately to improve tail-class guidance and soft-label quality. Reported as the first effective method for long-tailed dataset distillation.

GEOM — Navigating Complexity: Toward Lossless Graph Condensation via Expanding Window Matching

Yuchen Zhang, Tianle Zhang, Kai Wang et al. · ICML 2024notableGraphspaper ↗code ↗

Identifies that the prior state-of-the-art graph trajectory-matching method supplies biased and restricted supervision signals from the original graph, limiting both the scale and efficacy of the condensed graph, and proposes GEOM, which trains expert trajectories with a curriculum-learning strategy for more diverse supervision and transfers that information into the condensed graph with an expanding-window matching scheme; reports state-of-the-art results toward lossless graph condensation across several datasets.

SFGC — Structure-free Graph Condensation: From Large-scale Graphs to Condensed Graph-free Data

Xin Zheng, Miao Zhang, Chunyang Chen et al. · NeurIPS 2023notableGraphspaper ↗code ↗

SFGC drops the joint node-and-structure optimization of prior graph condensation methods and instead distills only a condensed node set with topology reduced to the identity matrix (graph-free data), implicitly folding structural information into the node attributes; a training-trajectory meta-matching scheme aligns the long-term GNN learning behavior between the original and condensed graph, and a closed-form graph neural feature score dynamically evaluates candidate condensed data during training. The paper reports superior performance across different condensation ratios versus prior graph condensation baselines.

DynaFed — DYNAFED: Tackling Client Data Heterogeneity with Global Dynamics

Renjie Pi, Weizhong Zhang, Yueqi Xie et al. · CVPR 2023notablepaper ↗code ↗

Instead of gathering client data, DynaFed reserves a short trajectory of global-model snapshots on the server and synthesizes a pseudo dataset so that a model trained on it mimics that trajectory's dynamics, then uses the synthetic data once to correct client drift during aggregation in later rounds. Reports effectiveness across extensive non-IID federated learning benchmarks without needing an external server-side dataset.

BTM — Geometric Characterisation and Structured Trajectory Surrogates for Clinical Dataset Condensation

Pafue Christy Nganjimi, Andrew Soltan, Danielle Belgrave et al. · arXiv 2026Other datapaper ↗

Gives a geometric account of why trajectory matching is hard to supervise with a small fixed synthetic set: such a set can only reproduce a limited, low-rank span of the parameter changes real SGD training induces, so a spectrally broad supervision signal creates a representability bottleneck. Proposes Bezier Trajectory Matching (BTM), which replaces SGD expert trajectories with quadratic Bezier surrogates between initial and final model states, reducing trajectory storage and better matching what a fixed synthetic set can represent. Reports BTM matching or improving on standard trajectory matching across five clinical tabular-EHR and time-series datasets (three NHS emergency-department cohorts, eICU and MIMIC-III), with the largest gains at low prevalence and low synthetic-data budgets.

HoPA — Omnimodal Dataset Distillation via High-order Proxy Alignment

Yuxuan Gao, Xiaohao Liu, Xiaobo Xia et al. · arXiv 2026Audio–visual & omnipaper ↗

Extends dataset distillation beyond two modalities to omnimodal settings by identifying the factor that bounds endpoint discrepancy as the number of modalities grows, then aligns modalities through a compact proxy that captures high-order cross-modal structure directly instead of modeling every pairwise combination; the proxy is compatible with trajectory matching. Reports better compression-performance trade-offs than bimodal distillation baselines across several omnimodal benchmarks, supported by a spectral-perspective theoretical analysis.

TGDD — TGDD: Trajectory Guided Dataset Distillation with Balanced Distribution

Fengli Ran, Xiao Pu, Bo Liu et al. · AAAI 2026paper ↗code ↗

Reformulates distribution matching as dynamic alignment along the model's training trajectory rather than a single static feature space, capturing how feature representations evolve during training and adding a distribution-constraint regularizer to reduce class overlap, reporting a 5.0-point accuracy gain on high-resolution benchmarks with no added optimization overhead across ten datasets.

HoP-TM — High-Order Progressive Trajectory Matching for Medical Image Dataset Distillation

Le Dong, Jinghao Bian, Jingyang Hou et al. · MICCAI 2025paper ↗code ↗

Extends trajectory matching for medical image distillation by matching a shape-wise potential that captures the geometry of intermediate points along expert parameter trajectories, not just terminal states, and by an easy-to-complex strategy that progressively matches parameters in order of complexity. Reports improved distillation performance on medical image classification while preserving privacy and keeping accuracy comparable to training on the original datasets.

Video DC study — A Large-Scale Study on Video Action Dataset Condensation

Yang Chen, Sheng Guo, Bo Zheng et al. · arXiv 2024Videopaper ↗code ↗

This large-scale study systematically examines video dataset condensation along three axes: temporal processing of video data, the evaluation protocol, and how condensation algorithms adapt to the space-time domain. It finds that labeling method matters a great deal for condensation performance, that simple sliding-window sampling suffices for temporal processing, and that dataset-distillation methods win in harder scenarios while sample-selection methods win in easier ones; it also proposes a unified evaluation protocol and reports state-of-the-art results on HMDB51, UCF101, SSv2 and K400 under it.

TimeDC — Less is More: Efficient Time Series Dataset Condensation via Two-fold Modal Matching--Extended Version

Hao Miao, Ziqiao Liu, Yan Zhao et al. · VLDB 2025Time seriespaper ↗code ↗

TimeDC condenses time series with two-fold modal matching, decomposition-driven frequency matching to preserve spectral structure and curriculum training-trajectory matching (with a buffer of precomputed expert trajectories) to preserve temporal dependencies, aiming for effective and generalized condensation rather than a single-domain surrogate objective. The paper reports effectiveness and efficiency gains over prior time-series condensation baselines across real datasets.

Teddy — Teddy: Efficient Large-Scale Dataset Distillation via Taylor-Approximated Matching

Ruonan Yu, Songhua Liu, Jingwen Ye et al. · ECCV 2024paper ↗

Replaces the multi-step-gradient dependence of bi-level, trajectory-style dataset distillation with a memory-efficient first-order Taylor approximation, and replaces repeated per-iteration model training with a pre-cached pool of weak models generated from a single base model, reporting up to 12.8-point gains and 46.6% less runtime over prior methods on Tiny-ImageNet and full-size ImageNet-1K.

NSD — Neural Spectral Decomposition for Dataset Distillation

Shaolei Yang, Shen Cheng, Mingbo Hong et al. · ECCV 2024paper ↗code ↗

Represents the whole distilled dataset as a shared set of spectrum tensors combined pairwise with per-image transformation matrices, rather than as independent images, so information is shared across the synthetic set through simple matrix multiplication, and optimizes this low-rank representation with a trajectory-matching objective guided by the real distribution, reporting state-of-the-art results on CIFAR-10/100, Tiny-ImageNet and an ImageNet subset.

ATT — Dataset Distillation by Automatic Training Trajectories

Dai Liu, Jindong Gu, Hu Cao et al. · ECCV 2024paper ↗code ↗

Replaces MTT's fixed synthetic-step count with an automatically and adaptively adjusted trajectory length to counter the 'Accumulated Mismatching Problem' caused by forcing the synthetic set to conform to one fixed-length segment of every expert trajectory, improving cross-architecture generalization and stability over fixed-length trajectory matching.

DDFAD — DDFAD: Dataset Distillation Framework for Audio Data

Wenbo Jiang, Rui Zhang, Hongwei Li et al. · arXiv 2024Other datapaper ↗

Applies dataset distillation to audio for the first time: extracts a Fused Differential MFCC feature (FD-MFCC) from each clip, distills it with the matching-training-trajectory method, and reconstructs playable audio from the distilled features with a Griffin-Lim-based algorithm. Reports effectiveness across several audio datasets.

MCT — Towards Stable and Storage-efficient Dataset Distillation: Matching Convexified Trajectory

Wenliang Zhong, Haoyu Tang, Qinghai Zheng et al. · CVPR 2025paper ↗code ↗

Diagnoses three problems with matching training trajectories (MTT) — instability of SGD-generated expert trajectories, slow convergence, and high storage cost for stored trajectories — and replaces the target trajectory with a convex combination of expert trajectories derived from linearized neural-tangent-kernel dynamics, giving a smaller, continuously samplable, more stable target for the student to match.

Progressive trajectory matching for medical dataset distillation

Zhen Yu, Yang Liu, Qingchao Chen · arXiv 2024paper ↗

Finds that plain trajectory matching (randomly matching parts of real training trajectories) is unstable and gives inferior results on medical images, so proposes a progressive trajectory-matching strategy for stability plus a dynamic overlap mitigation module that eliminates overlap across synthetic images and retrains parts of them to recover the diversity lost to the more stable matching. Reports 8.33% average improvement over prior state-of-the-art methods, and 11.7% at 2 images per class, on a new medical dataset distillation benchmark across multiple modalities.

CondTSC — Dataset Condensation for Time Series Classification via Dual Domain Matching

Zhanyu Liu, Ke Hao, Guanjie Zheng et al. · KDD 2024Time seriespaper ↗code ↗

CondTSC adapts image-style dataset condensation to time series classification with dual surrogate matching objectives computed in both the time and frequency domains (rather than time domain alone), combined with multi-view data augmentation and dual-domain training, to better capture the temporal and spectral structure that direct adaptations of image condensation methods miss. The paper reports outperforming other time-series and adapted image/graph condensation baselines, producing a condensed set that better matches the real data's distribution.

Discovering Galaxy Features via Dataset Distillation

Haowen Guan, Xuan Zhao, Zishi Wang et al. · NeurIPS 2023 Workshoppaper ↗code ↗

Uses dataset distillation on a class-balanced Galaxy Zoo 2 subset to visualize what a galaxy-morphology classifier relies on, treating the synthesized prototypical images as human-inspectable summaries of the features a neural net uses to classify galaxy morphology. Introduces a self-adaptive variant of trajectory matching to automate the distillation process, reporting enhanced performance on standard computer-vision benchmarks as a byproduct.

SeqMatch — Sequential Subset Matching for Dataset Distillation

Jiawei Du, Qin Shi, Joey Tianyi Zhou · NeurIPS 2023paper ↗code ↗

Argues that optimizing an entire synthetic dataset as one static, uniformly-updated entity couples its instances together and prevents later-training-epoch high-level features from being captured, and instead generates the synthetic instances sequentially so each new subset acquires knowledge the earlier subsets have not, improving over prior matching-based methods on SVHN, CIFAR-10/100 and Tiny-ImageNet.

Dataset Distillation for Medical Dataset Sharing

Guang Li, Ren Togo, Takahiro Ogawa et al. · AAAI 2023 Workshoppaper ↗code ↗

Applies trajectory-matching dataset distillation (the linked code builds on MTT) to a medical imaging dataset to produce a small, shareable synthetic set for cross-institution medical data sharing, following the same authors' prior soft-label distillation work on gastric X-ray images.

Wearable ImageNet: Synthesizing Tileable Textures via Dataset Distillation

George Cazenavette, Tongzhou Wang, Antonio Torralba et al. · CVPR 2022 Workshoppaper ↗code ↗

Generates tileable distilled "textures" by sampling random crops from a toroidal canvas of trajectory-matching-distilled pixels while enforcing that every crop is itself effective distilled training data for its class, producing infinite repeating patterns that visually summarize an ImageNet category and are suitable for printing on fabric or clothing.