Pre-training, transfer and foundation models

KRR-ST — Self-Supervised Dataset Distillation for Transfer Learning

Dong Bok Lee, Seanie Lee, Joonho Ko et al. · ICLR 2024corePre-training & transferpaper ↗code ↗

KRR-ST distills an unlabeled dataset for self-supervised pre-training by replacing the naive bilevel SSL objective, whose synthetic-sample gradient is biased by augmentation/masking randomness, with an inner objective that regresses learnable target feature representations (MSE, no randomness) and an outer objective that matches the resulting model's representations to a self-supervised target model trained on the full dataset. With the feature extractor fixed, the final linear head is obtained in closed form via kernel ridge regression, and the method is validated on several transfer-learning applications.

CLP-DD — Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models

Bincheng Peng, Guang Li, Ping Liu et al. · arXiv 2026Pre-training & transferpaper ↗

Proposes CLP-DD for distilling data used to train linear probes on frozen pretrained vision encoders: a sample-space kernel-ridge solver gives the linear probe induced by the synthetic set in closed form (exploiting that frozen-feature linear probing has an exact solution, with no NTK approximation or inner-loop trajectory), and the synthetic images are then updated by a temperature-scaled cross-entropy evaluating that induced classifier on real features. Reports that pairing the closed-form solver with a discriminative outer loss (rather than MSE) is decisive, and that on ImageNet-1K, CLP-DD matches or surpasses trajectory-based LGM-with-DSA on three of four backbones while running about 14x faster and using under one-eighth the GPU memory.

Linear Gradient Matching — Dataset Distillation for Pre-Trained Self-Supervised Vision Models

George Cazenavette, Antonio Torralba, Vincent Sitzmann · NeurIPS 2025notablePre-training & transferpaper ↗code ↗

Linear Gradient Matching distills a dataset for training linear probes on top of a frozen, pretrained self-supervised vision model, optimizing synthetic images so that gradients they induce in the linear classifier match those from real data. The paper reports that the resulting synthetic sets outperform all real-image baselines, generalize across pretrained backbones (e.g. a set distilled via DINO trains a competitive linear CLIP probe), and are effective for fine-grained classification and for interpretability probes such as measuring embedding-space similarity between models.

MKDT — Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networks

Siddharth Joshi, Jiayi Ni, Baharan Mirzasoleiman · ICLR 2025notablePre-training & transferpaper ↗code ↗

MKDT shows that applying supervised trajectory matching directly to a self-supervised objective fails because the SSL gradient has high variance, and instead trains a small student to match the representations of a larger SSL-trained teacher (knowledge distillation), then distills a synthetic dataset by matching the training trajectories of these lower-variance student models. The paper reports up to 13 percentage points higher downstream accuracy than prior work when labeled data is limited, across a variety of transfer tasks, as the first effective dataset distillation method for self-supervised pre-training.

SC-DD — Self-supervised Dataset Distillation: A Good Compression Is All You Need

Muxin Zhou, Zeyuan Yin, Shitong Shao et al. · arXiv 2024Pre-training & transferpaper ↗code ↗

SC-DD observes that as the pretrained model used for SRe2L-style inversion grows, supervised pretraining's batch-norm statistics flatten and become less informative for recovering data, whereas self-supervised pretrained models keep larger BN-statistic variance that yields stronger gradient signal during synthesis; it therefore performs decoupled inversion from self-supervised pretrained models instead of supervised ones. On CIFAR-100, Tiny-ImageNet and ImageNet-1K, the paper reports outperforming prior supervised decoupled-distillation methods (SRe2L, MTT, TESLA, DC, CAFE and others) by large margins when larger backbones are used, under the same recovery and post-training budgets.

SRG — Self-Supervised Representation-Guided Generative Dataset Distillation

Mingzhuo Li, Guang Li, Linfeng Ye et al. · arXiv 2026Pre-training & transferpaper ↗

Targets distillation for downstream training with frozen pretrained self-supervised encoders and lightweight adapters rather than randomly initialized networks: builds class-wise prototypes from real-image SSL representations and translates three SSL-space objectives (prototype alignment, inter-class discrimination, intra-class assignment) into diffusion guidance, anchoring early denoising to the latent of the nearest real image and later denoising to the SSL objectives. Reports consistent gains over evaluated generative baselines across datasets and IPC settings, with transfer across different pretrained representation spaces.

Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation

Sheng-Feng Yu, Jia-Jiun Yao, Wei-Chen Chiu · ICLR 2025Pre-training & transferpaper ↗

Improves self-supervised dataset distillation with a low-dimensional basis parameterization shared by images and their self-supervised representation targets, fixed (non-random) augmentations to stabilize training against a source of instability specific to self-supervised objectives, and a lightweight network that links the representations of augmented views from the same image for more compact pairs; reports gains in distillation efficiency, cross-architecture generalization and downstream transfer performance.

ReLA — Efficiency for Free: Ideal Data Are Transportable Representations

Peng Sun, Yi Jiang, Tao Lin · NeurIPS 2024notablePre-training & transferpaper ↗code ↗

ReLA (Representation Learning Accelerator) argues that a publicly available, task- and architecture-agnostic "prior model" can be used to produce efficient training signal that is transportable across tasks and architectures, substituting learned representation targets for the labels/pipeline a representation-learning method would otherwise need. The paper reports that using a CIFAR-10-pretrained ResNet-18 as the prior model to inform ResNet-50 training on ImageNet-1K cuts computational cost by 50% while matching the accuracy of the original BYOL recipe trained at full cost.

Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation

Sheng-Feng Yu, Jia-Jiun Yao, Wei-Chen Chiu · ICLR 2025Pre-training & transferpaper ↗

Improves self-supervised dataset distillation with a low-dimensional basis parameterization shared by images and their self-supervised representation targets, fixed (non-random) augmentations to stabilize training against a source of instability specific to self-supervised objectives, and a lightweight network that links the representations of augmented views from the same image for more compact pairs; reports gains in distillation efficiency, cross-architecture generalization and downstream transfer performance.