Kernel and closed-form inner solvers

KIP · ICLR 2021

Kernel Inducing Points uses the correspondence between infinitely wide networks and kernel ridge regression to learn datasets one to two orders of magnitude smaller for KRR, and reports that the learned data transfers to finite-width networks.

FRePo · NeurIPS 2022

Neural feature regression with a pool of models: a truncated alternative to differentiating through the whole inner loop, reported to need an order of magnitude less memory and to train two orders of magnitude faster, with results up to ImageNet-1K.

RCIG · ICML 2023

Computes meta-gradients by implicit differentiation on a convexified problem, learning on top of a frozen finite-width neural tangent kernel with analytically computed final-layer parameters.

RFAD · NeurIPS 2022

Replaces KIP's exact kernel matrix, quadratic in the synthetic set size, with a random-feature approximation of the NNGP kernel that is linear in it, for at least a 100-fold speedup on one GPU.

KRR-ST — Self-Supervised Dataset Distillation for Transfer Learning

Dong Bok Lee, Seanie Lee, Joonho Ko et al. · ICLR 2024corePre-training & transferpaper ↗code ↗

KRR-ST distills an unlabeled dataset for self-supervised pre-training by replacing the naive bilevel SSL objective, whose synthetic-sample gradient is biased by augmentation/masking randomness, with an inner objective that regresses learnable target feature representations (MSE, no randomness) and an outer objective that matches the resulting model's representations to a self-supervised target model trained on the full dataset. With the feature extractor fixed, the final linear head is obtained in closed form via kernel ridge regression, and the method is validated on several transfer-learning applications.

MMDD — Efficient Multi-modal Dataset Distillation via Analytic Parameter Matching

Deyu Bo, Xinchao Wang · ICML 2026notableVision–languagepaper ↗code ↗

MMDD replaces the bi-trajectory distillation used by prior multimodal dataset distillation methods (aligning expert and student trajectories per modality, which needs many stored checkpoints and double backpropagation) with analytic parameter matching, directly matching the closed-form analytic parameters of the modal projectors, which the paper connects theoretically to matrix whitening. It caches only two matrices instead of trajectory checkpoints and computes the match in a single forward pass, reporting up to 65x storage reduction and 9.6x distillation speedup while scaling to 1000 synthetic pairs on Flickr30k and MS-COCO.

Provable KRR DD — Provable and Efficient Dataset Distillation for Kernel Ridge Regression

Yilan Chen, Wei Huang, Tsui-Wei Weng · NeurIPS 2024notablepaper ↗

Proves that for kernel ridge regression one data point per class is necessary and sufficient to exactly recover the original model in many settings, derives necessary and sufficient conditions for exact recovery with linear and surjective-feature-map kernels, and shows k+1 points suffice for deep linear networks with k classes; the resulting closed-form construction outperforms KIP while reported up to 15,840x faster on CIFAR-100.

Rethinking Backdoor Attacks on Dataset Distillation: A Kernel Method Perspective

Ming-Yu Chung, Sheng-Yen Chou, Chia-Mu Yu et al. · ICLR 2024notablepaper ↗

Counters the view that kernel-based dataset distillation counteracts backdoor risk, deriving two new theory-driven trigger-generation methods specialized for kernel-based distillation from a theoretical analysis of backdoor attacks under kernel methods. Reports its optimization-based trigger design produces resilient backdoor attacks that evade conventional backdoor detection and mitigation methods.

SFGC — Structure-free Graph Condensation: From Large-scale Graphs to Condensed Graph-free Data

Xin Zheng, Miao Zhang, Chunyang Chen et al. · NeurIPS 2023notableGraphspaper ↗code ↗

SFGC drops the joint node-and-structure optimization of prior graph condensation methods and instead distills only a condensed node set with topology reduced to the identity matrix (graph-free data), implicitly folding structural information into the node attributes; a training-trajectory meta-matching scheme aligns the long-term GNN learning behavior between the original and condensed graph, and a closed-form graph neural feature score dynamically evaluates candidate condensed data during training. The paper reports superior performance across different condensation ratios versus prior graph condensation baselines.

Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation

Noel Loo, Ramin Hasani, Mathias Lechner et al. · ICLR 2024notablepaper ↗

Strengthens the dataset-reconstruction attack on trained network parameters and proves it can recover the entire training set in the infinite-width Neural Tangent Kernel regime, showing empirically that success depends on deviation from the frozen infinite-width limit and that easily-reconstructed images tend to be dataset outliers. Turns this into a dataset distillation method by retraining on the reconstructed images, reporting high predictive accuracy from the reconstructed-then-distilled set.

Distill-CF — Infinite Recommendation Networks: A Data-Centric Approach

Noveen Sachdeva, Mehak Preet Dhaliwal, Carole-Jean Wu et al. · NeurIPS 2022notableOther datapaper ↗code ↗

Uses the Neural Tangent Kernel equivalence of an infinitely-wide autoencoder (infinity-AE) to give recommendation-model training a closed-form solution, then builds Distill-CF on top of it with differentiable Gumbel-sampling to synthesize tiny, high-fidelity summaries of large, sparse user-item interaction matrices; reports 96-105% of infinity-AE's full-data performance using as little as 0.1% of the original interactions.

KIP-ConvNet — Dataset Distillation with Infinitely Wide Convolutional Networks

Timothy Nguyen, Roman Novak, Lechao Xiao et al. · NeurIPS 2021notablepaper ↗code ↗

Extends KIP's kernel-ridge-regression meta-learning to infinitely wide convolutional networks via a distributed NTK computation, reporting over 65% CIFAR-10 test accuracy from just 10 datapoints — up from a prior best of 40% — with matching state-of-the-art results across MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100 and SVHN.

CLP-DD — Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models

Bincheng Peng, Guang Li, Ping Liu et al. · arXiv 2026Pre-training & transferpaper ↗

Proposes CLP-DD for distilling data used to train linear probes on frozen pretrained vision encoders: a sample-space kernel-ridge solver gives the linear probe induced by the synthetic set in closed form (exploiting that frozen-feature linear probing has an exact solution, with no NTK approximation or inner-loop trajectory), and the synthetic images are then updated by a temperature-scaled cross-entropy evaluating that induced classifier on real features. Reports that pairing the closed-form solver with a discriminative outer loss (rather than MSE) is decisive, and that on ImageNet-1K, CLP-DD matches or surpasses trajectory-based LGM-with-DSA on three of four backbones while running about 14x faster and using under one-eighth the GPU memory.

DNTK — Efficient Analysis of the Distilled Neural Tangent Kernel

Jamie Mahowald, Brian Bell, Alex Ho et al. · arXiv 2026paper ↗

Shows that the neural tangent kernel's dependence on per-point Jacobians can be reduced by compressing the input data itself with NTK-tuned dataset distillation, cutting required Jacobian evaluations by 20-100x, and that per-class NTK matrices keep a low effective rank under this reduction. Combines NTK-tuned distillation with existing projection and sketching methods as the distilled neural tangent kernel (DNTK), reporting up to five further orders of magnitude reduction in NTK computation while preserving kernel structure and predictive performance.

Size & approximation error — On the Size and Approximation Error of Distilled Sets

Alaa Maalouf, Murad Tukan, Noel Loo et al. · NeurIPS 2023paper ↗

Gives the first proof, for kernel-ridge-regression-based distillation methods such as Kernel Inducing Points, that a small distilled set exists for shift-invariant kernels by transforming the ridge-regression problem into random Fourier feature space, showing the distilled-set size can be linear in the RFF-space dimension (or near-linear in the kernel's effective degrees of freedom) with an excess-risk bound that depends on the regularization parameter.

DP-KIP-ScatterNet — Differentially Private Kernel Inducing Points using features from ScatterNets (DP-KIP-ScatterNet) for Privacy Preserving Data Distillation

Margarita Vinaroz, Mi Jung Park · arXiv 2023paper ↗code ↗

Applies DP-SGD to KIP (kernel inducing points), but finds that the convolutional NTK features KIP needs for good accuracy are too expensive to compute privately at scale; DP-KIP-ScatterNet substitutes fixed wavelet ScatterNet features, which need no pretraining (avoiding an extra privacy cost) and run on a single GPU. Reports superior performance to prior differentially-private data-distillation methods across several image datasets under formal DP guarantees.

BIB — Bidirectional Learning for Offline Model-based Biological Sequence Design

Can Chen, Yingxue Zhang, Xue Liu et al. · ICML 2023Other datapaper ↗code ↗

Extends the bidirectional (forward/backward) offline model-based optimization idea to biological sequence design by replacing the NTK proxy with a linearized pretrained language model, giving a closed-form loss that keeps the biophysical information the NTK could not represent. A bi-level auxiliary model learns how to weight the forward and backward mappings, and a derived learning-rate-adaptation module is reported to improve DNA/protein sequence design over the NTK-only predecessor.

KIDD — Kernel Ridge Regression-Based Graph Dataset Distillation

Zhe Xu, Yuzhong Chen, Menghai Pan · KDD 2023Graphspaper ↗code ↗

KIDD (Kernel ridge regression-based graph Dataset Distillation) replaces the inner GNN training loop with kernel ridge regression, using a purpose-built graph kernel (LiteGNTK, related to the random-walk graph kernel) so the condensation objective has a closed-form solution. The paper reports strong empirical performance across seven real-world graph datasets, in some cases with condensed graphs outperforming the full training set using as few as 1.65% of the original training graphs.

BDI — Bidirectional Learning for Offline Infinite-width Model-based Optimization

Can Chen, Yingxue Zhang, Jie Fu et al. · NeurIPS 2022Other datapaper ↗code ↗

BDI adds a backward mapping to offline model-based optimization that distills information from the static design dataset into the high-scoring candidate designs it proposes, alongside the usual forward proxy. Using an infinite-width network and its neural tangent kernel makes the backward loss closed-form, which the paper reports mitigates the out-of-distribution failure of proxy-only gradient ascent across several design tasks.

Can we achieve robustness from data alone?

Nikolaos Tsilivis, Jingtong Su, Julia Kempe · ICML 2022 Workshoppaper ↗

Poses adversarially robust classification as a bi-level kernel-ridge-regression problem over Neural Tangent Kernels, optimizing a dataset once so that plain gradient descent on it yields a robust model with no specialized robust training algorithm. The paper reports experiments across standard vision benchmarks and models, and revisits a prior data-optimization approach to argue that robustness from standard training on an optimized dataset is harder to obtain than previously claimed.