Dataset Meta-Learning from Kernel Ridge-Regression

What came before

Responds to coreset/subset-selection methods (e.g. SVM support vectors, Borsos et al.'s bilevel coresets), which need asymptotically many points, and to bi-level, gradient-based dataset distillation (dd-2018, and gradient-matching's dc-2021 as a contemporaneous baseline), which needs second-order optimization through an iterative inner learner. Also positions itself against label-only distillation (SLDD, LD) as a narrower special case of what it can do.

The problem

Diagnoses that optimizing a synthetic dataset for a general learning algorithm requires differentiating through that algorithm's (iterative) inner training loop, which is expensive and second-order; the paper asks whether an inner learner can be chosen so the outer optimization becomes first-order and closed-form, and formalizes "how small can an approximating dataset be" as epsilon-approximation.

The idea

Kernel ridge regression has a closed-form solution, and the neural tangent kernel (NTK) makes KRR a proxy for training an infinitely wide neural network; so treat the small "support" set fed to KRR as the thing being learned (Kernel Inducing Points), backpropagating only through the closed-form KRR loss rather than through any iterative training.

How it works

Optimizes support images $X_s$ (and optionally support labels $y_s$, or labels alone via the closed-form Label Solve) to minimize the KRR loss $L(X_s,y_s)=\tfrac12\|y_t-K_{X_tX_s}(K_{X_sX_s}+\lambda I)^{-1}y_s\|^2$ evaluated against real target batches $(X_t,y_t)$, where $K$ is the NTK (or NNGP) of a fixed architecture (FC1 or shallow Myrtle conv nets); a random kernel is sampled per step from a family (FC/Conv, depths 1-3) so the learned set is not overfit to one kernel. Nothing about a network's weights is stored or trained end-to-end; the only artifact is the support set itself. Label Solve (LS) instead fixes $X_s$ and solves for the minimum-norm $y_s$ in closed form via a pseudo-inverse, requiring no gradient descent at all for the label channel.

Evidence

MNIST KRR, RBF kernel, IPC-equivalent support sizes 10/100/500/5000/10000 images: 89.6/97.3/98.3/ 98.7/98.7% (Table 1); with augmentation+learned labels ("a+l"), 90.6/97.8/98.9/99.3/99.3%, beating contemporaneous LeNet-based DD (79.5% at 100 images) and DC (93.9% at 100 images) despite using a much shallower FC1/RBF kernel. CIFAR-10 KRR, RBF (a+l): 40.3/53.8/60.1/65.6/66.3% at the same budgets (Table 2), and FC1-trained neural networks on 5000-10000 KIP images reach 50.1/51.7% versus AlexNet trained via DC (39.1% at 100) or DD (36.8% at 100) -- a shallow FC1 network on KIP data beats deeper architectures trained with prior distillation methods. Cross-kernel transfer (Sec 4.2): datasets trained on a random mixture of six FC/Conv kernels (depths 1-3) generalize better on average than single-kernel training; Label-Solve labels transfer between FC1 and Myrtle-10 kernels with only negligible accuracy loss. Cross-architecture / out-of-distribution transfer to neural networks whose kernel was never sampled during KIP training (LeNet, Wide ResNet) still beats natural images but with a smaller margin than in-distribution transfer. Corruption/privacy ablation: 90%-pixel-corrupted KIP images at 10K support size reach 49.9% on CIFAR-10 (FC1), exceeding even uncorrupted natural images of the same size (45.5%). No GPU-hour or wall-clock cost figure is reported; the paper instead notes that deeper/more diverse architectures were infeasible given compute limits, restricting experiments to shallow kernels (FC1, Myrtle-5/10).

Limitations

States explicitly that computational limits forced the use of the simplest kernels (FC1, shallow Myrtle nets) rather than deeper architectures like LeNet/AlexNet/ConvNet, leaving open whether gains persist at depth (frepo-2022 and rfad-2022 later address exactly this). The NTK-KRR proxy is exact only for infinite width, so a lazy-training gap opens when the learned data trains finite networks -- observed directly as weaker (though still positive) transfer to out-of-distribution architectures (LeNet, Wide ResNet) whose kernel was not sampled during optimization. Learning labels jointly with images ("a+l") sometimes hurts at very large support sizes, which the authors attribute to label overfitting to KRR. Evaluated only on MNIST, Fashion-MNIST and CIFAR-10; no CIFAR-100, Tiny-ImageNet or ImageNet-1K result, and the kernel Gram matrix used in the loss scales with support-set size, limiting how large a set can be optimized in practice.