What is Dataset Distillation Learning?

What came before

By mid-2024 the field had four families of small-scale synthesis objectives — BPTT (bi-level meta-model matching), distribution matching (DM), gradient matching (DC/GM), and trajectory matching (MTT) — each shown to reach high accuracy on CIFAR-10, but no prior work examined what information the resulting synthetic images actually encode once distillation is done. Zhong & Liu (2023, cited) had already shown distilled data does not transfer well across architectures.

The problem

High evaluation accuracy does not explain *how* a handful of synthetic images substitute for a full training set: is it representing the data manifold, mimicking a trained model's parameters, or something else? Without knowing what is stored, method design and evaluation are both guesswork.

The idea

Distilled data behaves like a compressed record of a real model's *early training dynamics* rather than a compressed sample of the data manifold: a model trained on distilled data ends up similar to a real model that was heavily regularized (early-stopped), not to a model trained on a small random subset of real data, and individual distilled points still carry distinct, humanly interpretable semantic content (e.g. "yellow car", "military plane in the sky") extractable via influence functions.

How it works

Uses one representative method per paradigm — BPTT (meta-model/bi-level), DM (distribution matching, random conv nets), DC/gradient matching, and MTT (trajectory matching, expert trajectories in the loop) — on CIFAR-10 with a 3-layer ConvNet trained via SGD for 300 iterations. No new synthesis method is proposed. Analysis tools: (1) an inverse pipeline — train on real data, evaluate on distilled images — to test whether distilled data lies on the real manifold and is recognizable; (2) prediction-agreement comparison between distilled-trained models and early-stopped vs. data-subset-trained models; (3) Hessian-trace loss-curvature analysis (Hutchinson's method / PyHessian) of a real-data-trained model evaluated against distilled images over training iterations; (4) exact leave-one-out influence functions $I_{x_d\to x_t}=L(x_t;\hat\theta_{-x_d})-L(x_t;\hat\theta)$ computed by retraining, plus LLaVA-generated semantic attribute annotations of real images to interpret which real-image concepts each distilled image is most influential for.

Evidence

CIFAR-10, ConvNet, all four synthesis methods (BPTT, DM, GM, MTT) at matched accuracy (no absolute accuracy is the point; the comparisons are all controlled at fixed accuracy or fixed data budget): real-trained models classify distilled images with high accuracy (Fig. 2) and distilled images sit inside real class clusters under UMAP, showing recognizability; yet distilled pixel values fall outside [0,1] and clipping BPTT images to [0,1] drops retrained accuracy from 58% to 44%, showing distilled data is off the real manifold. Mixing 10 distilled images/class with 0-250 real images/class (Fig. 3 right) *decreases* accuracy relative to distilled-only training, in sharp contrast to a random-real-image baseline that improves monotonically with more data — direct evidence that a reported distilled-set accuracy is not simply "more of the same kind of value" as real images and cannot be naively combined with them. Prediction-agreement analysis (Fig. 4, at matched test accuracy): distilled-trained models agree far more with early-stopped real models (iteration 35-130 depending on method) than with models trained on random 0.5-5% real subsets. A real-data model's classification accuracy on distilled images plateaus after ~150 of 300 training iterations even as its accuracy on the real test set keeps rising (Fig. 5), and loss-curvature (Hessian trace) with respect to BPTT/MTT-distilled images flattens within ~1.5 epochs (Fig. 6), both indicating distilled data encodes only early-training information. Influence functions show weak correlation with feature-space visual similarity (Fig. 7, confirming influence is not merely "looks similar"), are consistent across random seeds (Fig. 8 left) and independent of which other distilled images are present (Fig. 8 right, correlation preserved when leave-one-out uses real images instead of other distilled images). Precision-recall analysis against LLaVA-derived semantic tags (Fig. 9, Table 1) finds individual BPTT/MTT distilled images are predictive of specific non-class semantics (e.g. "yellow car", "parking lot") beyond the one-hot label.

Limitations

All experiments are on CIFAR-10 with small ConvNets and small-scale methods (BPTT, DM, GM, MTT); the paper does not test decoupled/relabel+KD methods (SRe2L-style) or ImageNet scale, so it is unclear whether "compresses early-training dynamics" still describes methods that rely on a pretrained teacher rather than a bi-level/surrogate objective. The curvature analysis explicitly could not draw the same clean conclusion for distribution/gradient matching data (curvature fluctuates rather than flattening cleanly), so the "early training dynamics" story is best supported for BPTT/MTT specifically. Influence functions require expensive leave-one-out retraining (only tractable at the 100-image scale studied) and the semantic-extraction step depends on a separate multimodal model (LLaVA) to generate ground-truth attributes, which is itself an added source of noise/bias not present in the class labels.