Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks

What came before

Existing distillation methods (dd-2018's BPTT, dc-2021/dsa-2021's gradient matching, kip-2021's kernel ridge regression) all assign each class its own independent set of learnable images, so storage grows linearly with the number of classes and any information shared between classes (e.g. related textures across visually similar categories) is re-learned redundantly in each class's images.

The problem

A per-class, independent-sample representation cannot exploit cross-class redundancy, does not generalize to continuous or unseen label spaces, and its previously reported weakness -- that back-propagation through time (BPTT) underperforms single-step gradient matching -- is, the paper argues, a property of how BPTT has been used (short unrolls, no momentum) rather than of BPTT itself.

The idea

Reformulate distillation as memory addressing: store one shared pool of basis vectors ("memories") plus learned per-query addressing matrices that linearly recombine bases into class-conditioned synthetic images, so the memory pool can be reused across classes and the compressed size decouples from the number of classes; separately, show that BPTT itself is a strong distillation algorithm once trained with momentum and long (100-200 step) unrolled inner loops.

How it works

A set of K basis vectors $\{b_1,...,b_K\}$ (each shaped like an image, optionally stored at half resolution and bilinearly upsampled) forms a shared memory $\mathcal{M}$. For a label vector $y$, r addressing matrices $\{A_1,...,A_r\}$ (one-hot or continuous) produce synthetic images $x_i'^T = y^T A_i [b_1;...;b_K]^T$. Both memories and addressing matrices are trained end-to-end by back-propagation through time: an inner loop trains a randomly initialized network from scratch on the addressed synthetic set for T steps (with momentum, T up to 200) via SGD, and the outer loop backpropagates a generalization loss on a real minibatch through the whole inner trajectory to update the memories and addressing matrices. Storage budget is counted as total parameter size: size(bases)+size(addressing matrices) is constrained to approximately match N*C*size(image) for a baseline of N images per C classes, i.e. an explicit equal-storage protocol against pixel-space baselines. Hard, discrete class labels.

Evidence

CIFAR-10, 1 image/class-equivalent storage, ConvNet: 66.4% vs. KIP 49.9%, MTT 46.3%, DM 26.0% (+16.5pp over the next best, Table 1). SVHN, same budget: 87.3% vs. KIP 57.3% (+30pp). CIFAR-100, 10 image/class budget: 42.9% vs. TM 40.1%, DM 29.7%. TinyImageNet, 1 image/class: 16.0% vs. TM 8.8%, DM 3.9%. Ablation (Table 2): single-step gradient matching 28.8% (CIFAR-10, 1 img/class) -> vanilla BPTT (no downsampling, no memory addressing) 49.1% -> + downsampling 55.2% -> + memory-addressing formulation 64.2% -> + data augmentation (full) 66.4%, isolating memory addressing's contribution at about 9pp on top of downsampling and about 36pp on top of single-step gradient matching. Momentum/unroll-length analysis (Fig. 4): adding momentum gains 7.0-9.2pp on CIFAR-10 at 10-100 inner steps; lengthening inner unrolls from 1 to 100+ steps gains 18.2pp (CIFAR-10) and 42.3pp (SVHN). Continual learning "compress-then-recall" (Table 3): 74.07% RA on MANY vs. best prior sp-La 50.81% (+23.2pp). Information sharing is visualized directly (Fig. 4): cosine similarity of addressing coefficients between CIFAR-100 classes shows visually similar classes (maple/oak/palm/pine/willow) share memory usage while dissimilar ones (lawn mower/rocket) do not.

Limitations

States that the costly inner-loop optimization (100-200-step unrolled BPTT per outer step) may become time-consuming on larger models or datasets, and suggests this could be addressed by pairing the memory formulation with a different (cheaper) learning framework -- i.e. the method as presented does not scale to ImageNet-size settings, only up to TinyImageNet. No wall-clock or GPU-hour figures are reported for synthesis cost, only "one GPU per experiment run." The equal-storage protocol (matching total parameter count) is a reasonable but not universally adopted accounting convention, and later parameterization papers (fred-2023, ddif-2025) use different conventions, complicating cross-paper comparison. The paper also flags a general societal risk that distilled data may under-represent minority populations, arguing its own higher accuracy is a partial mitigation rather than a solution.