Data-to-Model Distillation: Data-Efficient Learning Framework

What came before

DC, DSA, DM, CAFE, DAM, ATOM, IDC, KIP, FRePo, MTT, FTD, TESLA all distill into pixels and must fully re-distill when IPC changes; DiM already stores the dataset in a generator's weights via a models-pool logits-matching loss, but only demonstrated CIFAR/MNIST-scale results; GLaD and IT-GAN optimize a frozen generator's latents rather than its weights, so their storage still grows linearly with IPC (one latent per image) and both need complete retraining whenever the distillation ratio changes.

The problem

Pixel-parameterized distillation methods scale their storage and re-distillation cost linearly with IPC and resolution, degrade on unseen architectures (ResNet, DenseNet, ViT) because they are tuned to a small ConvNet, and had not been scaled past 64x64 ImageNet-1K or 128x128 subsets before this paper, largely because pixel-space optimization at that resolution is memory- and compute-prohibitive.

How it works

A BigGAN pretrained on the target dataset (StyleGAN-XL, a conditional GAN, and a CVAE are also tested as alternative generators in an ablation) is fine-tuned with two losses computed via a randomly-sampled network $\phi_\theta$ from a model pool of Depth-$n$ ConvNets and ResNet-18/32 with varied seeds: an embedding-matching loss $\mathcal{L}_{\text{EM}}$ that matches channel-wise attention maps (from intermediate layers) and final-layer features between real and generator-sampled batches sharing labels $Y$, and a prediction-matching loss $\mathcal{L}_{\text{PM}}$ that minimizes the KL divergence between temperature-softened ($T$) softmax predictions on real vs. generated logits (a KD-style soft-label matching signal used only during distillation, not carried to evaluation). The generator is updated by SGD on $\mathcal{L}=\mathcal{L}_{\text{EM}}+\lambda\mathcal{L}_{\text{PM}}$ (batch size 128, $K{=}60$ epochs, default $\lambda{=}10$ in the sensitivity ablation, $\lambda{=}100$/$T{=}4$ in the main implementation-detail setup, on two RTX A6000 GPUs). At evaluation, $\mathcal{S}=G(Z)=\{(s_i,y_i)\}$ is sampled from the frozen fine-tuned generator with hard labels $y_i$ equal to the conditioning class, and a fresh network is trained on these images from scratch — the same generator serves any IPC without re-optimization.

Evidence

CIFAR-10/100 and Tiny-ImageNet, ConvNet eval, hard labels (Table 1): CIFAR-10 IPC1 D2M 50.2% vs. FRePo 46.8%/MTT 46.3%; IPC10 67.8% vs. FRePo 65.5%; IPC50 74.4% vs. FRePo 71.7% (full-dataset upper bound 84.8%); CIFAR-100 IPC1 29.8% vs. FRePo 28.7%. ImageNet-1K at 64x64 (Table 2, hard labels): IPC1 D2M 8.3% vs. TESLA 7.7%/FRePo 7.5%; IPC50 D2M 28.3% vs. DAM 15.5% (a 19.1-point gap the paper highlights via Fig. 3, full-dataset 33.8%). 128x128 ImageNet-1K subsets (Table 2): ImageNette IPC1 D2M 49.9% vs. GLaD 38.7%; ImageWoof IPC1 32.3% vs. GLaD 23.4%; ImageSquawk IPC1 40.3% vs. GLaD 35.8%. Head-to-head with GLaD plugged into DC/DM/MTT on ImageFruit/ImageMeow/ImageNet-A-E at IPC1 (Table 3, all re-run by the authors): D2M beats every DC+GLaD/DM+GLaD/MTT+GLaD variant, e.g. ImageNet-A 55.2% vs. MTT+GLaD 50.7%. Re-distillation cost (Table 4): total GPU-hours to cover IPC1/10/50 on CIFAR-10 is 4.9 for D2M (all spent once at IPC1; redeploying to IPC10/50 costs 0 additional GPU-hours) vs. 47.4 for MTT and 10.3 for DC. Cross-architecture (Table 5, CIFAR-10 IPC50, unseen AlexNet/VGG-11/ResNet-50/DenseNet-121/ViT): D2M average 63.6% vs. best baseline DAM 59.7% (+3.9 points). Key ablation (loss components, CIFAR-10 IPC10): removing $\mathcal{L}_{\text{PM}}$ drops accuracy by 7.9 points and removing $\mathcal{L}_{\text{EM}}$ by 10.3 points relative to the full loss, whose combination gives a 12.7-point gain over a distillation-loss-free generator baseline — both matching terms are needed, with embedding matching contributing more. Generator-choice ablation (Table 7, CIFAR-10): StyleGAN-XL slightly outperforms BigGAN (75.2% vs. 74.4% at IPC50) but BigGAN is kept as default for its compute/performance balance — all four generators tested are trained on the target dataset, so this does not test an off-target generator. NAS application (Table 8): D2M-selected architectures reach 88.9% correlation-weighted accuracy and 0.80 Spearman rank correlation with real-data ranking, best among Random/DSA/DM/DAM/CAFE proxy sets.

Limitations

Every generator tested (BigGAN, CGAN, StyleGAN-XL, CVAE) is trained or fine-tuned on the exact target dataset, so, like DiM, none of the reported gain isolates a generator not matched to the target distribution; the paper's own stated limitation is that D2M needs more GPU memory than pixel methods at IPC1 (the fixed cost of the generator dominates at the smallest budget); evaluation is hard-label only despite using soft KL-divergence matching internally; the main experiments report GPU-hours for re-distillation but not the upfront cost of pretraining the base generator itself; ImageNet-1K experiments stop at 64x64 (only the 128x128 subsets, not full ImageNet-1K, are run at higher resolution).