Dataset Distillation via Factorization

What came before

Conventional distillation (dc-2021, dm-2023, mtt-2022) treats every synthetic sample independently and optimizes it in the original image shape, ignoring relationships between samples within a class. Concurrently, idc-2022 improves data efficiency by storing down-sampled images and upsampling at training time -- the paper notes IDC is a special case of its own framework where the hallucinator is a parameter-free upsampler.

The problem

Treating each synthetic sample independently wastes storage: a pilot experiment shows that adding more (correlated) MTT checkpoints as extra training data lowers downstream test loss, suggesting distilled sets could be improved by exploiting sample-wise relationships rather than only by making single samples better, but naively storing more samples increases memory.

The idea

Factorize the synthetic dataset into a small number of shared "bases" and several "hallucinator" networks; any basis fed to any hallucinator produces a distinct image, so |bases| x |hallucinators| effective images are available from a budget that stores only their sum, and an adversarial contrastive term forces different hallucinators to produce genuinely different outputs from the same basis.

How it works

Bases are image-shaped tensors (one per stored unit); hallucinators are lightweight encoder-affine transform-decoder networks (1 conv-relu block, 3-channel bottleneck) shared across all classes. A composed training image is $\tilde x_{ij}=H_{\theta_j}(\hat x_i)$ for basis i and hallucinator j. The synthetic set is trained by any existing distillation loss $\mathcal{L}_{DD}$ (trajectory matching by default, using mtt-2022's objective) plus a cosine-similarity term $\mathcal{L}_{cos}$ that pushes apart the feature-space embeddings of images composed from the same basis but different hallucinators; a feature extractor F is trained adversarially (min for F, max for the synthetic set) via a contrastive loss, alongside a classification loss on the composed images. Labels are hard, taken from the basis's class.

Evidence

CIFAR-10, MTT default loss, BPC (bases/class)=9 vs. IPC=10 (roughly equal storage after accounting for 5 shared hallucinators): HaBa 69.9% vs. MTT 65.3% (Table 2). CIFAR-100, BPC=49 vs. IPC=50: HaBa 47.0% vs. MTT 46.1%. SVHN, BPC=1 vs. IPC=1: HaBa 69.8% vs. MTT 58.5%. Cross-architecture (ConvNet-trained, evaluated on ResNet/VGG/AlexNet, Table 4): adding HaBa to MTT gains up to +17.6pp (AlexNet, CIFAR-10 IPC=51/BPC=50) and to DC/DM gains 1.2-12.9pp across architectures, confirming portability across three matching objectives (DC=gradient, DM=distribution, MTT=trajectory matching). Storage accounting: a hallucinator's parameters cost about as much as 2 synthetic images; with 5 shared hallucinators amortized across all classes, the paper sets BPC = IPC - 1 (one image-equivalent per class) to keep total storage close to the IPC baseline -- an approximate, not exact, equal-storage protocol. Ablation (Table 8): removing the cosine-similarity loss, the contrastive feature-extractor loss, or the task loss each costs accuracy (e.g. CIFAR-10 BPC=1: full 55.66% vs. w/o cos. 54.56%, w/o con. 54.91%, w/o task 54.62%). GPU budget: up to 4 24GB 3090s; memory is "slightly higher" than the underlying baseline (MTT) due to the extra losses, no exact figure given.

Limitations

States its own limitations: the online pairwise combination of hallucinators and bases adds time and GPU memory cost versus the baseline it wraps, and it inherits that baseline's limitations (e.g. diminishing returns from adding more images at large IPC). Storage accounting for hallucinator parameters is approximate (BPC=IPC-1 as a heuristic, not an exact byte-for-byte match), which complicates precise equal-storage comparison with pixel-only baselines. Uses trajectory matching (an expert-trajectory, bi-level-adjacent objective) as its default and strongest-performing loss, so its headline numbers inherit that objective's expert-trajectory precomputation cost, which is not separately reported here. Bases keep the same spatial shape as real images by default; only limited exploration of smaller/single-channel bases is given.