Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective

What came before

Names four prior categories it responds to: meta-model matching (DD, KIP, RFAD, FRePo, LinBa), gradient matching (DC, DSA, DCC, IDC), distribution matching (DM, CAFE, HaBa, IT-GAN, KFS) and trajectory matching (MTT, TESLA). All of these differentiate through, or otherwise couple to, an inner training loop on a model, which the paper identifies as the reason they stay confined to small or downsampled datasets.

The problem

The bilevel optimization shared by prior paradigms unrolls many inner-loop model updates per outer-loop synthetic-data update, which is computationally prohibitive at ImageNet resolution and scale, and truncated unrolling introduces its own bias. The paper's diagnosis is that coupling the synthetic data to a specific model being trained on it, rather than the objective itself, is what blocks scaling.

The idea

Split the pipeline into three independent, sequential, single-level stages: train a model once on the real data (squeeze), synthesize images by inverting that frozen model's batch-norm statistics and predictions with no real data present (recover), and train student models against the same model's soft labels on crops of the synthetic images (relabel). Because real and synthetic data are never co-resident and only summary statistics of the teacher are used, resolution and evaluation architecture become free choices.

How it works

Squeeze trains (or reuses off-the-shelf) a classifier $f_{\theta_{\mathcal T}}$ on the real data, e.g. torchvision ResNet-18/ResNet-50 for ImageNet-1K. Recover starts from Gaussian noise and updates pixels only (teacher weights frozen) to minimize classification loss against the target label plus a regularizer matching the synthetic batch's per-layer channel mean/variance to the teacher's stored BN running statistics: $\ell(f_{\theta_{\mathcal T}}(\tilde x),y)+\alpha\sum_l(\|\mu_l(\tilde x)-\mathrm{BN}_l^{RM}\|_2+\|\sigma_l^2(\tilde x)-\mathrm{BN}_l^{RV}\|_2)$, with RandomResizedCrop applied each iteration so only the cropped region is updated (found to matter more than any image-prior regularizer). Relabel stores FKD-style soft labels: the same teacher scores crops of the finished synthetic images, $\tilde y_i=f_{\theta_{\mathcal T}}(\tilde x_{R_i})$, and students are trained against these per-crop soft labels rather than the one-hot class label. Using a different architecture for recovery and relabeling is shown to hurt.

Evidence

ImageNet-1K, ResNet-18 student, soft (FKD) labels, Table 4: IPC=10 21.3%, IPC=50 46.8%, IPC=100 52.8%, IPC=200 57.0%; ResNet-101 student reaches 60.8% at IPC=50. Tiny-ImageNet IPC=50, ResNet-18: 41.1%. No hard-label (one-hot) number is reported anywhere. Synthesis cost (Table 1, one RTX-4090, Tiny-ImageNet, ConvNet-4): 52x faster and 11.6x less memory than MTT; on ResNet-18, 16x faster and 6.4x less memory. Key ablation (Table 8): removing the TV/$\ell_2$ image-prior regularizers and adding the per-iteration random crop lifts ImageNet-1K accuracy from 40.4% to 46.7% (ResNet-18, IPC=50) — the multi-crop optimization, not the pixel-prior terms, is the dominant lever. A separate ablation (Table 3) shows recovery budget matters: 0.5k/1k/2k/4k iterations give 38.9/43.7/46.7/46.7%.

Limitations

States a persistent accuracy gap to full-data training and flags the extra storage needed for soft labels as a cost, without quantifying it. No results below IPC=10 and no ImageNet-21K results are reported (scaling to 21K and to other modalities is left as future work, later taken up by CDA). ViT/DeiT-Tiny students perform far worse than convolutional students on the same condensed data (15-33% vs 47-61%), and recovering from a larger or more heavily-trained teacher (ResNet-50 vs ResNet-18, or longer squeeze schedules) is shown to produce worse, not better, synthetic data — an architecture- and training-recipe-dependence that the paper surfaces but does not resolve.