Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching

What came before

Builds directly on SRe2L, whose recovery matches BN mean/variance of a single backbone (ResNet-18) against a single synthetic-batch statistic — a "local-match-global" scheme the paper calls "one-sided." Also compares against MTT/FTD (trajectory matching), DC (gradient matching) and DM/CAFE (distribution matching), which it says never scale past small or downsampled ImageNet.

The problem

A single backbone, single layer type (BN only) and single statistic (channel mean/variance) cannot capture enough of the real distribution to transfer well to unseen evaluation architectures, and SRe2L underperforms classical distribution/gradient-matching methods on small datasets like CIFAR-100 despite winning on ImageNet-1K.

The idea

"Generalized matching": perform many local-match-global matches at once — across several pretrained backbones, both BN and convolutional layers, and both channel- and patch-level statistics — plus an explicit diversity loss over each class's feature Gram matrix, so no single backbone or statistic determines the synthetic data.

How it works

Squeeze uses 4 ImageNet-1K backbones (ResNet-18, MobileNetV2, EfficientNet-B0, ShuffleNetV2-0.5, torchvision weights) or 5 backbones trained from scratch for CIFAR/Tiny-ImageNet. Recover matches, via an EMA/score-distillation-style update, both BN statistics ($L'_{BN}$) and convolutional-layer channel- and patch-level mean/variance ($L'_{Conv}$, patch count 4-16 depending on dataset), adds a Data Densification loss (KL divergence over softmaxed eigenvalues of each class's batch Gram matrix, forcing full-rank/diverse batches), and randomly samples one backbone per iteration from the candidate set (Generalized Backbone Matching) rather than running all backbones in parallel. Relabel averages logits across all candidate backbones (optionally Frobenius-normalized) into a single ensemble soft label per crop, stored via FKD. Evaluation replaces SRe2L's KL loss with MSE plus a small one-hot cross-entropy term (weight 0.1), argued to be the $\tau\to\infty$ limit of KL. Post-eval: ImageNet-1K/Tiny-ImageNet largely follow SRe2L's 300-epoch recipe; CIFAR-10/100 use 1000 epochs on a 128-width ConvNet to match prior DC baselines.

Evidence

ImageNet-1K, ResNet-18 (Table 4): IPC=10, G-VBSM 31.4% vs SRe2L's own reported 21.3%; IPC=50, 51.8% vs 46.8%; ResNet-50/101 also improve at both budgets (e.g. IPC=50 ResNet-101: 61.0% vs 60.8%). Tiny-ImageNet IPC=50, ResNet-18: 47.6% vs SRe2L 41.1% (Table 4). CIFAR-100 IPC=10, 128-width ConvNet: 38.7% vs a G-VBSM-run SRe2L baseline of 31.6% (Table 5) — SRe2L never reported CIFAR results itself, so this is a first application rather than a re-run discrepancy; ImageNet-1K/Tiny-ImageNet SRe2L numbers are cited verbatim from SRe2L's own paper. Cross-architecture generalization at IPC=50 on unseen students (Table 2): DeiT-Tiny 29.4% vs SRe2L 15.4%; Swin-Tiny 57.4% vs 39.2%. Key ablation (Table 3), number of teacher backbones at IPC=10: 1 backbone 25.7%/30.2% (ResNet-18/50 eval) rising to 31.4%/34.5% with all 4 backbones — a +5.7/+4.3 point gain from ensembling backbones alone. No hard-label result is reported; all evaluations use soft or MSE+GT-distilled labels.

Limitations

No dedicated limitations section. Parallel multi-backbone matching is acknowledged as "extremely expensive," mitigated by random single-backbone sampling per iteration rather than solved; the extra convolutional-statistics matching adds computational burden, mitigated with a random dropout probability rather than removed; logit normalization helps small models but is shown to hurt ResNet-50/101 (Table 6), so it must be selectively toggled per architecture — the design space is not architecture-agnostic despite the paper's stated goal of generalization.