Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation?

What came before

SRe2L (2023) and CDA decouple synthesis from training via squeeze-recover-relabel: a teacher is pretrained once, images are synthesized by matching per-layer BatchNorm statistics, and every augmented crop is relabeled by the teacher at every training epoch. This relabel phase makes the stored soft-label set over 30x larger than the images themselves for ImageNet-1K. G-VBSM also re-batches within class but for a different reason (adding a model pool) and leaves BN statistics global. FKD (from model distillation) compresses labels by storing only the top-k logits.

The problem

Nobody had asked why the relabel phase needs so much storage in the first place. The paper diagnoses the cause: SRe2L/CDA construct each optimization batch from different classes to match global BN statistics, so images belonging to the same class are optimized independently and end up nearly identical to each other (feature cosine similarity of 0.84 within class on ImageNet-1K, Table 1, versus 0.695 for real images) — this lack of within-class diversity is what forces per-crop, per-epoch relabeling to supply enough distinct supervision.

The idea

If images within a class are made more diverse, the class no longer needs a fresh soft label for every augmentation and epoch; a large, redundant label pool can then be pruned by simple random sampling instead of a carefully engineered selection rule.

How it works

Modifies the SRe2L/CDA recover phase to batch and optimize images within one class together (rather than across classes) under class-wise BN running statistics computed in a one-epoch frozen-parameter pass during "squeeze" (class-wise supervision), increasing within-class feature diversity (Table 1: cosine similarity drops from 0.841/0.816 for SRe2L/CDA to 0.796 at IPC50) and lowering Maximum Mean Discrepancy to the real distribution. With images more diverse, the relabel-phase soft-label pool (one softmax vector per augmented crop per epoch, generated once by a pretrained ResNet-18/Timm teacher) is pruned by uniform random sampling at the batch level ("improved label pool") rather than any rule-based criterion (correctness, confidence, cut-ratio, etc., which Table 5 shows perform no better than random). In the loop: a pretrained teacher (frozen) for BN statistics and relabeling; no bi-level optimization or expert trajectories.

Evidence

ImageNet-1K, ResNet-18 eval, teacher relabel-with-KD regime throughout (Table 3): at IPC200, unpruned (1x) SRe2L reaches 57.0%, CDA 63.3%, LPLD 62.6%; at 40x label pruning LPLD reaches 59.6% versus SRe2L's unpruned 57.0% and CDA's degraded score, i.e. LPLD with 40x less label storage still beats unpruned SRe2L by 2.6 points, corresponding to compressing labels from 113 GB to 2.8 GB. At IPC10, 1x LPLD is 34.6% vs SRe2L 20.1%/CDA 33.3%, and even at 40x LPLD (20.2%) beats unpruned SRe2L (20.1%). Tiny-ImageNet (Table 2) shows the same pattern, including a stated 7.5% accuracy gain over CDA at 40x label reduction. Ablation (Table 4, IPC50/ResNet18/ImageNet-1K) isolates each mechanism: class-wise batching alone (+C) raises 1x accuracy 52.0->54.7 but still collapses at 100x pruning (22.6, worse than baseline's 25.4); adding class-wise BN supervision (+CS) raises 100x accuracy to 29.1; adding the improved (batch-level) label pool (+ILP) raises it further to 33.7 — showing diversity alone is not sufficient without the improved pruning/sampling scheme. Compute: 4x A100 80GB GPUs for ImageNet-1K/21K-P, 1x for Tiny-ImageNet (Appendix E.5).

Limitations

The paper states two limitations explicitly: generating the (large) soft-label pool is still required before pruning — pruning only reduces what is stored/reused, not the generation cost — and reducing label storage does not reduce training time, since the same number of training epochs is needed regardless of how many distinct labels are sampled. It only evaluates the decoupled BN-matching branch (SRe2L/CDA-style methods), not gradient/trajectory/distribution matching, and label-pruning quality is only validated at fairly high IPCs (10-200); scaling to larger networks (ResNet50/101) gives smaller or inconsistent gains than on ResNet18 (Table 2b).