Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator

What came before

Builds on SRe2L's decoupled BN-statistics-plus-logit recovery and its dynamic, per-crop per-epoch soft-label relabeling via MixUp, which the paper singles out for its storage cost (25.9GB of labels versus 0.7GB of images on ImageNet-1K).

The problem

Every synthetic instance in prior class-specific frameworks is tied to one pre-assigned one-hot label and optimized only to encode that class's intra-class features. As IPC grows, new instances increasingly duplicate existing intra-class features rather than adding information, and no instance is ever optimized to represent the inter-class feature regions that actually shape decision boundaries.

The idea

Learn a single additive Universal Feature Compensator (UFC) — a zero-initialized, image-shaped tensor optimized jointly across all classes in a subset — and add it to one real image per class to generate a full new synthetic instance per class from one UFC. Because the resulting instances already span multiple classes, their soft labels behave like MixUp-interpolated labels, so a single static soft label per instance can replace SRe2L's dynamically regenerated, stored-every-epoch labels.

How it works

Teacher ensemble (also used for relabeling): ResNet-18, MobileNetV2, EfficientNet-B0 (+ShuffleNetV2 for CIFAR/Tiny-ImageNet). The base set is one unoptimized real image per class; the only optimized quantity is the additive compensator $u_j$, trained with the same BN-statistics-plus-classification recovery loss as SRe2L on $\tilde s_i=x_i+u_j$, so $M$ compensators over $C$ real images yield $C\times M$ synthetic instances. Soft labels are computed once per instance as the teacher ensemble's average prediction $\tilde y_i=\frac1M\sum_m f^m_{\theta_{\mathcal T}}(\tilde s_i)$ and stored statically; MixUp at student-training time linearly combines these stored labels rather than recomputing them, cutting label storage up to 99.3%. Budget/compression is reported both as IPC (counting UFC-generated instances) and as a compression ratio that includes the UFC's own storage bits, so comparisons to SRe2L are at equal total stored bits, not equal image count. Post-eval: AdamW, 300 epochs (static labels) or 75 epochs (dynamic-label variant, reduced 4x for fair comparison), RandomResizedCrop+Flip+MixUp; students ResNet-18/50/101 for ImageNet-1K, plus MobileNetV2/EfficientNet-B0/ShuffleNetV2 for CIFAR-100 cross-architecture tests.

Evidence

ImageNet-1K, ResNet-18 (Table 2): IPC=10, INFER 37.0% vs SRe2L 21.3%; IPC=50, 54.3% vs 46.8% (a 7.5 point gain at IPC=50, ResNet-18 — the abstract's "34.5% at IPC=50" figure does not correspond to any ResNet-18 or ResNet-101 cell found in the paper's own tables and could not be reconciled). On ResNet-50, IPC=50: 62.5% vs 55.6%. CIFAR-100, ResNet-18: IPC=10, INFER 50.2% vs SRe2L 31.6%; IPC=50, 65.1% vs 49.5% (a 15.6-point gain, and with dynamic labels, INFER+Dyn reaches 68.9%, a 19.4-point gain). Storage: SRe2L needs 25.9GB of dynamic soft labels on ImageNet-1K versus 0.7GB of images; INFER's static labels cut this by up to 99.3%. Key ablation (Table 4, ImageNet-1K IPC=10): adding architectures to the UFC-optimization ensemble raises INFER from 45.0% (ResNet-18 only) to 50.2% (four architectures), and INFER+Dyn from 38.1% to 53.4%, isolating ensemble diversity as the mechanism, not just the UFC idea itself. No hard-label result is reported.

Limitations

States the single-UFC-per-subset design likely will not scale to ImageNet-21K-sized label spaces since one compensator must represent generality across all classes in the subset, and that its two-level structure (real per-class images for intra-class, UFC for inter-class) is not jointly optimized, left as future work. Compression-ratio accounting that folds in UFC storage is reported only for ImageNet-1K, not for the CIFAR/Tiny-ImageNet tables, leaving the fairness of those comparisons less transparent.