Dataset Distillation via Committee Voting

What came before

SRe2L inverts a single frozen teacher and stores its soft labels; G-VBSM and EDC extend this to several backbones but weight every committee member equally when forming the recovery loss and the relabeling logits. CDA improves the augmentation curriculum and RDED replaces inversion with real-crop selection, but neither addresses multi-model weighting.

The problem

Single-backbone synthesis (SRe2L) restricts diversity and bakes in one model's bias; equal-weight multi-backbone synthesis (G-VBSM) assumes every pretrained model is equally informative, which the paper argues is false and dilutes the distilled set's quality; and post hoc soft labels computed with the teacher's real-data running batch-norm statistics are shown (Fig. 5) to mismatch the actual batch statistics of synthetic images, producing suboptimal supervision.

The idea

Weight a committee of pretrained backbones by each one's own measured "prior performance" (its distill-then-train-a-student accuracy) rather than uniformly, both when forming the recovery gradient and when generating soft labels, and recompute batch-norm statistics from each synthetic batch itself (not from the teacher's stored real-data running statistics) before producing the labels used to train students.

How it works

Builds SRe2L++ first: real-image initialization (instead of Gaussian noise), RandomResizedCrop augmentation during recovery, a smoothed cosine learning-rate schedule and smaller evaluation batch size. On top of this, Algorithm 1 pretrains each committee model $\Phi$, distills a set with it alone, trains a student on that set with KD, and records the student's held-out accuracy as $\alpha_\Phi$ ("prior performance"). During synthesis, a random subset of $N{=}2$ committee members is sampled per step and their per-model losses are combined with a softmax-over-$\alpha_\Phi$ weighting (temperature $T{=}5$, Eq. 4) instead of equal averaging; the same prior-weighted softmax combines committee predictions into soft labels. Batch-Specific Soft Labeling (BSSL) recomputes batch-norm mean/variance (Eqs. 6-7) from the current synthetic batch's own activations, holding the rest of the teacher fixed, before scoring soft labels; for non-BN architectures (ViT) a BN-ViT variant substitutes BatchNorm for LayerNorm so BSSL still applies. No student network is in the synthesis loop; five backbones (ResNet-18/50, DenseNet-121, ShuffleNetV2, MobileNetV2) form the default committee for ImageNet-1K/Tiny-ImageNet.

Evidence

ImageNet-1K, ResNet-18, relabel+KD, 300 evaluation epochs (Table 1): IPC=10, CV-DD 49.5% vs its own SRe2L++ baseline 43.1%, RDED 42.0%, CDA 33.6%; IPC=50, CV-DD 59.5% vs SRe2L++ 57.6%, RDED 56.5%, CDA 53.5%. Against vanilla-ensemble methods under a matched protocol (Table 2, ResNet-18): IPC=10, CV-DD 49.5% vs EDC 48.6%, G-VBSM 31.5%; IPC=50, CV-DD 59.5% vs EDC 58.0%. CIFAR-100 IPC=10: CV-DD 61.8% vs SRe2L++ 56.7%, RDED 42.6%, CDA 49.8%. Cross-architecture at IPC=10 (Table 6, nine backbones): CV-DD leads on every one, e.g. ResNet-101 57.2% vs EDC 51.7%, RegNetX-8GF 60.9% vs SRe2L++ 53.4%. Cost: committee "prior evaluation" (pretraining + distillation + evaluation per backbone) totals 84.7 h on ImageNet-1K (Table 9a, one RTX-4090 per model, e.g. ResNet-50: 14.4 h pretrain + 4.4 h generation + 1.5 h eval, 23.0 GB peak); full IPC=50 distillation including that prior-evaluation cost is 137.5 h vs a re-estimated 187.5 h for G-VBSM (Table 9b); per-image per-iteration recovery cost is 1.91 ms for CV-DD vs 4.32 ms (G-VBSM) and 4.99 ms (EDC) on an RTX-4090 (Table 10). Ablation: BSSL alone adds +7.0 points to CV-DD at ImageNet-1K IPC=10 (42.5% to 49.5%, Table 3e) and +4.6 to SRe2L++ (38.5% to 43.1%); prior-weighted voting over equal-weighted voting adds a further +1.1 (60.7% to 61.8%, CIFAR-100 IPC=10, Table 3b) and over random weighting +2.0; committee size scaling from one to five backbones raises ImageNet-1K IPC=10 accuracy from 43.1% to 49.5% monotonically (Table 3d). CV-DD as a plug-in on top of RDED (Table 5, ImageNet-1K IPC=10) raises 42.0% to 44.8% with prior voting vs 43.2% with equal voting. No hard-label result is reported anywhere in the paper.

Limitations

No hard-label evaluation is reported, so the accuracy gains cannot be separated from the paper's own soft-label/relabeling machinery. The committee's "prior performance" scores must be computed once per backbone via a full distill-train-evaluate cycle (Algorithm 1) before any synthesis can begin, an upfront cost (84.7 h on ImageNet-1K, Table 9a) the headline efficiency comparison folds into total time but that is not amortizable across datasets. $N{>}2$ experts per step actively hurts accuracy (Table 3c), so the diversity benefit of the committee is capped by how many models can be combined per step, not by committee size $|S|$ itself. Soft-label storage size is not reported. The conclusion flags a general risk: committee members inherit and can propagate biases of the pretrained models used to build the committee.