GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost

What came before

Decoupled/teacher-relabel methods (SRe2L, RDED, G-VBSM, CDA) and trajectory-matching methods with learned soft labels (DATM) each pick their own loss for training the student on soft labels (e.g. SRe2L uses KL divergence) without justifying the choice or testing sensitivity to it; RDED evaluates specifically with the AdamW optimizer.

The problem

Different distillation methods use different, unexamined loss functions for soft-label supervision, and the paper shows this choice moves accuracy by double digits: swapping SRe2L's KL-divergence loss for soft cross-entropy costs 13.3 points on Tiny-ImageNet (IPC10), and changing RDED's optimizer from AdamW to Adam (keeping its KL loss) drops accuracy from 47.5% to 17.8% (Table 8). So headline numbers partly reflect a loss/optimizer pairing tuned to one recipe, not the intrinsic value of the synthetic images or labels.

The idea

Two changes make soft-label training both stronger and robust to optimizer choice at no added compute: (1) refine the stored soft label by blending it with a smoothed hard label (correcting for the teacher's own errors and adding inter-class dispersion the soft label alone lacks), and (2) replace the training loss with a cosine-similarity-based loss, which a mutual-information argument (an upper bound on InfoNCE/V-information restated via cosine similarity, Theorem 1) shows is close to optimal and which is scale-invariant, removing the loss-magnitude sensitivity that makes Adam/AdamW updates blow up or vanish.

How it works

Plug-and-play, applied at evaluation/training time only (no change to how images or the original labels were synthesized): (1) Label refinement combines the pre-generated soft label y-tilde with a label-smoothed hard label y via a convex combination y_refined = gamma*y/||y|| + (1-gamma)*y-tilde/||y-tilde||, gamma=0.1 found optimal empirically; (2) trains the student with a cosine-similarity loss between student output and refined label instead of KL/cross-entropy. In the loop: whatever teacher(s)/experts the base method (SRe2L/RDED/G-VBSM/CDA/DATM) already used to generate soft labels; GIFT adds no new network and no new forward/backward pass beyond the loss computation.

Evidence

All results add GIFT on top of an existing method's fixed images+soft-labels, i.e. isolate the label-utilization loss under the teacher relabel-with-KD regime. CIFAR-100/Tiny-ImageNet, ConvNet eval (Table 1): consistent gains across SRe2L/RDED/DATM/G-VBSM, e.g. SRe2L+GIFT +4.3 points on CIFAR-100 IPC10 (33.7->38.0), RDED+GIFT +2.5 (47.5->50.6); on ResNet-18 eval gains are smaller (+0.3 to +1.6). ImageNet-1K (Table 2, ConvNet/ResNet-18): RDED+GIFT +1.8 at IPC10/ResNet-18 (41.4->43.2), G-VBSM+GIFT +1.2; gains shrink as IPC grows (e.g. RDED +0.5 at IPC100). Higher-IPC CDA comparison (Table 3) up to IPC200 on ImageNet-1K: CDA+GIFT +0.5 (63.4->63.9). Cross-optimizer generalization (Table 8, IPC10): the headline effect — RDED with SGD goes from 1.9% to 53.4% (+51.5) with GIFT, and with Adam from 17.8% to 53.7% (+35.9); SRe2L with SGD goes from 1.5% to 43.0% (+42.6) on CIFAR-100 and 0.6% to 43.8% (+43.2) on Tiny-ImageNet — showing the base methods are nearly untrainable with SGD/Adam without GIFT and were implicitly tuned to AdamW. Cost (Table 6, IPC10, ResNet-18): GIFT adds no extra memory and only 1-70 seconds of extra training time out of ~2000s total on ImageNet-1K (~3% overhead), confirming the "near-zero cost" claim. Ablation (Section 5.6) shows both label refinement and the cosine loss contribute; the cosine loss is compared against alternative loss functions and found best, and gamma=0.1 is the tuned optimum (Figure 2).

Limitations

All reported gains are relative to a fixed base method's images and initial soft labels — GIFT cannot fix a poor synthesis method, only how its labels are used, so it never appears as a standalone comparison against random-image or coreset baselines under its own protocol. Gains shrink as IPC grows and are smaller on ResNet-18 than ConvNet, suggesting benefits partly compensate for weaknesses in low-data/loss-mismatch regimes rather than adding new information. The paper does not report a no-GIFT-no-teacher (hard label) baseline in its main tables, so it cannot be used to gauge how much of overall accuracy soft labels contribute versus this specific loss-function fix. DATM's ImageNet-1K synthetic data was unavailable so DATM+GIFT is untested at that scale. No formal "limitations" section is given by the authors.