Hard Labels In! Rethinking the Role of Hard Labels in Mitigating Local Semantic Drift

What came before

SRe2L stores one soft label per crop, and LPLD reduces label-storage cost by capping the number of stored soft labels per class (SLC) and reusing them across training epochs. GIFT mixes hard information into soft targets directly; the paper's own Joint Objective baseline sums a soft and a hard loss with a fixed weight.

The problem

Storing per-crop soft labels is expensive (ImageNet-1K distilled images: 750 MB vs. soft labels: 28.33 GB, i.e. up to 38x larger, App. F.2/Table 21). Reducing crops per image (LPLD's approach) saves storage but the paper shows it introduces Local-View Semantic Drift (LVSD): with only a few stored crop-level soft labels, ambiguous local crops can have teacher predictions that point toward the wrong class, and Theorem 3.5/3.6 formally lower-bound the resulting train/ideal-objective gap and excess generalization loss at $\Omega(1/s)$ in the number of stored crops $s$, vanishing only as $s\to\infty$.

The idea

Hard labels are content-invariant (a crop's ground-truth class does not depend on which part of the image it shows), so briefly training on ground-truth-anchored, heavily label-smoothed CutMix targets between two soft-label training phases acts as a variance-reducing corrective signal that recalibrates the model before returning to fine-grained soft supervision, without needing to store any additional labels.

How it works

A three-phase Soft-Hard-Soft schedule applied to an already-distilled image set (default: images generated by FADRM; also tested on SRe2L/LPLD/RDED images). Phase A trains on a fixed pool of $s{=}\text{SLI}\times\text{IPC}$ pre-generated teacher soft labels (crop-level, from the frozen teacher) for $T_A=\lfloor n_{\text{soft}}/2\rfloor$ epochs. Phase B (de-LVSD) trains for $n_{\text{hard}}$ epochs on freshly-resampled crops and CutMix pairs labeled with heavily label-smoothed, ground-truth-anchored targets $t_{\lambda,\alpha}(y,y')$ (not strict one-hot) — zero additional storage since these targets are computed on the fly from the class index, not the teacher. Phase C resumes Phase A's fixed soft-label pool for the remaining $T_C$ epochs. No new teacher, generator, or image synthesis is introduced; the mechanism is purely a training-schedule change over an existing soft-label budget. Theoretical support: Theorem 3.7 shows soft- and hard-label gradients become increasingly cosine-aligned as training converges (validated in Fig. 3), and Corollary 3.8 argues this alignment lets hard-label calibration act as if it increased the effective number of soft-label samples.

Evidence

ImageNet-1K, ResNet-18, using FADRM-generated images, relabel+KD from a fixed soft-label pool plus HALD's label-smoothed hard-calibration phase (Table 2): at SLI=5 (SLC=250, 475 MB), IPC=50, HALD 49.5% vs its own re-run Soft-Only baselines FADRM 45.5%, LPLD 39.4%, SRe2L 39.5%, RDED 34.9%; at SLI=10 (SLC=500, 950 MB), IPC=50, HALD 53.7% vs FADRM 52.7%, LPLD 48.6% (marked as the reported number). IPC=10, SLI=5 (SLC=50, 95 MB): HALD 30.3% vs LPLD 14.5%, FADRM 16.1%. Storage-matched headline (Table 5, IPC=50): at a fixed 285 MB budget, HALD 42.7% vs LPLD 33.7% (+9.0 points); at 95 MB, HALD 36.9% vs LPLD 14.3% (+22.6); this is the abstract's headline "42.7% with 285 MB (100x storage reduction vs. per-crop FKD's 28.33 GB), +9.0 over LPLD." No image synthesis time is reported since HALD reuses an existing generation method's images; only label-side storage is measured. Ablation isolating the mechanism (Table 3): a Joint Objective (single loss = soft + $ \lambda$*hard) baseline peaks at $\lambda{=}0$ and degrades as $\lambda$ increases, i.e. naively mixing hard and soft losses in one objective *hurts*, while HALD's staged soft-hard-soft schedule with the identical label budget outperforms both Soft-Only and Joint Objective — isolating the staged schedule itself, not merely "using hard labels," as the source of the gain. Table 15 (Appendix) further shows teacher-prediction agreement on unseen data improves after the final soft-refinement phase compared to omitting it, isolating that phase's contribution. No result is reported for a schedule that skips Phase A or Phase C (pure hard-label training throughout).

Limitations

HALD is a training-schedule/label-regime method layered on top of an existing image generator (FADRM by default); it introduces no new synthesis and its accuracy gain depends on the quality of whichever generation method supplies the underlying images (its own Table 2 shows the gain over Soft-Only shrinks or reorders across SRe2L/RDED/LPLD/FADRM images). The hard-label phase uses heavily label-smoothed CutMix targets rather than strict one-hot labels, so it is not a test of pure hard-label training, and no ablation isolates $\alpha{=}0$ (true one-hot) hard supervision. The soft/hard/soft epoch split depends on estimating $n_{\text{soft}}$, the convergence budget for the soft-only phase, which the paper sets heuristically rather than deriving in closed form. Results are reported only on Tiny-ImageNet and ImageNet-1K classification; no result on CIFAR or on datasets without a decoupled-synthesis-style crop/relabel evaluation pipeline is given (though Table 6 extends to CIFAR-100/ImageWoof for the ablation set). The paper's own conclusion flags a risk of reinforcing bias present in hard labels and over-calibrating to a specific dataset.