Enhancing Dataset Distillation via Non-Critical Region Refinement

What came before

SRe2L updates every pixel of a synthetic image by inverting BN statistics, capturing class-general information but losing fine-grained detail. RDED instead selects and stitches high-confidence real crops (via a pretrained observer), capturing instance-specific detail but with no further refinement toward class-general structure. Both store dense per-crop soft labels (SRe2L: up to 120 GB for ImageNet-1K at IPC=200), a cost the paper frames as a separate open problem.

The problem

Methods that refine all pixels toward class-general features (SRe2L-style) discard instance- specific detail; methods that select real patches without refinement (RDED) capture instance detail but no class-general structure. The paper diagnoses this as an unforced binary choice, and separately diagnoses the soft-label storage cost of every prior relabel+KD method as needlessly large, since a full 1,000-dimensional soft-label vector is stored per crop when only a scalar signal is later used to compute the KD loss.

The idea

Use Class Activation Mapping to separate an image into critical (instance-specific) and non- critical (safe-to-change) regions, and update only the non-critical pixels toward class-general statistics, leaving the critical, patch-selected content untouched. Separately, replace the stored soft-label vector with just two scalar cross-entropy distances (from the mixed image's soft label to each of the two source images' one-hot labels), reasoning that only those two distances are needed to reconstruct the KD-style training signal.

How it works

Three stages, one pretrained teacher $\mathcal{T}$, no student in the loop. (1) Critical-based Initial Data Discovery: compute per-image CAM, crop into patches, keep the top-$t$ highest-CAM-value patches in a pool; unlike RDED (which keeps the highest-confidence patches), NRR-DD selects the $g=\beta\times\text{IPC}$ *lowest*-confidence patches from that pool (the "hardest" examples) and combines $\beta$ of them per synthetic image, as in RDED's stitching. (2) Non-Critical Region Refinement: build a per-pixel mask $M=\max\{0,\epsilon-C\}$ from the CAM matrix $C$ (near-zero on high-CAM pixels, larger on low-CAM pixels) and update the image only where $M>0$: $\tilde x \leftarrow \tilde x - M\times\eta\nabla_{\tilde x}\mathcal{L}_C$, with $\mathcal{L}_C=\mathcal{L}_{ce}(\mathcal{T}(\tilde x),\tilde y)+\alpha_{bn}\mathcal{L}_{bn}( \mathcal{T}(\tilde x))$ (cross-entropy plus SRe2L-style BN-statistics matching). (3) Distance-Based Representative (DBR) knowledge transfer: for a CutMix/Mixup pair $(\tilde x_{org}, \tilde x_{aug})$, compute the teacher's soft label on the mixed image and store only $d^T_{org}=\mathcal{L}_{ce}( \tilde y_{soft},\tilde y_{org})$ and $d^T_{aug}=\mathcal{L}_{ce}(\tilde y_{soft},\tilde y_{aug})$ (two scalars, not a full soft-label vector); the student is trained by matching its own two analogous distances to these stored targets (Eq. 9), optionally with an added label-refinement term $\mathcal{L}_{lr}$.

Evidence

ImageNet-1K, full soft-label relabel+KD regime (Table 1, IPC values as columns): ResNet-18, IPC=10 NRR-DD 46.1% vs RDED 42.0%, SRe2L 21.3%; IPC=50 NRR-DD 60.2% vs RDED 56.5%, SRe2L 46.8%. ResNet-101, IPC=10 NRR-DD 51.3% vs RDED 48.3%, SRe2L 30.9%; IPC=50 NRR-DD 64.3% vs RDED 61.2%, SRe2L 60.8%. Genuine hard-label and compressed-label results (Table 2, ImageNet-1K, ResNet-18, IPC=10): RDED with soft labels (SL) 42.0%, RDED with one-hot hard labels (OH) 16.3%, RDED with 2-element compact labels (CL) 22.2%, NRR-DD with DBR alone (no label refinement) 34.3%, NRR-DD with DBR+label- refinement 37.2% -- a "Recover rate" (DBR-OneHot)/(SoftLabel-OneHot) of 70%, i.e. DBR recovers 70% of the soft-label-vs-hard-label accuracy gap while storing only 2 scalars per crop instead of 1,000 (IPC=200 ImageNet-1K: 0.2 GB vs 120 GB for full soft labels, a ~500x reduction, stated in the abstract/intro and demonstrated via the Table 2 recover-rate results at IPC=10/50). At IPC=1, DBR (no soft label at all) already beats RDED-with-soft-labels on several datasets (stated in Section 4.3). Cost (Table 11, ResNet-18, 100 images on ImageNet-1K): NRR-DD 520.65 ms vs SRe2L 2113.23 ms and RDED 39.89 ms; peak memory 9.14 GB, matching SRe2L, versus RDED's 1.57 GB -- refinement costs more than RDED's selection-only pipeline but far less than full BN-statistics inversion from noise. Key ablation (Table 4, referenced in text): CIDD alone (patch selection only, no refinement) already beats RDED; adding NRR on top adds roughly a further 15-point gain on CIFAR-10/ResNet-18, isolating the refinement step from the selection step.

Limitations

Stated: performance depends on CAM quality, which is sensitive to how the teacher itself was trained, and the paper does not explore improving CAMs directly; DBR's memory savings trade off against fine-grained-detail preservation on some complex tasks, left unresolved. Observed: NRR-DD with full soft labels still requires the same per-crop teacher relabeling infrastructure as SRe2L/ RDED to build its Table 1 numbers -- DBR is presented as an alternative, not a replacement, and the paper's headline SOTA claims (Table 1) use full soft labels, not DBR; DBR's own accuracy (34.3-37.2% at ImageNet-1K IPC=10) remains meaningfully below the soft-label number (46.1%), so it recovers most, not all, of the label information; synthesis is slower and more memory-hungry than RDED, the method it is closest to and builds on.