Condensing Large-Scale Datasets Directly with Minimal Information Loss

What came before

The squeeze-recover-relabel pipeline (SRe2L, CDA, G-VBSM, EDC, DWA, CV-DD, DELT, NRR-DD) inverts a pretrained classifier's batch-norm statistics and logits to recover pixels, then relabels with the same classifier. RDED instead selects and stitches real crops using a pretrained observer, with no inversion step at all.

The problem

The paper argues squeeze (data to model) and recover (model back to images) form an implicit dual compression that loses information twice, degrading cross-architecture generalization and costing many optimization iterations. It further shows (Table 7 / App. E) that this loss shifts the distilled images' distribution away from the real data the relabeling classifier was trained on: disabling Relabel collapses SRe2L and G-VBSM to near-chance on ImageNet-1K (1.1% and 0.8% at IPC=10), and applying Relabel post hoc to non-Relabel methods (ADD, DataDAM) helps only in the earliest, least-optimized stage of their distillation before images drift from real-image initialization (Fig. 1) -- evidence, formalized as Proposition 1 (a two-Gaussian distribution-shift bound), that a relabeler trained on real data becomes unreliable once its input distribution shifts.

The idea

Abandon inversion entirely: select real-image subsets per class (by default with RDED's selection score), then directly minimize a bounded "effective information gap" between each real subset and its corresponding synthetic image (RandomCrop views of a real-image-initialized canvas), matching intermediate-layer features of a single pretrained model under multiple transformations rather than matching BN statistics. Because the synthetic images stay distributionally close to real data by construction, the same pretrained model can then relabel them reliably.

How it works

One pretrained observer/teacher network $\phi_{\theta_T}$ (e.g. ResNet-18), no student in the loop. Stage 1 selects, per class, the top $N\times\text{IPC}$ real images by teacher cross-entropy loss from a candidate subset of size 300 (RDED-style, but CIM is stated to be selection-agnostic; Table 6 shows it also works with Random/K-means/Herding selection). Stage 2 initializes each distilled image by concatenating $N{=}4$ selected real images (resized), then iteratively (M=200 iterations) applies RandomCrop augmentations and minimizes an $\ell_2$ feature-gap loss (Eq. 9) between a middle-layer representation of the distilled crop and the corresponding real image, bounding the intractable KL "effective information gap" (Def. 2, Thm. 4.1) by an $\ell_2$ feature distance; middle layers are used (not logits) to preserve texture as well as semantics (Fig. 4c ablation). Relabeling generates one soft label per transformed view $\zeta_k(\tilde x)$ (Eq. 10), extending SRe2L's single-crop relabel to multiple transformation types. No bi-level coupling to a student; images can be optimized one at a time rather than as a synchronized batch.

Evidence

ImageNet-1K, ResNet-18, relabel+KD (Table 2): IPC=10 CIM 48.7% vs DELT 45.8%, NRR-DD 46.1%, GIFT 43.2%, RDED 41.1%, WMDD 38.2%, G-VBSM 35.7%, CIM's own re-run of SRe2L 31.1% (note: this SRe2L re-run differs both from SRe2L's own-reported 21.3% and from DELT's re-run of 41.9% at the same cell -- a third, still different value for the same nominal baseline); IPC=50 CIM 60.4% vs NRR-DD 60.1%, WMDD 57.6%, DELT 59.2%; IPC=100 CIM 62.4%, tied with DELT. ResNet-50 (Table 3): IPC=10 CIM 54.3% vs EDC 54.1%, CV-DD 51.3%, RDED 46.2%; IPC=50 CIM 65.9% vs EDC 64.3%. Cross-architecture at IPC=10 (Table 4, ResNet-18-distilled data transferred): CIM leads on all 6 tested architectures including ViT-T/16 (10.8% vs RDED 8.5%, SRe2L 3.2%). Cost: distills ImageNet-1K IPC=10 in 80 minutes on one RTX-4090 (abstract, Conclusion); Table 5 per-100-image generation time on ResNet-18: CIM 25.34s / 1.56GB vs SRe2L 191.14s / 3.62GB and G-VBSM 259.84s / 4.94GB, but RDED remains faster and lighter (2.78s / 0.92GB) as the optimization-free reference. Genuine hard-label result (Table 7, App. E, ResNet-18, IPC=10, no relabel): ImageNet-1K -- CIM 22.0% vs RDED 19.7%, SRe2L 1.1%, G-VBSM 0.8%; CIFAR-10 -- CIM 51.6% vs RDED 35.2%, G-VBSM 32.3%, SRe2L 10.9%. Key ablation: removing Relabel collapses SRe2L/G-VBSM to near-chance on ImageNet-1K (Table 7) while CIM and RDED (both real-content-anchored) degrade far less, isolating distributional proximity to real data, not the relabeling mechanism itself, as what makes Relabel effective.

Limitations

The paper's own stated limitation (App. A) is that CIM cannot match RDED's optimization-free efficiency despite being much faster than inversion-based methods. The 80-minute headline cost excludes teacher pretraining. CIM still depends on one pretrained observer for both selection-score ranking and the feature-gap objective, so it inherits a dependence on that model's own training recipe (a dependency this family's other papers, e.g. sre2l-2023, already flag as unresolved). Soft-label storage across the multiple transformed views per image is not quantified. Gains over the next-best methods (NRR-DD, DELT) are modest at IPC=50/100 (roughly 0.3-2.8 points) even though the IPC=10 margin is larger, and the hard-label advantage over RDED (Table 7) is smaller than the relabel+KD advantage, suggesting part of CIM's edge still comes from the relabeling step it argues is only reliable once distribution shift is fixed.