FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation

What came before

Directly re-evaluates RDED, EDC and CV-DD under one recipe. Frames all of them, plus SRe2L/G-VBSM, as uni-level pixel-space optimization methods that only reference a surrogate teacher loss once real data has been used for initialization, never touching real pixels again during optimization.

The problem

Proves an information-theoretic bound (mutual information between synthetic and real data is upper-bounded by the number of classes times the recovery model's output entropy) showing that pixel-space optimization against a surrogate loss alone cannot avoid "information vanishing" — real-image detail present at initialization is progressively overwritten as optimization proceeds, independent of how long synthesis runs.

The idea

Borrow the residual-connection idea from model architecture (skip connections preserve gradient flow across depth) and apply it at the data level: periodically re-blend the currently-optimized synthetic image with a resampled patch of the real initialization image at a tunable mixing ratio, so newly optimized signal never fully overwrites the real image's local information.

How it works

Single-model FADRM recovers/relabels with ResNet-18; multi-model FADRM+ ensembles ResNet-18, DenseNet-121, ShuffleNetV2, MobileNetV2. The "Adjustable Residual Connection" periodically (every $b=\lfloor B/(k{+}1)\rfloor$ steps, $k$ times total) re-mixes the image as $\tilde x\leftarrow\alpha\tilde x+(1-\alpha)\,\mathrm{Resample}(P_s,\cdot)$ where $P_s$ is the real initialization patch, before a final unmixed optimization stage; best settings $\alpha{=}0.5$, $k{=}3$. Mixed-precision training (FP16 forward/logits, FP32 gradients) and multi-resolution optimization (optimize at a downsampled size, e.g. 200px, then refine) cut synthesis time and peak memory roughly in half. Relabeling uses a batch/crop-based soft-label scheme (BSSL, following Xiao & He 2024) rather than SRe2L's dense per-crop FKD store; no explicit label storage size is given. Post-eval: AdamW, cosine LR, KL-divergence loss, RandomResizedCrop+Flip(+CutMix), 300 epochs (ImageNet-1K/Tiny-ImageNet at IPC 10/50) or 1000 epochs (CIFAR-100, and Tiny-ImageNet IPC=1); evaluated on ResNet-18/50/101, EfficientNet-B0, MobileNetV2, ShuffleNetV2, Swin-Tiny and others.

Evidence

ImageNet-1K, ResNet-18, IPC=10, re-run baselines in FADRM's own Table 1: RDED 42.0%, EDC 48.6%, CV-DD 46.0%, FADRM 48.4%, FADRM+ 50.9% (the paper's abstract instead quotes 47.7%/50.0%, which the paper's own patch-configuration ablation (Table 3, "2x2" variant) shows is a different residual configuration than Table 1's default — the two headline numbers in the paper are not the same setting). IPC=50: RDED 56.5%, EDC 58.0%, CV-DD 59.5%, FADRM 60.1%, FADRM+ 61.2%. CIFAR-100 IPC=10: FADRM 67.4%, FADRM+ 67.9% vs EDC 63.7%, CV-DD 62.7% (re-run). Efficiency (Table 2, one RTX-4090, ImageNet-1K): 0.47s/image and 2.9GB peak memory for FADRM vs 4.99s/2.99s and 17.9GB for EDC and 8.20s and 23.4GB for CV-DD; FADRM+ saves roughly 54 GPU-hours versus EDC generating a full IPC=50 ImageNet-1K set. Key ablation (Table 5): removing the residual connection and multi-resolution optimization drops accuracy from 47.7% to 46.4%; sweeping the mixing ratio $\alpha$ from 0.9 down to 0.4 peaks at $\alpha{=}0.5$ (47.7%), showing a moderate, not maximal, amount of real-pixel remixing is optimal. No hard-label result is reported.

Limitations

States the residual assumption may not generalize to non-visual, abstract modalities (language, time series), and flags a risk of perpetuating source-data biases without auditing. Periodic real-pixel remixing directly reduces the "novelty" of the synthesized image relative to pure pixel-space optimization, and the paper itself shows too many residual re-mixes ($k>3$) reintroduces redundant local detail at the expense of global structure. Multi-model FADRM+ costs more than twice the time and memory of the single-model variant for its accuracy gain.