Frequency Domain-based Dataset Distillation

What came before

idc-2022 downsamples images and upsamples with a static bilinear function, which the paper argues is "highly vulnerable to instance-specific information loss" because the spatial domain spreads a signal's important information roughly uniformly across every pixel; haba-2022 uses a parameterized (hallucinator) decoder, which requires a separate storage budget for the decoder network itself.

The problem

Spatial-domain parameterization cannot cheaply identify which dimensions of an instance matter most -- empirically, spatial-domain pixels show a roughly uniform explained-variance ratio and a roughly uniform distillation-loss gradient magnitude across dimensions -- so any spatial compression (downsampling) must discard information indiscriminately rather than selectively.

The idea

Transform each instance to the frequency domain first, where natural images have an "energy compaction" property (information concentrated in a few frequency components); this lets a class-wise binary mask select only the top-k frequency dimensions ranked by explained variance ratio (EVR) for optimization and storage, using a free, parameter-free, differentiable inverse transform (no learned decoder) to reconstruct an image-shaped instance for any downstream distillation loss.

How it works

A synthetic frequency memory $F=\{(f^{(i)},y^{(i)})\}$ stores $d$-dimensional frequency-domain vectors (one per stored instance), each initialized from the DCT of a randomly sampled real image. A class-wise binary mask $M^{(c)}\in\{0,1\}^d$, computed once from the dataset's per-dimension EVR (no training required), selects the top-k dimensions to keep; the other $d-k$ entries are exactly 0 and are not counted against the budget. Training optimizes only the $k$ non-zero frequency coefficients per stored instance under any existing distillation loss $\mathcal{L}_{DD}$ (default: trajectory matching, mtt-2022), applied to $\tilde S=\mathcal{F}^{-1}(M\odot F)$. Because each instance now costs only $k<d$ dimensions, the same total budget $n\times d$ accommodates $|F|=\lfloor n(d/k)\rfloor>n$ instances. Hard, fixed labels.

Evidence

CIFAR-10, IPC=1 (#Params=30.72k, i.e. equal total parameter budget across all methods including pixel baselines): FreD 60.6% vs. second-best IDC 50.0% (+10.6pp, Table 1); SVHN, same budget: FreD 82.2% vs. HaBa 69.8% (+12.4pp). CIFAR-100, IPC=50: FreD 47.8% vs. IDC (not reported)/HaBa 47.0%. Orthogonality across objectives (Table 2, CIFAR-10): applying FreD's parameterization on top of DC/DM/TM losses each gives the best result among parameterizations, up to +10.8pp over the second-best combination and +10.6pp in cross-architecture generalization (AlexNet/VGG11/ResNet18). ImageNet-Subset (128x128), IPC=2 (#Params=983.04k, Table 3): FreD beats IDC and HaBa at equal parameter budget. Decoded-instance increment at equal budget (vs. IPC=1 baseline): IDC and HaBa both decode 5x more instances; FreD decodes 16x more at small budgets (IPC=1), tapering to 4x at large budgets (IPC=50), because larger per-instance budgets let it allocate more dimensions per instance rather than maximizing instance count. Ablation (Fig. 8a): EVR-based masking beats low-pass, band-stop, random, and amplitude-based masking criteria at equal k; high-pass masking is drastically worse (14.3% vs. ~60% at IPC=1). Ablation on frequency transform (Fig. 8b): DCT, DFT and DWT all improve over the spatial baseline, DCT best. Budget-allocation ablation (Fig. 9): smaller budgets favor more instances at lower per-instance k; larger budgets favor fewer instances with higher k.

Limitations

States its own limitations (Appendix G): the frequency-domain approach is demonstrated only for 2D/3D grid data; connecting frequency transforms to non-grid domains such as text is non-trivial and may not be as effective. Performance is "highly dependent on masking strategy" -- EVR masking assumes task-relevant information concentrates in low/mid frequencies, which the paper notes may not hold for tasks where discriminative signal sits in high-frequency regions (e.g. fine medical-imaging detail, digital watermarking), and no data-driven alternative to EVR is evaluated empirically (only a gradient-based masking idea is proposed as future work). Storage accounting treats the $d-k$ zeroed dimensions as free ("negligible small bytes"), i.e. the binary mask's own storage cost is not counted against the budget, which is generous relative to a strict bit-accounting standard.