Distilling Dataset into Neural Field

What came before

Static decoders (idc-2022's bilinear upsampling, fred-2023's inverse frequency transform) are fast and training-free but fixed, which the paper argues caps expressiveness; parameterized decoders (haba-2022, rtp-2022) are more flexible but the decoder's own parameters consume storage budget, and deep generative priors (e.g. glad-family latent optimization) need a large pretrained generator and slow backward passes.

The problem

All prior decoding functions map a low-dimensional code into a fixed, instance-sized output, so their representational capacity is tied to the data dimension D: as D grows (high resolution, video, 3D), the decoder or code must grow too, which is exactly the scalability problem parameterization is meant to solve, and no prior parameterization method has a theoretical account of its own expressiveness.

The idea

Store each synthetic instance as a small neural network (a neural field) that maps a coordinate to a value, rather than as a code plus decoder tied to the instance's shape; because a neural field's parameter count is independent of the output resolution and grid dimensionality, the same budget-constrained network can decode images, video frames, audio samples or 3D voxels by simply changing the input coordinate set, and (proven via a feasible-space argument) achieves a strictly larger feasible output space than fred-2023 under an equal per-instance parameter budget.

How it works

A coordinate set $\mathcal{C}$ (the lattice of pixel/voxel/time-sample locations) is fixed and never optimized or stored. Each synthetic instance is one small L-layer sine-activated MLP $F_\psi$ (a SIREN); a decoded instance is $\tilde x=F_\psi(\mathcal{C})$. The synthetic set is $\mathcal{S}=\{(F_{\psi_j} (\mathcal{C}),\tilde y_j)\}_{j=1}^{|\Psi|}$, trained end-to-end under any distillation loss (default: trajectory matching), with a warm-up step that fits each field to a randomly sampled real instance before distillation begins. Budget is the number of parameters $b$ per field (independent of data dimension D) times the number of fields $|\Psi|$, constrained to the total budget $B$; width/depth are tuned to fit. A theorem (feasible-space comparison) shows that, at equal per-instance parameter budget, DDiF's output space strictly contains FreD's, so DDiF should reach a lower or equal distillation loss for the same instance count -- confirmed empirically by a -0.89 correlation between reconstruction error and downstream accuracy. Hard, fixed one-hot labels by default (soft-label compatibility is shown but not used in the headline tables).

Evidence

ImageNet-Subset, 128x128, IPC=1 (#Params=491.52k, equal-parameter budget across all parameterizations): DDiF 72.0% on ImageNette vs. FreD 66.8%, SPEED 66.9%, Vanilla+RTP 69.6%, HaBa 51.9%, GLaD 38.7% (Table 1). ImageSquawk, same budget: DDiF 67.0% vs. FreD 57.0%, SPEED 60.9%. At 256x256, IPC=1: DDiF 67.8% vs. FreD 54.2%, Vanilla (full pixels) 32.1% -- DDiF beats even the uncompressed pixel baseline at higher resolution. Fixed-decoded-instance-count comparison (Table 7): at the same number of decoded instances per class (8), DDiF matches FreD's accuracy (67.1 vs 66.8 on ImageNette) using only 77.04k of the 491.52k budget FreD needs (15.7% of the budget) -- isolating coding efficiency from raw budget size. Cross-objective (Table 4): DDiF improves over baselines when paired with gradient matching (DC) and distribution matching (DM), not only trajectory matching, confirming compatibility beyond its default loss. Other modalities: video (miniUCF) at 1.7% of SDD's (a video-specific method) storage budget reaches competitive accuracy; audio (Speech Commands) and 3D voxel (ModelNet/ShapeNet) each show DDiF as best among tested parameterizations. Ablation (Fig. 7): performance improves consistently across neural-field width and depth choices and is robust to swapping the field architecture for an FFN-based one.

Limitations

States its own limitations (Appendix D.4): less efficient on low-dimensional datasets, because a sufficiently expressive-yet-small neural field is harder to design when the output dimension is already small (the coding-efficiency advantage is shown mainly on 128x128+/multi-modal high-dimensional data, less so at CIFAR-scale). One-to-one correspondence between a synthetic instance and its neural field means there is no shared component storing cross-instance or cross-class information, unlike rtp-2022's shared memories or haba-2022's shared hallucinators/bases -- the paper suggests adding modulation/conditional codes as future work rather than solving it here. As with fred-2023, its default headline loss is trajectory matching (mtt-2022), so absolute numbers inherit that objective's expert-trajectory cost, not separately reported for DDiF's own overhead.