Dataset Distillation with Neural Characteristic Function: A Minmax Perspective

What came before

dm-2023 matches only class-wise mean embeddings (empirical MMD with a linear kernel) in random networks; idm-2023, m3d-2024 and datadam-2023 enrich the embeddings or statistics but still use a fixed, hand-chosen discrepancy. Trajectory-matching methods (mtt-2022 and successors such as DATM) remain the accuracy reference but at far higher memory cost, running out of memory on CIFAR-100 at IPC=50 on an 80GB A100.

The problem

Fixed discrepancy measures used in prior distribution matching -- pointwise MSE or linear-kernel MMD -- only capture a first moment or a hand-picked kernel and are not sufficient conditions for the two feature distributions to actually coincide, so real and synthetic distributions can look matched under the metric while remaining different.

The idea

Cast distribution matching as a minmax game: a lightweight sampling network learns which frequency arguments of the characteristic function make the real/synthetic discrepancy largest (an adversarially optimized, comprehensive discrepancy measure that provably determines the full distribution), while the synthetic images minimize that learned discrepancy. The characteristic-function gap decomposes into an amplitude term (distribution scale, i.e. diversity) and a phase term (distribution centers, i.e. realism).

How it works

Real and synthetic batches are embedded by a hybrid feature extractor formed by beta-blending an early and a late checkpoint (beta ~ U(0,1) each step) drawn from a pool that mixes randomly initialized and pretrained networks. A sampling network psi parameterizes a scale-mixture-of-Gaussians distribution over frequency arguments t (1024 by default); psi is trained to maximize the neural characteristic function discrepancy (NCFD) while the synthetic pixel images (with IDC-style multi-formation, scale factor rho=2) are trained to minimize it, balancing amplitude and phase terms with a hyperparameter alpha. NCFD costs linear time in the number of sampled frequencies versus MMD's quadratic time. Soft labels from a pretrained network and dataset fine-tuning are supported as optional add-ons but the paper states they are not required for the headline numbers.

Evidence

CIFAR-10, IPC=10, ConvNet-3, hard labels: NCFM 71.8% vs. IDM 58.6%, M3D 63.5%, DM 48.9% (Table 1). CIFAR-100, IPC=1: NCFM 34.4% vs. DM 11.4% (+23.0pp) and IDM 20.1%. Tiny-ImageNet, IPC=50: NCFM 29.6% vs. IDM 27.7%. ImageSquawk (128x128), IPC=10: NCFM 72.8% vs. MTT 52.3% (+20.5pp, Table 2). Cross-architecture (CIFAR-10, IPC=50, trained on ConvNet): NCFM 77.4/75.5/75.5/73.8% on ConvNet/AlexNet/VGG/ResNet vs. DM 65.2/61.3/59.9/57.0% (Table 4). Cost (Table 3, CIFAR-100, IPC=50, A100): NCFM 1.36s/iter and 7.22GB vs. TESLA 28.24s/OOM and MTT/FTD/DATM all OOM -- roughly 20x faster and >300x less memory than the trajectory-matching baselines the abstract cites. Ablation (Table 5): removing the sampling network psi (holding data curation fixed) costs 3.2pp on CIFAR-10 IPC=50 (74.2%->77.4%) and 10.1pp on Tiny-ImageNet IPC=10 (14.2%->24.3%), isolating the adversarial frequency-sampling mechanism from the CFD metric itself.

Limitations

The ablation section states explicitly that its "no sampling network" comparison uses no fine-tuning or soft-label integration, implying the headline Table 1/2/4 numbers do include those data-curation add-ons by default; since the DM/IDM/M3D/MTT baselines it is compared against in those tables report hard-label, no-curation numbers from their own papers, part of the gap may be attributable to this protocol difference rather than the NCFD metric alone, and the paper does not report a hard-label-only, no-curation version of its headline table to isolate this. The hybrid random+pretrained feature extractor and the fine-tuning/ soft-label option both introduce a pretrained-teacher dependency; the paper does not quantify its cost. No ImageNet-1K full-scale result is reported (only 128x128 ImageNet subsets and low-resolution benchmarks).