CAFE: Learning to Condense Dataset by Aligning Features

What came before

Gradient matching (dc-2021) and its differentiable-augmentation refinement (dsa-2021) synthesize images by matching per-step gradients between real and synthetic batches inside a bi-level loop.

The problem

The paper diagnoses that only a small number of hard or noisy samples produce dominant gradients late in training, so gradient matching overfits to those samples and lacks global supervision of the data distribution; because which samples are "hard" varies by architecture, this also hurts cross-architecture generalization.

The idea

Explicitly align real and synthetic feature statistics at every layer of a network instead of matching gradients, keep a discrimination loss so the synthetic feature centers remain valid classifiers, and replace the fixed inner/outer loop counts of gradient matching with a dynamic schedule that stops each loop once validation accuracy stops improving.

How it works

Optimizes synthetic pixel images. A 3-layer ConvNet is alternately trained on the current synthetic set (inner loop, re-initialized periodically) and used to compute a layer-wise feature alignment loss -- MSE between real and synthetic class-wise mean feature maps at every convolutional layer -- plus a discrimination loss in which synthetic feature centers classify real samples via cross-entropy. A "dynamic bi-level optimization" (Algorithm 1) uses performance queues over recent validation accuracy to decide when to end the inner and outer loops, rather than a fixed iteration budget. The scheme is still bi-level: unlike the later distribution-matching methods it inspired, a network is trained on the synthetic data throughout synthesis.

Evidence

SVHN, 1 IPC, ConvNet (avg of 100 trained networks): CAFE 42.6% vs. DC 31.2% (+11.4pp, Table 1). CIFAR-10, 50 IPC: CAFE 55.5% vs. DC 53.9%. CIFAR-100, 50 IPC: CAFE 37.9% vs. DC 42.8% (CAFE underperforms here). Training time (Table 5, CIFAR-10 IPC=50): DC ~460 min vs. CAFE ~367 min (gamma=10) or ~117 min (gamma=5) -- the speedup comes from adaptive early stopping of the loops, not from removing the inner loop. Ablation (Table 2, CIFAR-10 IPC=50): discrimination loss alone 49.78%, layer-wise feature alignment alone 53.96%, both 54.53%, adding the dynamic schedule 55.50%.

Limitations

Only evaluated at MNIST/CIFAR/SVHN (32x32) scale, no ImageNet. Still requires training a network on the synthetic set throughout synthesis, so it does not reach the efficiency of the bi-level-free distribution matching (dm-2023) it is grouped with. Underperforms plain gradient matching (DC) on CIFAR-100 at 50 IPC. Cross-architecture accuracy remains modest (25-40%) despite the gain over DC. The dynamic schedule adds tunable thresholds (lambda_1, lambda_2, gamma) with a reported sensitivity study but no principled way to set them without a validation split.