Dataset Condensation with Contrastive Signals

What came before

Responds to dc-2021 (DC) and dsa-2021 (DSA), both of which match gradients class-by-class (each synthetic class's gradient is matched only to its own real class's gradient), and to selection-based baselines (random, C-score, GraNd, EL2N) and kip-2021 (KIP), which it also compares against directly on fine-grained subsets of ImageNet classes.

The problem

Proves formally and empirically that class-wise gradient matching can perform *worse* than random real-image selection when a class's images are dominated by task-irrelevant, class-common features (e.g. wheels, headlights and roads shared by every "truck" image) rather than task-relevant, class-discriminative features (a truck's logo or trailer shape). Because the matching loss for each class is computed independently, nothing in the objective tells it to prefer the discriminative features over the common ones, so it can spend the synthetic set's limited capacity on information that does not help classification.

The idea

Add an inter-class contrastive term to the gradient-matching loss so that, alongside matching each class's gradient to its own class's real gradient, the loss also explicitly discourages the synthetic set from producing gradients that other classes could equally well produce -- steering the limited synthetic capacity toward class-discriminative rather than class-common information. Stabilize the resulting harder optimization with a short bi-level warm-up phase before switching to the single-level contrastive matching.

How it works

Builds on dc-2021's per-step gradient-matching objective (network re-initialized repeatedly; gradients on synthetic vs. real class batches matched via a distance $D$), but replaces the purely class-wise loss with a modified objective (Sec. 3) that adds a contrastive term pushing a class's synthetic gradient away from other classes' real gradients, not just toward its own. Training begins with a short bi-level warm-up phase (directly optimizing a short unrolled objective, dd-2018-style) before switching to the single-level contrastive matching loss for the remainder of synthesis, which the paper shows via tracked kernel-velocity analysis stabilizes early optimization. DCC denotes the method applied to plain (unaugmented) matching; DSAC denotes the same contrastive loss combined with dsa-2021's Siamese differentiable augmentation. Only per-class synthetic images (pixels, hard labels) are stored; no extra artifact beyond dc-2021's.

Evidence

Table 2 (fine-grained ImageNet-derived classes, ConvNet, hard labels; ipc=10 shown): on Automobile, Random 12.2% vs. DC 11.0% (DC *below* random) vs. DSA 19.1% vs. DCC 18.6% vs. DSAC 22.1%; on Terrier, Random 5.6% vs. DC 4.6% (below random) vs. DSA 5.1% vs. DCC 6.4% vs. DSAC 6.2%; on Fish, Random 14.7% vs. DC 13.5% (below random) vs. DSA 18.7% vs. DCC 20.4% vs. DSAC 22.3% -- DC falls below random selection on 3 of 6 fine-grained subsets tested, while DCC/DSAC recover above random and above DC/DSA in every case. Table 3 (SVHN/CIFAR-10/CIFAR-100, DC/DSA/DCC/DSAC): SVHN IPC1/10/50 -- DC 34.6/76.2/ 82.7%, DSA 36.0/78.9/84.4%, DCC 34.3/76.2/83.3%, DSAC 47.5/80.5/87.2% (full 92.1+/-0.2%); CIFAR-10 IPC1/10/50 -- DC 28.2/44.7/54.8%, DSA 28.7/52.1/60.6%, DCC 32.9/49.4/61.6%, DSAC 34.0/54.5/64.2% (full 81.6+/-0.3%); CIFAR-100 IPC1/10/50 -- DC 12.8/26.6/32.1%, DSA 13.9/32.4/38.6%, DCC 13.3/30.6/ 40.0%, DSAC 14.6/33.5/39.3% (full 52.5+/-0.3%) -- DCC beats DC at every setting; DSAC beats DSA at every setting except CIFAR-100 IPC50 (39.3 vs. 38.6, a narrow win) and CIFAR-10 IPC10 where DCC alone trails DSA (49.4 vs. 52.1, showing the contrastive term alone is not always enough without augmentation). Cross-architecture (condensed data evaluated across ConvNet/LeNet/AlexNet/VGG/ResNet): DSAC reaches 64.1/42.6/48.2/56.0/53.9% vs. DSA's 60.4/40.3/46.0/50.7/49.7% and DC's 54.8/33.8/40.9/ 39.3/23.9% -- consistently ahead at every architecture. Ablation (Table, bi-level warm-up on/off, CIFAR-10/100 IPC1/10/50): warm-up adds 0.1-4.6 points depending on setting (e.g. CIFAR-10 IPC1 DCC 28.3->32.9 with warm-up; CIFAR-100 IPC10 DSAC 29.3->33.5), confirming the warm-up's stabilizing effect though the gain is uneven across settings and datasets.

Limitations

The below-random-selection critique and the contrastive fix are demonstrated mainly on constructed fine-grained/task-irrelevant-heavy ImageNet subsets and standard SVHN/CIFAR-10/100 benchmarks, not at Tiny-ImageNet or larger scale; still single-step gradient matching at its core, inheriting the short-horizon-signal weakness the trajectory-matching family was built to address. The contrastive term alone (DCC) does not always beat augmentation alone (DSA) -- e.g. CIFAR-10 IPC10, DCC trails DSA -- so the strongest results (DSAC) require combining both fixes rather than the contrastive signal replacing augmentation. No GPU-hour or memory figure reported.