Prioritize Alignment in Dataset Distillation

What came before

Built directly on datm-2024 as its backbone (same distillation/evaluation configuration, including datm-2024's learned soft labels and difficulty-aligned trajectory segments), and responds more broadly to mtt-2022-style trajectory matching, framing both gradient- and trajectory-matching methods as sharing a common flaw regardless of which segment or expert they match.

The problem

Argues that matching-based distillation methods extract and embed "misaligned information" at two separate points in the pipeline, independent of which trajectory segment is matched: (1) extraction -- the agent (expert) model is trained on the full real dataset even though the synthetic set will only ever need to represent a small compression-ratio-sized slice of it, so some of what the agent learns is never usable by the synthetic data; (2) embedding -- matching shallow-layer agent parameters injects low-level, quickly-saturated texture information that a small synthetic set cannot usefully represent, crowding out the higher-level information deep layers carry.

The idea

Align what the agent model is allowed to extract and what gets embedded in the matching loss with the actual compression ratio: prune the real target dataset (removing the easiest, least-informative samples) before training experts, so experts only ever see data at roughly the same "difficulty density" the synthetic set will need to represent; and mask out shallow-layer agent parameters during matching, using only deep-layer parameters, since shallow layers were empirically found to carry redundant, low-level signal that saturates almost immediately.

How it works

Two add-on modules layered on datm-2024's trajectory-matching pipeline (same synthetic images, learned soft labels, and difficulty-aligned matching window): Filtered Information EXtraction (FIEX) removes a tuned ratio of the easiest real training samples before expert trajectories are trained, so experts are trained on a pruned, harder target set matched to the compression ratio; Filtered Information EMbedding (FIEM) adds a parameter-selection mask that excludes a ratio of shallow-layer agent parameters (ordered shallow-to-deep) from the trajectory-matching distance, so only deeper-layer parameters contribute to the loss. Both ratios are tuned hyperparameters. What is stored: identical to datm-2024 -- per-class synthetic images (pixels) plus learned soft labels; expert trajectories remain a synthesis-time cache, now trained on pruned data.

Evidence

Table 4 (ConvNet, learned soft labels, DATM value in parens): CIFAR-10 IPC1/10/50/500/1000 = 47.2+/-0.6 (46.9) / 67.4+/-0.3 (66.8) / 77.0+/-0.5 (76.1) / 84.6+/-0.3 (83.5) / 86.7+/-0.2% (85.5) vs. full-data 84.8+/-0.1%; CIFAR-100 IPC1/10/50/100 = 28.4+/-0.5 (27.9) / 47.8+/-0.2 (47.2) / 55.9+/-0.3 (55.0) / 58.5+/-0.3% (57.5) vs. full-data 56.2+/-0.3%; Tiny-ImageNet IPC1/10/50 = 17.7+/-0.2 (17.1) / 32.3+/-0.4 (31.1) / 41.6+/-0.4% (39.7) vs. full-data 37.6+/-0.4% -- PAD beats datm-2024 at every IPC in this table, by margins of roughly 0.3-1.9 points. Cross-architecture (CIFAR-10 IPC50): PAD 55.91/52.35/44.97/45.92% on ConvNet/ResNet18/VGG/AlexNet vs. DATM 55.03/51.71/45.38/45.74% -- PAD slightly ahead on ConvNet/ResNet18 but marginally behind DATM on VGG/AlexNet. Against RDED (Table 8, a decoupled-synthesis/realistic-selection method, outside this family): PAD wins CIFAR-10 IPC1/50/500 (26.8/62.3/89.6% vs. RDED's 23.5/50.2/87.2%) and CIFAR-100 IPC10/50 (50.2/59.3% vs. 48.1/57.0%), but *loses* to RDED on CIFAR-100 IPC1 (16.4% vs. 19.6%) and on Tiny-ImageNet IPC1/10 (10.1/37.2% vs. 12.0/39.6%) -- direct evidence that a decoupled-synthesis method beats the trajectory-matching family's best result at exactly the harder, more-classes end of the CIFAR/Tiny-ImageNet range. Cost: run on 4x NVIDIA A100 (80GB each), the same hardware as datm-2024. Ablation (Table 3, two benchmarks): neither FIEX nor FIEM alone reaches the full method -- e.g. baseline 66.7%, +FIEM 66.9%, +FIEX 67.2%, both 67.4% (one benchmark), and baseline 55.0%, +FIEM 55.5%, +FIEX 55.8%, both 56.2% (another) -- isolating extraction-stage and embedding-stage misalignment as two separate, additive, real problems, with FIEX (data pruning) contributing somewhat more than FIEM (layer masking) in both cases. A separate comparison shows FIEX beats BLiP (a prior data-utility-based pruning method) at matched IPC.

Limitations

The FIEX pruning ratio and FIEM shallow-layer-masking ratio are additional hyperparameters tied to the specific compression ratio being targeted, tuned per dataset/IPC rather than derived analytically. Inherits datm-2024's full cost and label-regime profile: the same cached-expert-trajectory infrastructure, 4x 80GB-A100 compute, and learned soft labels (so PAD's numbers are not comparable to any hard-label trajectory-matching result). No ImageNet-1K evaluation. Against RDED, a much cheaper decoupled-synthesis method outside this family, PAD's advantage is inconsistent -- it loses on CIFAR-100 IPC1 and Tiny-ImageNet IPC1/10, suggesting the alignment fix narrows but does not close the gap to non-bilevel, teacher-driven synthesis at the harder end of the benchmark range.