Hierarchical Features Matter: A Deep Exploration of Progressive Parameterization Method for Dataset Distillation

What came before

GLaD fixes one intermediate StyleGAN-XL layer, chosen by manual preliminary search, as the optimization space for a given matching method (DC/DSA/DM/MTT/TESLA); the paper shows this fixed choice is architecture- and dataset-dependent, and can even underperform plain pixel-space distillation in the same-architecture setting it was tuned for (Fig. 1, e.g. TESLA+GLaD 50.7% vs. Pixel 51.7% on ImageNet-A, Table 1).

The problem

A single fixed GAN layer cannot serve every combination of dataset, matching method and IPC well, and finding the right layer by manual, per-setting search is itself expensive; meanwhile diffusion-based generative distillation (Minimax, D4M) degrades toward coreset-like random sampling at the extreme compression ratios (IPC=1/10) that GAN-latent optimization methods target.

The idea

Do not commit to one GAN layer: progressively optimize through every hierarchical layer of the frozen generator in sequence, carrying the best latent from one layer forward as the next layer's initialization, and replace expensive real-accuracy evaluation of intermediate candidates with a cheap class-relevant feature-distance proxy (gradient-weighted class-activation-map distance) so the layer search stays affordable.

How it works

The same frozen, per-dataset StyleGAN-XL used by GLaD is decomposed into layers $G_{K-1}\circ\cdots\circ G_0$. Starting from an averaged-noise latent (mean of several Gaussian samples through the mapping network, avoiding costly GAN-inversion/clustering initialization), each layer $G_i$ is optimized for a fixed number of SGD steps (100 for TESLA/DSA, 20 for DM) against the chosen matching loss $\mathcal{L}= \mathcal{M}(\phi(G_{K-1}\circ\cdots\circ G_i(z_i)),\phi(\mathcal{T}))$; within that budget the latent with lowest class-relevant feature distance $\mathcal{D}(\mathcal{S},\mathcal{T})$ (Eq. 6, built from a pretrained feature extractor's activations weighted by the CAM gradient of the true class) is kept as $z_i^*$ and passed through $G_i$ to initialize the next layer. After traversing all layers to the pixel domain, the single best-performing intermediate synthetic set (by real validation accuracy, tracked only across the small number of per-layer candidates) is output. The generator itself is never fine-tuned; only the latents are optimized, exactly as in GLaD. The framework is validated on top of DC, DSA, DM, MTT/TESLA (bilevel/matching methods) and, separately, on top of SRe2L (decoupled, BN-statistics-matching synthesis) to reach Tiny-ImageNet and full ImageNet-1K.

Evidence

ImageNet-Subset (128x128) · IPC=1 · same architecture as backbone · hard labels (Table 1): H-PD beats both Pixel and GLaD for every base method, e.g. TESLA: Pixel 51.7% -> GLaD 50.7% (worse than pixel) -> H-PD 55.1% on ImageNet-A; DM: Pixel 39.4% -> GLaD 41.0% -> H-PD 42.8%. Cross-architecture (Table 5, IPC=1): TESLA ImageNet-A Pixel 33.4% -> GLaD 39.9% -> H-PD 40.2%; DM ImageNet-A Pixel 27.2% -> GLaD 31.6% -> H-PD 34.9%. Against diffusion-based generative methods at IPC=1 on ImageNette/ImageWoof/Birds/Fruits/Cats, re-run under GLaD's hard-label, no-mixup, no-training-time-matching protocol for a fair comparison (Table 6): H-PD 45.4%/28.3%/39.7%/25.6%/29.6% vs. Minimax 22.8%/17.8%/23.2%/17.5%/19.8% vs. D4M 15.2%/17.4%/18.2%/17.6%/23.4%, i.e. optimization-based GAN-latent search greatly outperforms diffusion sampling once both are stripped to the same hard-label, no-teacher, extreme-IPC protocol. Extended to decoupled synthesis via SRe2L on Tiny-ImageNet and ImageNet-1K, ResNet-18 (Table 2): at IPC-1, Pixel 0.1% -> GLaD 1.2% -> H-PD 2.6% on ImageNet-1K; at IPC-10, 21.3% -> 21.9% -> 23.5%. Cost (Table 7, ImageNet-A-E average): H-PD roughly matches or beats GLaD's wall-clock time despite searching more layers, e.g. DM: GLaD 64 min / 37.4% vs. H-PD 15 min / 39.1% (fewer optimization steps per layer for DM). Key ablation (Table 8, incremental): starting from GLaD-TESLA 51.9% (ImageNet-B), adding averaged-noise initialization alone gives 53.5%, adding hierarchical-layer search on top gives 56.2%, and adding the class-relevant distance metric on top of that gives 57.4% — hierarchical search is the dominant component, and the distance-metric gain is only accessible once the space is unfixed.

Limitations

Only validated as an add-on to GAN-latent parameterization (StyleGAN-XL), not tested with diffusion generators; still requires a StyleGAN-XL trained on (or matched to) the target dataset, so it inherits GLaD's generator-availability dependency and does not test out-of-distribution/random generators as GLaD did; the per-layer greedy search with early stopping by a proxy metric is not guaranteed to find the jointly optimal layer/latent combination; absolute accuracy at full ImageNet-1K IPC=1 remains very low (2.6%) even though it triples GLaD's number; the paper notes the released GLaD code lacks TESLA's CIFAR-10 augmentation/hyperparameters, which depresses the ConvNet baseline it compares against on CIFAR-10 (Table 4).