HIERAMP: Coarse-to-Fine Autoregressive Amplification for Generative Dataset Distillation

What came before

Diffusion-guided methods (Minimax, D3HR, CaO2) and inversion-based RDED all shape synthesis or selection toward global semantic proximity to a class prototype or distribution statistic, treating an object as a single point in feature space.

The problem

Global proximity ignores that object semantics are hierarchical -- a bird's eye position is constrained by its head outline, fine texture is constrained by coarse layout -- so optimizing one flat similarity score cannot separately control how much diversity belongs at the layout level versus the detail level, and prior generative methods have no mechanism to inject different guidance at different levels of visual granularity.

The idea

Exploit a visual autoregressive (VAR) model's coarse-to-fine token scales as ready-made levels of visual granularity: inject class tokens at every scale that dynamically identify the salient regions for that scale, and amplify attention toward those regions, which the paper finds naturally diversifies coarse layout choices while concentrating and sharpening fine-detail token usage, rather than optimizing one global-proximity objective.

How it works

Starts from a pretrained VAR (depth 16, 256x256, originally ImageNet-pretrained) and fine-tunes it for 5 epochs to add a class token used for semantic-guided attention (so the generator itself is adapted, not training-free). The model's scales (1-9) are grouped into coarse (1-3, global layout), mid (4-6, semantic structure) and fine (7-9, texture) stages. At each scale $n$, a learnable class token produces an attention map identifying salient positions; a positive logit bias is added to the top-$\rho$% attended positions, $\tilde L_n^{(h)} = L_n^{(h)} + \beta_n \cdot \mathbb{1}_{L_n^{(h)}\in \text{top-}\rho} \cdot a_n^\top$, with amplification strength $\beta_n>0$ tuned per stage (coarse/mid/fine amplification schedule, e.g. 5-5-0.5). Only pixels are stored; the fine-tuned VAR and its class tokens are not part of the distilled artifact.

Evidence

ImageNet-1K, hard-label, ResNet-18/ResNet-101 (Table 1): IPC10 47.6% / 52.1% vs CaO2 46.1% and RDED 44.3% (ResNet-18); IPC50 60.8% / 66.4% vs CaO2 60.0%; IPC100 62.7% (matching CaO2's 62.7%). ImageWoof IPC10, ResNet-18: 45.8% vs RDED 40.1%, D3HR 39.6%. CIFAR-10 IPC10, ResNet-18: 44.3% vs D3HR 41.3%, RDED 37.1%. FID at IPC10 (Table 6): 17.3+/-0.1 vs Minimax 18.3+/-0.2, D3HR 19.0+/-0.2 (lower is better). Only Minimax, D3HR, RDED and CaO2 are compared; no D4M, MGD3 or IGD comparison anywhere, even in related work. Cost (Tables 8-9): inference 0.147+/-0.001 s/image vs D3HR's DDIM-based 0.456+/-0.002 s/image; peak memory 1.840+/-0.001 GB, marginal overhead over the unmodified VAR's 1.770 GB. Ablation (Table 3): 5 amplification steps at 50% top-attention across all stages is the best single configuration (47.6+/-0.1%); Table 4 shows a balanced coarse-mid-fine schedule (5-5-0.5) needed, and coarse-stage amplification contributes the largest single share of the gain.

Limitations

Stated: no dedicated Limitations section (the Conclusion only states the method's achievements). Observed: requires fine-tuning the VAR generator per dataset (5 epochs with a class-token head added), so it depends on the same target-trained-generator pattern the training-free branch of diffusion-guided synthesis is trying to remove; only compared against Minimax, D3HR, RDED and CaO2, omitting D4M, MGD3, CoDA, DAP, IGD and the entire training-free/VLM-conditioned diffusion literature, so its standing against the field's strongest diffusion baselines is untested; label regime is described as hard-label evaluation though the RDED protocol it follows uses a ResNet-18 teacher, leaving the degree of teacher-relabeling ambiguous in the paper's own framing; amplification strengths $\beta_n$ and $\rho$ are tuned per stage without a stated cross-dataset transfer study.