Dataset Distillation via Vision-Language Category Prototype

What came before

D4M clusters a pretrained diffusion model's latent features into per-class image prototypes and conditions generation on the class label text; Minimax fine-tunes the diffusion model with representativeness/diversity criteria. Both condition synthesis on image features (and at most a bare class name), never on a description of the sample.

The problem

The paper diagnoses that label-only conditioning in D4M and Minimax produces images missing the target object entirely, anatomically incoherent objects (its example: a dog rendered with five legs), and co-occurrence bias where frequently co-occurring background elements (e.g. fish with green plants) dominate over the class subject -- because the generator has no semantic grounding beyond the class name.

The idea

Generate a per-image textual description with an open-source vision-language model (LLaVA), then build one "text prototype" per image cluster by keeping only the words that are frequent within a cluster but not generic to the whole class, and picking the real caption that best matches those words; condition the diffusion model jointly on this text prototype and the existing image-latent cluster prototype.

How it works

Stable Diffusion v1-5 is fine-tuned per dataset (batch 8, 8 epochs) on LLaVA-captioned (image, text) pairs with the standard denoising loss $\mathcal{L}_{DM}=\lVert\epsilon_\theta(z_t,c)-\epsilon\rVert_2^2$ (no extra distillation-specific loss). A pretrained VAE encoder compresses images to latents; Local Outlier Factor removes low-density outliers per class; K-means then partitions each class's latents into IPC clusters, whose centers are the image prototypes $z^c$ (following D4M). For text, per-class LLaVA captions are tokenized into a word-frequency table; words occurring in more than a threshold $\beta$ fraction of a class's captions are marked non-representative and discarded; the remaining per-cluster words are ranked by frequency into a top-$k$ keyword set; every caption in the cluster is scored by keyword overlap and the highest-scoring real caption is kept verbatim as the text prototype $T^c$ ("Descriptions of Closest Samples", DCS) rather than a synthetic sentence. Sampling denoises $z_t^c$ with the U-Net cross-attending to the CLIP text embedding of $T^c$, and the VAE decoder produces the final image; nothing is optimized at synthesis time beyond this conditioning choice.

Evidence

ImageWoof at 256x256 (Table 1), IPC 10/50 with ResNet-18: 39.9±2.6 / 58.9±1.5, versus D4M 32.3±1.2 / 53.7±2.2 and Minimax 35.7±1.6 / 53.7±0.6. ImageNette/ImageIDC on ResNetAP-10 (Table 2), IPC 10/50: Nette 64.8±3.6 / 81.2±0.8 vs D4M 60.9±1.7 / 77.7±1.1; IDC 57.0±1.4 / 71.9±0.4 vs D4M 50.3±1.0 / 69.1±2.4. ImageNet-1K at 224x224 (Table 3), IPC 10/50: 46.7±0.4 / 60.5±0.2 vs Minimax 44.3±0.5 / 58.6±0.3 and RDED 42.0±0.1 / 56.5±0.1. CIFAR-100 (Table 4), IPC 10: 50.6±0.7 vs RDED 42.6±0.2 (+8.0 points). The isolating ablation (Table 5) compares four conditioning texts at fixed image prototypes -- label only (L), label+feature-keywords (L+FK), GPT-generated sentences (GGS), and the paper's real-caption selection (DCS) -- and DCS wins in 5 of 6 dataset/IPC cells (e.g. ImageNette IPC50: 81.2 vs L 76.6, L+FK 76.2, GGS 78.0), showing gains come from selecting a real, cluster-matched description rather than from adding any text at all (GGS, synthetic sentences, occasionally beats DCS, e.g. ImageIDC IPC50 72.1 vs 71.9). Evaluation networks are ConvNet-6/ResNetAP-10/ResNet-18 trained from scratch on the synthetic set; no soft-label teacher-relabeling step is described in the main text.

Limitations

Stated: restricted to classification; extending to detection/segmentation is left to future work. Observed: requires per-dataset fine-tuning of Stable Diffusion on LLaVA-captioned pairs, adding cost beyond training-free members of the family (MGD3, ManifoldGD) that the paper does not quantify against its own reported speed claims; depends on LLaVA caption quality and may inherit its biases or hallucinations into the text prototype; the LOF and text-prototype hyperparameters ($\alpha$, $\beta$, top-$k$) are shown to be sensitivity-prone (Fig. 4) and tuned per dataset; no comparison to CoDA-style training-free text-to-image conditioning or to a diffusion model never trained on the target dataset, so it does not address the target-trained-generator dependency later papers (CoDA, IMS3) raise.