P4 · Generative priors · since 2022 · active

Optimization in a generative latent space

Keep a pretrained generator frozen and optimize its latent codes, instead of pixels, with a matching objective.

The bet every member shares

Constraining synthetic samples to a generator's manifold regularizes them toward realistic images, which improves generalization to unseen architectures and shrinks the search space.

Defining objective

\min_{\{z_i\}}\;\mathcal{L}_{\text{match}}\bigl(\{G(z_i)\}_{i=1}^{m},\,\mathcal{T}\bigr),\qquad G\ \text{frozen}

The matching loss can be any of gradient, trajectory or distribution matching; G has been a GAN, a StyleGAN (at chosen intermediate layers) or a latent diffusion model.

How the family developed

The papers that moved the family, in the role each one played.

Origin

IT-GAN · NeurIPS 2022 Workshop

Freezes a pretrained GAN and learns latent vectors whose images are informative for training rather than realistic, presented as a route to dataset condensation.

Evidence

GLaD · CVPR 2023

Distills into intermediate feature vectors of a pretrained generator and, added to existing matching objectives, improves cross-architecture generalization in every setting tested.

Improvement

H-GLaD · CVPR 2025

Optimizes across a hierarchy of GAN layers rather than one fixed latent space, with a class-relevant feature distance to reduce evaluation cost.

Extension

LD3M · NeurIPS 2025

Backpropagates through a pretrained latent diffusion model to learn distilled latents and class embeddings; a decaying skip connection from the initial noise keeps gradients alive across steps.

What it gets right

Better cross-architecture generalization than pixel-space optimization with the same objective.
A lower-dimensional search space and realistic-looking samples.

Where it is weak

Requires a generator that covers the target distribution; its biases become the dataset's.
Still pays the full cost of the underlying matching objective, now through the generator.

Applied to

Papers per year

Competing answers

All papers in this family 9

Papers not already discussed above, ordered by tier, then newest first.Open in the explorer →

2025-03

Condensing Action Segmentation Datasets via Generative Network Inversion

Guodong Ding, Rongyu Chen, Angela Yao · CVPR 2025notableVideo paper ↗

Condenses procedural video datasets for temporal action segmentation into compact latent codes using a generative prior learned from the dataset and network inversion, reducing storage across both the temporal and channel dimensions, plus a diversity-driven sampling step that selects representative action sequences to cut video-wise redundancy; on the Breakfast dataset it reports over 500x storage reduction while retaining 83% of full-dataset segmentation performance, and improves downstream incremental-learning performance.

2024-01

GSDD — GSDD: Generative Space Dataset Distillation for Image Super-resolution

Haiyu Zhang, Shaolin Su, Yu Zhu et al. · AAAI 2024notableDense prediction paper ↗

GSDD distills a low-resolution/high-resolution image-pair dataset for super-resolution by optimizing codes in the latent space of a pretrained GAN via GAN-inversion, rather than selecting or synthesizing pixels directly, so the stored artifact is a small set of latent codes plus the frozen generator. The paper reports super-resolution performance comparable to prior state-of-the-art distillation methods with about an 8x increase in training efficiency and roughly 93.2% less storage, and shows generalization to real-world degraded images.

2023-12

FedDG — Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents

Yuqi Jia, Saeed Vahidian, Jingwei Sun et al. · ECCV 2024notablepaper ↗code ↗

Moves dataset distillation for federated learning from the client to the server: instead of clients distilling and uploading synthetic data, the server uses prior knowledge from pretrained deep generative models' latents to synthesize data representations from heterogeneous client model architectures, letting local devices train smaller surrogate models while the server trains a larger global model on a multi-modal distribution. Reports up to 40% accuracy improvement over non-distillation FL and 18% over existing dataset-distillation FL methods under high heterogeneity, plus faster convergence, with a theoretical argument that the process asymptotically resembles centralized training.

2023-11

Dataset Distillation in Latent Space

Yuxuan Duan, Jianfu Zhang, Liqing Zhang · arXiv 2023paper ↗

Moves three mainstream dataset-distillation algorithms from pixel space into the latent space of a pretrained autoencoder, cutting time and memory cost and allowing more latent codes to be stored than pixel images at the same budget, which extends distillation to higher-resolution datasets and larger data ratios than the pixel-space versions reach.

2023-01

Generative Dataset Distillation

Jovan Cicvarić · University of Tübingen 2023Other data paper ↗

Master's thesis on generative dataset distillation by optimizing latent codes of pretrained GANs (StyleGAN2 and StyleGAN-XL) instead of pixels directly, evaluated on ImageNet-1K, CIFAR-10/100 and an imitation-learning driving task; the resulting approach placed second and won a best-paper award in the Generative Track of the 2024 Dataset Distillation Challenge.