Synthesizing Informative Training Samples with GAN

What came before

Bilevel meta-learning distillation (DD, GTN) and the then-new single-level matching methods DC and DSA and DM (Zhao & Bilen) all optimize pixels directly; separately, GAN-based data generators (BigGAN) and GAN inversion had been used to make training-sample generators, but only to look realistic, not to be informative.

The problem

A BigGAN trained on CIFAR-10/100 and sampled normally, or inverted to reconstruct real images pixel-for-pixel, produces images that look realistic but train a ResNet18 far worse than real data (77.8% and 82.9% vs. 93.4% on CIFAR-10): realism and informativeness for training are different objectives, and GAN inversion's pixel/feature reconstruction loss does not target the second one.

The idea

Freeze a pretrained GAN and, instead of optimizing pixels or matching a discriminator, optimize the GAN's latent vectors directly with a dataset-condensation matching loss (distribution matching, following DM), so the generator becomes a vehicle for informative rather than realistic samples.

How it works

A BigGAN is pretrained per dataset (CIFAR-10, CIFAR-100) with differentiable-augmentation GAN training and then frozen. One latent vector $\bm{z}_i$ is kept per real training image, initialized by GAN inversion ($\arg\min_z \|\psi(G(z))-\psi(x)\|^2 + \lambda_{pixel}\|G(z)-x\|^2$) so the whole latent set $\mathcal{Z}$ has the same cardinality as $\mathcal{T}$. $\mathcal{Z}$ is then optimized (BigGAN still frozen) to minimize a class-wise distribution-matching loss $\mathcal{L}_{con}$ between real and generated features under randomly sampled embedding networks $\psi_{\vartheta}$ and differentiable Siamese augmentation, plus a per-sample regularizer $R$ that keeps a generated image's features close to its paired real image's features so that independently-trained latent batches stay diverse rather than homogenizing. Because $|\mathcal{Z}|\approx|\mathcal{T}|$, the set is split into batches (e.g. 1250/class for CIFAR-10, 500/class for CIFAR-100) trained independently and then pooled ("latent vector ensemble"). At evaluation, latent vectors are decoded through the frozen BigGAN to produce images with their original hard labels, and a fresh network is trained on these generated images from scratch with SGD.

Evidence

Whole-set learning (CIFAR-10/100, using the full learned latent set, ResNet18, hard labels, Table 1): GAN 77.8%/45.2%, GAN Inversion 82.9%/55.0%, IT-GAN 85.7%/60.1%, real-data upper bound 93.4%/74.1%. Against DM at matched 25%-of-dataset storage budget (Table 2, hard labels): CIFAR-10 ConvNet 82.8% vs. DM 80.8% (upper bound 84.8%); CIFAR-10 ResNet18 85.7% vs. DM 85.1%; CIFAR-100 ConvNet 55.7% vs. DM 50.5% (upper bound 56.2%); CIFAR-100 ResNet18 60.1% vs. DM 56.7%. Cross-architecture (Table 3, latents learned with ConvNet only): 82.8-86.0% on CIFAR-10 and 55.7-60.4% on CIFAR-100 across ConvNet, VGG19, ResNet18, WRN-16-8, MobileNetV2, i.e. no architecture-specific collapse. Key ablation (Table T6, appendix): with latent batches mixed/re-split randomly every iteration, accuracy without the regularizer $R$ ($\lambda=0$) is 69.7% vs. 71.3% with tuned $\lambda$, confirming $R$ is what prevents independently-optimized latent batches from becoming redundant copies of each other. No GPU-hour or wall-clock cost is reported; storage cost is quantified instead (40.8MB total: 16.4MB frozen BigGAN + 24.4MB of 128-d latents, vs. 162MB raw CIFAR).

Limitations

Tested only at CIFAR resolution (32x32) and only at a very large "25% of dataset" budget (roughly 1000+ latents/class), not the low-IPC regime (1/10/50 images per class) that dominates the field's tables, so it is unclear how IT-GAN behaves at small budgets; the generator is a BigGAN trained from scratch on the exact target dataset (CIFAR-10 or CIFAR-100), so none of the reported gain is attributable to a generator pretrained on other data, and the paper does not test transferring one dataset's GAN to another; evaluation uses only hard labels with no teacher relabeling; the cost of pretraining and storing a BigGAN per dataset, and of pretraining hundreds of ConvNets to serve as embedding functions, is not counted against the reported storage savings.