PRISM: Diversifying Dataset Distillation by Decoupling Architectural Priors

What came before

SRe2L uses one teacher for both the logit-matching loss and the BN-statistics regularizer. G-VBSM and EDC diversify by matching several backbones' BN statistics jointly (intra-distillation re-sampling, intra-class batches) but keep the two objectives coupled to whichever teacher is sampled at each step. D3S rotates an ensemble of same-architecture ResNet-18 teachers; CV-DD votes across teachers to build soft labels. RDED sidesteps inversion with real-crop selection; DELT splits IPC budget into early/late-optimized subtasks within a class.

The problem

Every synthetic image in a class is optimized against the same global BN-statistics target from one teacher (or, in G-VBSM/EDC, the same *pooled* signal from several teachers used jointly), so the resulting images inherit that teacher's specific inductive bias and converge toward homogeneous, overly smooth solutions -- visible as high intra-class cosine similarity (0.86-0.92 for SRe2L/G-VBSM/DELT, Fig. 4) and confirmed qualitatively (Fig. 5: SRe2L images within a class converge to similar colors/textures from the same real-image initialization PRISM also uses).

The idea

Split the two terms of the SRe2L objective -- the logit-matching classification loss and the BN-statistics regularizer -- and supervise them with architecturally *different* teacher models, rather than one model doing both; generalize the BN term to a randomly-sampled subset of several diverse teacher architectures per image, so the images are pulled toward "natural" statistics from multiple distinct architectural priors at once instead of one.

How it works

Follows the SRe2L recovery objective $X_S^*=\arg\min_{X_S}\ell(\varphi_\phi(X_s),Y_s)+\lambda \mathcal{R}_{\text{reg}}$ but decouples $\phi$ (the logit/gradient teacher) from $\theta$ (the BN teacher): dual-teacher decoupling uses two distinct models; the full PRISM generalizes the BN term to $\mathbb{E}_{\mathcal{M}_{\text{sub}}\sim P(\mathbb{M}_{\text{valid}})}[\lambda\sum_{\omega\in \mathcal{M}_{\text{sub}}}\mathcal{R}_{\text{BN}}^\omega(X_s)]$, sampling a random subset (up to $k_{\max}{=}4$, VRAM-limited) of a teacher pool (ResNet-18/34, ShuffleNetV2-0.5, MobileNetV2, EfficientNet-B0, later swapped for AlexNet) per image. A pre-distillation teacher-selection strategy (fixed teacher set per image before optimization begins) is shown to beat intra-distillation resampling (Table 2). Initialization uses one real image per synthetic image (no multi-image stitching as in RDED/EDC, no coreset selection as in DELT); cross-class batches (one IPC index per class per batch, following SRe2L) rather than intra-class batches (G-VBSM/EDC/DELT), which the authors argue keeps synthesis simple and parallelizable across GPUs -- diversity comes from architectural decoupling, not intra-batch diversity terms. Post-recovery: an MSE-based ensemble-relabeling loss with 0.1 ground-truth mixing, batch size 50 for relabeling, and EDC's SSRS cosine LR schedule ($\zeta{=}2.5$).

Evidence

ImageNet-1K, PRISM's own evaluation protocol (Table 1, mean of 3 seeds): ResNet-18, IPC=10 49.4% vs EDC 48.6%, RDED 42.0%, G-VBSM 31.4%, and PRISM's own re-run of SRe2L 21.3% (identical to SRe2L's own-reported value, unlike DELT/CIM/CV-DD's re-runs of the same nominal cell, which range 31.1-43.1%); IPC=50 PRISM 59.0% vs EDC 58.0%; IPC=100 PRISM 60.9%. ResNet-101, IPC=100: PRISM 68.6% vs EDC's IPC=50 value 64.9% (EDC does not report ResNet-101 IPC=100). Under DELT's evaluation protocol (Table 1, marked $\ddagger$): ResNet-50, IPC=100, PRISM 69.4% vs DELT's own 66.1% (at IPC=50); ResNet-101, IPC=100, PRISM 70.4% vs DELT 67.6%. Diversity metric (Fig. 4): mean intra-class cosine similarity for PRISM is below 0.83, versus 0.86-0.92 for SRe2L/G-VBSM/DELT, using a pretrained ResNet-18 feature extractor. Recovery-only, no-KD diversity analysis (Table 2, ImageNet-1K, IPC on the order of the full per-class image count, ResNet-18, VRAM at synthesis batch size 100): SRe2L 17.9% vs a real-data upper bound of 70.0%; PRISM's full multi-teacher, pre-distillation, 4-BN-teacher configuration reaches 39.1% at 32.5 GB VRAM, versus 19.0% for dual-teacher decoupling alone (13.0 GB) and 18.3% for single-teacher intra-distillation alignment -- isolating architectural decoupling itself (not KD, not the LR schedule) as the source of the gain in a setting stripped of post-recovery optimization. No hard-label result and no ImageNet-21K or CIFAR/Tiny-ImageNet result are reported; the authors state this choice is deliberate since PRISM's contribution is only visible at large class count and scale.

Limitations

VRAM scales linearly with the number of simultaneous BN teachers (6.5 GB for 1 teacher to 32.5 GB for 4, Table 2), capping how many architectural priors can be combined per image on a single GPU; no wall-clock synthesis time or GPU-hour comparison to SRe2L/G-VBSM/EDC is reported, only VRAM. The method still depends entirely on BN-statistics inversion and is not extended to normalization- free or LayerNorm/GroupNorm architectures (stated limitation). Cross-architecture generalization to unseen student backbones (ViT, ConvNeXt) is explicitly not tested, on the argument that PRISM's own teacher pool already spans architectures -- an argument, not a measurement. No hard-label (no-relabel) result is reported anywhere, so the accuracy gains cannot be separated from PRISM's own ensemble-relabeling scheme. Restricted to ImageNet-1K; no small-scale (CIFAR/Tiny-ImageNet) or ImageNet-21K results are given.