Dataset Quantization

What came before

Gradient/trajectory/distribution-matching DD (DC, DSA, IDC, CAFE, DM, MTT) synthesizes images by matching a metric computed through a specific network, which the paper argues biases the result toward that architecture; DM alone needs 28,000 GPU hours to compress 60% of ImageNet-1K, and DD methods saturate below full-dataset accuracy as the keep ratio grows. Prior coreset methods (GraphCut/submodular selection, Craig, GradMatch, Glister, k-Center, Herding) generalize across architectures but, the paper argues, select in a single one-shot pass that is biased toward high-density regions, losing diversity especially at low keep ratios (DC-2018 itself notes coresets trail DD in the low-data regime).

The problem

Neither existing family is deployable at ImageNet scale with lossless compression: DD is architecture-biased and too expensive to run once, let alone re-run per keep ratio; one-shot coreset selection is cheap and architecture-agnostic but its selected samples cluster in high-density regions of feature space, losing diversity and underperforming at low keep ratios.

The idea

Diagnose *why* one-shot submodular coreset selection loses diversity (a closed-form radius bound showing later selections are dominated by "distance to the unselected remainder," not diversity, when the keep ratio is low) and fix it by selecting recursively into a sequence of non-overlapping bins instead of one coreset, then sampling uniformly across all bins so the final set spans the full range of representativeness-vs-diversity trade-offs the bins cover; drop the least informative image patches to cut storage further and reconstruct them at train time with a pretrained MAE.

How it works

Given a submodular gain $P(x_k)=\sum_{p\in S_1^{k-1}}\|f(p){-}f(x_k)\|_2^2 - \sum_{p\in D\setminus S_1^{k-1}}\|f(p){-}f(x_k)\|_2^2$ (GraphCut-style, using a feature extractor $f$ from a classifier pretrained 10 epochs on the full data), the paper proves the next selected sample's feature norm is bounded by $(\tfrac{2k}{M-2k})^2(R_1^{k-1})^2$, showing diversity is unguaranteed when $M\gg k$ (low keep ratio) because the "distance to the remaining data" term dominates; a second selection round on the remainder has a smaller denominator ($M-K-2k$) and provably larger diversity bound. Dataset Quantization (DQ) exploits this by recursively carving the dataset into $N$ non-overlapping bins (default $N{=}10$) via the same submodular-gain rule applied to successive remainders, then uniformly samples a fraction $\rho$ (the data keep ratio) from every bin so the final set $S^*$ spans bins of differing representativeness/diversity trade-off rather than only the most "representative" region. Independently, each kept image is divided into patches, a patch-importance score (GradCAM-based) is used to drop the least informative fraction $\theta$ (default 25%), and at training time the dropped patches are reconstructed by a pretrained MAE decoder. In the loop: one pretrained feature-extractor classifier (ResNet-18/ViT-Base) for bin construction and one pretrained MAE for patch reconstruction at train time; no bi-level optimization, no per-architecture matching metric during selection.

Evidence

ImageNet-1K, hard labels, lossless-compression headline (Fig. 5b, Sec. 4.3): 60% keep ratio trains ResNet-18 to within the full-dataset accuracy band ("lossless"); the same 60% pretrained ResNet-50 weights transfer to COCO detection at 39.0% vs. 39.2% mAP full-data and ADE20K segmentation at 42.3% vs. 42.5% mIoU full-data (Sec. 1, Fig. 6). Cross-architecture generalization, CIFAR-10, same ResNet-18-sourced compressed set, five student architectures (Table 2): at $\rho{=}10\%$, DM (distribution matching) averages 39.5% across R18/R50/ViT/Swin/ConvNeXt (dropping to 21.6% on ViT, a 59.4-point loss relative to its own 81.2% on R18-not-shown-here-cf-Fig2c) while DQ averages 69.4% (+29.9 over DM) at the same $\rho$, with the DQ-DM gap narrowing but persisting up to 30% ($+35.5$ at $\rho{=}30\%$ is the largest gap reported). Compute cost (Table 4, ImageNet 60% keep ratio): DM needs 91 GPU-hours total across its per-ratio re-runs (7/14/29/41 hours at 10/20/40/60% plus implied earlier ratios) vs. DQ's 1 GPU-hour (bin creation only; sampling thereafter is free per ratio) — the paper's headline 388x figure (28,000 vs. 72 GPU-hours) is stated in the abstract/ introduction for a different, larger prior DM run. Scalability sweep (Fig. 4a-b, CIFAR-10, ResNet-18): at very low keep ratio (1%) GraphCut coreset selection is worst of the three; at high keep ratio DM saturates and ends about 5 points below random sampling; DQ is highest across the full range in both regimes. Language task (Alpaca instruction tuning, LLaMA-7B, Table 5): 20% of instructions selected by DQ match 100%-data performance on the BBH/DROP/MMLU/HumanEval average (27.1% vs. 27.7%); the abstract's stronger claim of comparable performance from 2% of the data applies specifically to the BBH and DROP subsets, not the full four-benchmark average.

Limitations

States its own limitation directly: bins are still selected recursively from the whole dataset, which is extra computational effort relative to a true one-shot method, and lists designing a single-pass DQ as future work, alongside extending to video and generative (AIGC) tasks. Observed: dataset-distillation baselines (DM, DC, gradient/trajectory matching) are compared only on CIFAR-10 and not run on ImageNet-1K "due to the extremely large computational cost" — so the headline ImageNet lossless-compression claim is never benchmarked against any DD method at that scale, only against coreset selection and random sampling. The recursive bin construction and MAE patch-reconstruction depend on a classifier pretrained on the full dataset (for features) and a pretrained MAE (for reconstruction) respectively, both themselves full-data-trained priors whose cost is not included in the reported "1 GPU-hour" bin-creation figure. Ablations (bin number $N$, patch-drop ratio $\theta$) are run only on CIFAR-10/ResNet-18, not re-verified at ImageNet-1K scale.