Fetch and Forge: Efficient Dataset Condensation for Object Detection

What came before

Classification-focused dataset condensation splits into meta-learning (bi-level optimization, e.g. DD, KIP, RFAD, FRePo) and data-matching (gradient matching DC/DSA, distribution matching DM/CAFE/DataDAM/ IDM, trajectory matching MTT) frameworks, all of which involve a bi-level or matching loop tied to a single object/label per image; decoupled-synthesis methods for classification (SRe2L) instead train one model once on real data and invert it to synthesize images via BN-statistics matching, and DiM compresses a dataset into a generator's parameters rather than the images themselves.

The problem

Object detection has two properties that break both frameworks: (i) it is inherently multitask (simultaneous localization and classification of possibly many objects per image), which the bi-level/ matching machinery was not designed to handle, and (ii) detection datasets are large-scale and high-resolution, which makes the existing bi-level optimization loop's cost (already limited to small datasets like CIFAR and simple backbones like ConvNet-3/AlexNet/ResNet-18 in classification) impractical to extend to complex detectors and high-resolution multi-object images.

The idea

Decouple detection dataset condensation into two separate, non-bi-level stages inspired by SRe2L-style model inversion: first train a real detector once on the full dataset (Fetch — key localization/ classification information is stored implicitly in the detector's weights), then freeze that detector and synthesize images by inverting it (Forge), using two detection-specific mechanisms — Foreground- Background Decoupling and Incremental PatchExpand — to handle the fact that a detection image, unlike a classification image, contains multiple objects at different positions, sizes, and categories.

How it works

Stage I (Fetch): train a detector $\psi_\theta$ (YOLOv3-SPP) on the real dataset $\mathcal{T}$ with the standard composite detection loss $\mathcal{L}_{det}=\mathcal{L}_{loc}+\mathcal{L}_{cls}$ (Eq. 3-4); this is a single standard training run, not a bi-level loop. Stage II (Forge): freeze the trained detector; initialize synthetic images by random sampling from the real dataset with their real bounding-box/class targets; apply Foreground-Background Decoupling ($\mathcal{F}_{BD}$) — a binary mask from the box coordinates separates foreground/background, background pixels are suppressed by a factor $\alpha<1$ (limiting how much the background updates, to preserve context and avoid blending unrelated scene semantics) while foreground pixels are enhanced via random erasure to force the model to keep refining them; apply Incremental PatchExpand ($\mathcal{I}_{PE}$) — pad each image to a uniform size, divide it into a $k\times k$ grid of patches, and optimize each patch toward a different target label, with the number of active patches increased incrementally (curriculum-style) during optimization to add diversity of object position/size/shape. Optimize the synthetic images $\widetilde{X}$ to minimize $L_{det}(\psi_{\theta_\mathcal{T}}(\widetilde{X}),\widetilde{A}) + R_{reg}$, where $R_{reg}=R_{pixel}+ R_{feature}$: $R_{pixel}$ is a total-variation + $\ell_2$ term (Eq. 9-10), $R_{feature}$ matches per-layer batch-norm mean/variance statistics of the synthetic images to the frozen detector's stored BN statistics (Eq. 11, an SRe2L-style term). No labels are learned; bounding boxes and classes are fixed from real annotations throughout.

Evidence

Table 1 (Pascal VOC, YOLOv3-SPP condensation+evaluation network, hard labels, compression ratios 0.5%/1%/2%): DCOD reaches mAP 23.7%, AP50 50.7% at ratio 2% (full dataset 46.5% mAP, 76.4% AP50), vs. best coreset baseline (Herding) 8.5% mAP/28.1% AP50 at the same ratio — roughly 3x the coreset mAP. Table 2 (MS COCO, same setup, ratios 0.25%/0.5%/1%): at 1%, DCOD reaches 12.1% mAP, 24.7% AP50 (full 36.1% mAP, 63.6% AP50) vs. best coreset baseline (Herding) 4.1% mAP/12.5% AP50. Table 3 (VOC, 1% ratio, ablation): removing both $\mathcal{F}_{BD}$ and $\mathcal{I}_{PE}$ (baseline = plain model-inversion Forge) gives only 8.3% mAP/25.9% AP50; adding $\mathcal{I}_{PE}$ alone raises this to 17.7% mAP; adding $\mathcal{F}_{BD}$ alone gives 10.0% mAP; combined they reach 19.8% mAP/46.4% AP50 — an 11.5-point mAP improvement over the inversion-only baseline, and the paper notes that without these two components "the detection loss quickly converges in early iterations, missing key positional and classification details." Table 4 (cross-architecture, YOLOv3-SPP-condensed data evaluated with Faster R-CNN, a two-stage detector): performance drops substantially at low compression (VOC 0.5%: 14.2% mAP same-arch vs. 6.3% cross-arch), narrowing but not closing at 1% (19.8% vs. 13.8% mAP). Table 5 (initialization ablation, VOC 0.5%): Random initialization (14.2% mAP) beats K-center (11.9%) and Herding (11.5%), the opposite finding from image classification, attributed to core-set methods not accounting for detection-specific class/object-size distribution. Figure 7 shows DCOD's advantage over random selection shrinks and the two curves converge once the compression ratio exceeds ~20%.

Limitations

Explicitly not extended to more complex/transformer-based detectors like DETR, which the authors state "might require more specialized designs to accommodate their structures"; cross-architecture transfer (YOLO-condensed data evaluated with Faster R-CNN) remains substantially weaker than same-architecture evaluation, especially at low compression ratios, and the paper states this "remains an area that needs improvement"; all experiments use a single detector family (YOLOv3-SPP) for condensation; labels/boxes are fixed from real annotations rather than learned or relabeled, so the method inherits whatever localization noise exists in the original dataset; only a single GPU (V100) wall-clock/memory budget is implied by the experimental setup but not explicitly quantified as a cost table.