A Large-Scale Study on Video Action Dataset Condensation

What came before

Wang et al. (video-sdd-2024) gave the first systematic study of video distillation but used only small-scale HMDB51/MiniUCF, segment-based temporal sampling, and did not investigate how evaluation settings (labeling, augmentation, loss) affect reported condensation performance — mirroring the label-regime confound already documented for image condensation (RDED, EDC, DATM, SRe2L-family methods), which this paper imports into the video setting essentially unstudied.

The problem

Because video condensation papers each pick their own temporal processing, labeling scheme and training recipe, it is unclear whether reported gaps between methods reflect the condensation algorithm itself or the evaluation protocol around it, and no work had systematically adapted state-of-the-art image condensation methods (distribution/statistical matching, trajectory matching, realism-based selection) to video or tested them at the scale of Kinetics-400/SSv2.

The idea

Treat video condensation as three separable design axes — temporal processing (sampling + interpolation), the condensation algorithm, and the evaluation protocol (labels, augmentation, loss) — and vary each independently to isolate which one actually drives reported accuracy differences, proposing sliding-window sampling and a unified evaluation protocol as the fixes.

How it works

Adapts three image condensation methods to video by treating clips as an extra time axis: RDED (training-free realism/diversity-scored real-patch selection, extended by concatenating video tubes spatially), EDC (SRe2L-style statistical/BN-moment matching, extended to space-time statistics), and DATM (multi-step trajectory matching with a difficulty-increasing curriculum). Introduces sliding-window sampling (overlapping clips sampled sequentially along time, replacing segment sampling) and compares interpolation schemes (none, duplication, linear, learnable) for aligning condensed clip length to the evaluation model's input length. Systematically ablates labeling regime (hard, single soft label, Multi-SL — multiple augmented views of one clip each with their own teacher logits, as in SRe2L/G-VBSM/EDC), CutMix augmentation, and loss function (KL vs. MSE against ground-truth), then fixes a single evaluation protocol (Multi-SL, no CutMix, MSE-GT loss) for its head-to-head comparisons.

Evidence

Table 8 (HMDB51/UCF101 top-1, SSv2/K400 top-5, soft labels, MiniC3D eval, various IPC): the paper's reimplementations outperform video-sdd-2024's numbers at matched IPC (e.g. HMDB51 IPC=1, DATM 9.0% vs. video-sdd-2024's best prior 8.6%; UCF101 IPC=5, DATM 20.8% vs. video-sdd-2024 not reported at that scale) and reach new state-of-the-art on all four datasets, with RDED best on HMDB51/UCF101 and DATM best on the larger, more class-diverse SSv2/K400 (full-dataset upper bounds: HMDB51 26.1%, UCF101 52.6%, SSv2 47.5%, K400 44.6%). Figure 4 / Tables 2-3 (UCF101, IPC=1): labeling method changes accuracy by up to 8.6 points — more than augmentation (3.3 pts) or loss function (2.4 pts) combined — and can reverse method rankings (DATM without CutMix, 12.1%, beats RDED with CutMix, 11.0%, under soft labels). Table 4 (K400): sliding-window sampling beats segment sampling for all three methods (e.g. RDED 11.5% vs. 10.6%); Table 5 (SSv2): all training-free interpolators (duplication, linear) hurt accuracy relative to no interpolation. Cost: DATM uses up to 40GB GPU memory vs. EDC's under 10GB (Sec. 4.4), with no wall-clock numbers given.

Limitations

Does not propose or test a trainable interpolator despite showing training-free ones fail; explicitly excludes generative/diffusion-based video condensation methods "due to limitation of time and energy"; evaluation still relies on relatively small backbones (MiniC3D primary, R(2+1)D/I3D/SlowOnly for transfer) rather than modern video transformers; DATM's high memory cost (up to 40GB) is flagged as limiting its scalability but not resolved.