Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglement

What came before

Naive video distillation directly ports image-domain matching (DC's gradient matching, DM's distribution matching, MTT's trajectory matching, FRePo's kernel closed-form regression) to video by treating every synthetic clip as a small stack of frames matched against a bipartite set of real frames, with no special handling of the time axis.

The problem

The paper's own pre-analysis (Sec. 3) shows this naive approach wastes most of its budget: increasing the number of real/synthetic frames matched, or the number of temporal segments, gives at most a few points of accuracy while multiplying compute and memory (16x GPU memory going from 1 to 16 frames), and distilling to a single still frame per class already recovers most of the achievable accuracy — meaning the temporal dimension of naively distilled synthetic clips contributes little of what is learned.

The idea

Disentangle what a synthetic clip has to represent into a static component (appearance, well suited to image-style distillation) and a dynamic component (motion, cheap to represent compactly), distill each separately, and recombine them with a small learned integrator network rather than distilling every frame jointly.

How it works

Stage 1 (static learning): treat one randomly sampled frame per video, per epoch, as an image dataset and run DC-style gradient matching on a ConvNet to learn a static memory $\mathcal{S}$ (one or more frames per class); this reduces the video problem to $N_{syn}{=}N_{real}{=}K{=}1$ in the paper's "segmented matching and interpolation" formalism. Stage 2 (dynamic fine-tuning): freeze $\mathcal{S}$, initialize a dynamic memory $\mathcal{D}$ (single-channel frames) and an integrator network $\mathcal{H}$ from noise, and jointly optimize both by applying an existing distillation matching loss $\mathcal{A}$ (DM, MTT, or FRePo, used as plugins) between $\mathcal{H}(\mathcal{D},\mathcal{S})$ (reconstructed synthetic clips) and multi-frame samples of the real videos. Stored artifact = static memory + dynamic memory + $\mathcal{H}$'s weights, kept under the same total-tensor storage budget as the naive baseline (82% of baseline storage for DM/MTT, 42% for FRePo).

Evidence

Table 2: MiniUCF (50-class UCF101 subset) IPC=1, ConvNet3D eval, hard labels — FRePo+Ours 22.0% vs. FRePo alone 20.3% (full data 57.2%) using ≤42% of FRePo's storage; MTT+Ours 23.3% vs. MTT alone 19.0% at 82% storage. HMDB51 IPC=1: FRePo+Ours 8.6% vs. FRePo 7.2% (full data 28.6%). Table 3 (top-5, large-scale): Kinetics-400 IPC=5, MTT+Ours 11.5% vs. MTT 9.1% (full 34.6%); SSv2 IPC=1, MTT+Ours 5.5% vs. MTT 3.9% (full 29.0%). Table 4 (cross-architecture, MiniUCF IPC=1): MTT+Ours transfers much better to unseen CNN+GRU (14.8% vs. 8.4%) and CNN+LSTM (13.4% vs. 7.3%) architectures than naive MTT. Key ablation (Table 1, Sec. 3.3): distilling to a single still frame per class already reaches >17% accuracy on MiniUCF with DM, and increasing real/synthetic frame counts from 1 to 8 gains under 3 points while using up to 16x more GPU memory, directly motivating the disentangled design; a second ablation (Fig. 8, Table 8) shows the disentangled method specifically improves accuracy on the "dynamic" (high inter-frame-change) half of MiniUCF classes relative to a "Static-DC" (frame duplication only) baseline, isolating that the dynamic memory is capturing real motion information.

Limitations

Gains shrink on the largest, most class-rich dataset (Kinetics-400, 400 classes) because the integrator network $\mathcal{H}$ is shared across all classes and the paper explicitly declines to scale up per-class capacity due to training cost; only short, low-resolution clips are tested (16 frames at 112x112 for small datasets, 8 frames at 64x64 for Kinetics/SSv2); evaluation is restricted to action recognition with fairly small backbones (MiniC3D, CNN+GRU/LSTM), not modern video transformers; FRePo's own results are flagged by the authors as "reference only" because FRePo does its own label learning and uses a different optimizer, complicating direct comparison.