strong-followup
Theia: Distilling Diverse Vision Foundation Models for Robot Learning
· Boston Dynamics AI Institute · 2024-07 · arXiv:2407.20179
Why it matters
Applies the same multi-teacher distillation bet in robotics, where a single backbone that inherits several specialists is worth more than any one of them.
What this paper does
read: full textBefore this
Robot vision-based policy learning needs visual representations useful across many downstream tasks at once (recognizing objects, understanding depth, spatial layout), not just one narrow task like classification or segmentation, but individual off-the-shelf vision foundation models (CLIP, DINOv2, SAM, Depth-Anything) were each trained for one such task and none alone covered the full range robot policies need.
The problem
How to produce a single visual backbone for robot learning that captures the combined visual knowledge of several specialist vision foundation models, is small and cheap enough to run on a robot, and actually improves downstream policy-learning performance rather than just matching classification benchmarks.
The idea
Distill several off-the-shelf vision foundation models trained on different tasks into one compact backbone using lightweight per-teacher feature translators during training only, so the resulting encoder inherits the combined visual knowledge needed for robot learning while staying small enough for real-time robot use.
How it works
A ViT backbone (Tiny/Small/Base sizes) is trained from scratch on ImageNet for 50 epochs, with shallow CNN feature translators mapping its spatial token representations into each teacher's feature space; the translators exist only for training and are discarded afterward, so downstream tasks use only the backbone, frozen or fine-tuned depending on task difficulty. Five teachers were tried (CLIP, DINOv2, ViT, SAM, Depth-Anything); ablations found the best combination uses only three (CLIP, DINOv2, ViT, called CDiV). Distillation loss combines cosine similarity and smooth-L1 loss, weighted beta=0.9 toward cosine, and distills spatial tokens only, not CLS tokens. The paper explicitly names RADIO as its closest prior work and distinguishes itself on four points: targeting robot-learning tasks outside any single teacher's original training, distilling spatial tokens only, using a different teacher set, and analyzing each teacher's individual contribution to robot performance.
Evidence
On CortexBench (MuJoCo subset, aggregated score), Theia-B reaches 79.79 +/- 0.14, above VC-1-L-sp (69.56 +/- 0.80), MVP-L-sp (77.42 +/- 3.14), and R3M (76.51 +/- 0.79). On real WidowX-arm tasks, Theia-B gets 92% success on door opening versus E-RADIO's 72%, MVP-L's 32%, VC-1-L's 12%, and R3M's 48%; on pick-and-place, 85% (dropping to 75% in a harder variant) versus E-RADIO's 75%/55%. Theia was trained on only the 1.2M images of ImageNet in about 150 H100 GPU-hours, versus roughly 5M images in prior comparable work and 1B images used to train RADIO, while Theia-B (86M parameters) uses fewer multiply-accumulate operations than VC-1-L-sp (303M) and E-RADIO (390M).
Limitations
The paper concedes it could not evaluate the Move Cube, ObjectNav, and MobilePick CortexBench tasks due to reproducibility issues. On the long-horizon Toy-Microwave Cooking task, a frozen Theia encoder achieved 0% success and required fine-tuning to work at all, showing the representation alone is not sufficient for harder tasks without adaptation. Training only on ImageNet may miss robot-specific visual patterns that video or robot data would provide. On standard ImageNet linear probing, Theia-B reaches only 72.1% versus DINOv2-L's 84.5%, a real trade-off against general vision performance. The feature-norm-entropy hypothesis correlates less strongly for distilled models (R=0.638) than for regular models (R=0.943), which the paper flags as not yet fully understood.
Why it matters
Theia shows the multi-teacher distillation idea transfers to a different domain (robot policy learning) and a different research group (Boston Dynamics AI Institute, not NVIDIA), with its own specific design choices (spatial-token-only distillation, a different teacher set, far less training data). It explicitly engages with RADIO as related prior work rather than presenting itself as derived from it, and it outperforms an actual RADIO variant (E-RADIO) on the real-robot tasks tested, which supports reading multi-teacher distillation as an idea with independent, competing implementations rather than one company's product line, while making clear Theia is a deliberate departure from RADIO's design, not an unaware reinvention of it.
Abstract, in the authors' own words
Vision-based robot policy learning, which maps visual inputs to actions, necessitates a holistic understanding of diverse visual tasks beyond single-task needs like classification or segmentation. Inspired by this, we introduce Theia, a vision foundation model for robot learning that distills multiple off-the-shelf vision foundation models trained on varied vision tasks. Theia's rich visual representations encode diverse visual knowledge, enhancing downstream robot learning. Extensive experiments demonstrate that Theia outperforms its teacher models and prior robot learning models using less training data and smaller model sizes. Additionally, we quantify the quality of pre-trained visual representations and hypothesize that higher entropy in feature norm distributions leads to improved robot learning performance. Code, models, and demo are available at https://theia.theaiinstitute.com.
Research line
Builds on
Nothing recorded yet.
Built on by
Nothing recorded yet.
This is a strong-followup entry — a short-form summary. Full paper-page explanations (before/problem/ core idea/representation/architecture/objective/conditioning/training/inference/results/ ablations/limitations) are written for landmark and core papers first; see PROJECT_STATE.md for the schedule.