emerging
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
· Meta AI · 2025-06 · arXiv:2506.09985
Why it matters
Carries the joint-embedding predictive idea to video and shows the resulting model supports action-conditioned prediction and robot planning. Included as a pointer: video is out of this atlas's scope, but the representation family is the same one generative image models draw on.
What this paper does
read: full textBefore this
Predictive world models had mostly been trained from real-world interaction data such as state-action sequences, which is scarce and constrains scale, or from video generation models optimized for visual fidelity rather than for planning ability. The original V-JEPA used a smaller 300M-parameter encoder on 2 million videos and stayed in the understanding regime without an action-conditioned planning component.
The problem
The paper targets understanding, prediction and planning learned largely by observation, combining internet-scale video with only a small amount of robot interaction data and no explicit rewards or task-specific training.
The idea
Pretrain a joint-embedding predictive architecture that predicts a masked video's learned representation rather than its pixels, then post-train a lightweight action-conditioned predictor on a small amount of robot video, so the same latent world model supports both understanding benchmarks and zero-shot planning.
How it works
The supervision is self-distillation in representation space, JEPA-style masked latent prediction rather than pixel reconstruction. An EMA teacher encoder reads an unmasked view, an online encoder reads a masked view, and a predictor is trained with an L1 loss to match the stop-gradiented teacher representation. The encoder scales from ViT-L at 300M to ViT-g at 1B parameters, trained on over 1 million hours of internet video plus images from a 22M-clip VideoMix22M dataset, with progressive resolution up to 64 frames at 384px. After pretraining the encoder is frozen, and a roughly 300M-parameter action-conditioned autoregressive predictor, V-JEPA 2-AC, is trained on only 62 hours of unlabeled Droid robot video to predict future latent states conditioned on actions, enabling zero-shot planning. The paper explicitly compares this against a video generation baseline built on Cosmos, a latent-diffusion 7B model with a continuous tokenizer, fine-tuned the same way on Droid data, and argues JEPA is preferable because it predicts only the predictable structure of a scene while generative objectives spend capacity modeling unpredictable pixel detail. So the encoder itself is the world model here, generation is discussed only as a rejected alternative baseline, and the paper does not propose its own features as input to a downstream generative image model.
Evidence
V-JEPA 2 reaches 77.3 percent top-1 accuracy on Something-Something v2 with the ViT-g384 model on 64-frame clips at 384px, against InternVideo2's 67.7 percent and VideoPrism's 68.5 percent. It reaches 39.7 recall-at-5 on Epic-Kitchens-100 action anticipation against the prior best PlausiVL's 27.6 percent at 8B parameters, a 44 percent relative improvement. After LLM alignment at 8B scale it reaches 84.0 on PerceptionTest and 76.9 on TempCompass. V-JEPA 2-AC achieves 65 percent average grasp success and 72.5 percent average pick-and-place success on real Franka arms in two labs with zero robot-specific data collection, planning in 16 seconds per action against the Cosmos baseline's 4 minutes and lower success rates of 0 to 60 percent.
Limitations
The paper concedes the model must implicitly infer the action coordinate axis from monocular RGB input alone, which causes errors, that autoregressive rollout error accumulates so long-horizon planning reliability drops, and that the current formulation assumes visual goals are available even though language-specified goals may be more natural for in-the-wild robot deployment.
Why it matters
Video and robotics sit outside this atlas's scope, so this entry is a pointer node, but it belongs to the same self-distillation representation family that image-side generative-latent work draws on, and it is the clearest existing empirical argument against generative pretraining, showing an encoder trained to predict only predictable latent structure can beat one trained to reconstruct or generate pixels when the downstream task is planning.
Abstract, in the authors' own words
A major challenge for modern AI is to learn to understand the world and learn to act largely by observation. This paper explores a self-supervised approach that combines internet-scale video data with a small amount of interaction data (robot trajectories), to develop models capable of understanding, predicting, and planning in the physical world. We first pre-train an action-free joint-embedding-predictive architecture, V-JEPA 2, on a video and image dataset comprising over 1 million hours of internet video. V-JEPA 2 achieves strong performance on motion understanding (77.3 top-1 accuracy on Something-Something v2) and state-of-the-art performance on human action anticipation (39.7 recall-at-5 on Epic-Kitchens-100) surpassing previous task-specific models. Additionally, after aligning V-JEPA 2 with a large language model, we demonstrate state-of-the-art performance on multiple video question-answering tasks at the 8 billion parameter scale (e.g., 84.0 on PerceptionTest, 76.9 on TempCompass). Finally, we show how self-supervised learning can be applied to robotic planning tasks by post-training a latent action-conditioned world model, V-JEPA 2-AC, using less than 62 hours of unlabeled robot videos from the Droid dataset. We deploy V-JEPA 2-AC zero-shot on Franka arms in two different labs and enable picking and placing of objects using planning with image goals. Notably, this is achieved without collecting any data from the robots in these environments, and without any task-specific training or reward. This work demonstrates how self-supervised learning from web-scale data and a small amount of robot interaction data can yield a world model capable of planning in the physical world.
Research line
Builds on
Nothing recorded yet.
Built on by
Nothing recorded yet.
This is a emerging entry — a short-form summary. Full paper-page explanations (before/problem/ core idea/representation/architecture/objective/conditioning/training/inference/results/ ablations/limitations) are written for landmark and core papers first; see PROJECT_STATE.md for the schedule.