V-JEPA 2-AC

World models · 2025

The idea

You have a frozen video encoder that understands motion. How do you turn it into something that can act?

V-JEPA 2-AC is the answer in Assran et al. 2025. Freeze V-JEPA 2. Train a new ~300M-parameter block-causal transformer that autoregressively predicts the representation of the next frame, conditioned on end-effector actions and proprioception. Training data: about 62 hours of unlabeled robot video from DROID (Franka Panda teleops) - no task labels, no success flags, just video plus end-effector state.

At test time the loop is model-predictive control. Given the current camera frame and a goal image, search for an action sequence that minimizes the L1 distance between the imagined future latent and the goal’s latent (cross-entropy method is the usual optimizer). Execute the first action, replan. Deployed zero-shot on arms in labs the model never saw, the paper reports successful prehensile manipulation from a monocular RGB camera.

That is a world model in the strict sense: (latent, action) → next latent, used to plan before acting.

The misconception to name

It is tempting to credit the million-hour video pretrain with the robot result. The pretrain supplies the encoder; the 62 hours of interaction data supply the dynamics that planning needs. Without the action-conditioned stage, you have a probeable video representation - see LeVJEPA for a later encoder that deliberately stops there.

Verdict

What it fixedWith ~62 hours of unlabeled robot video, an action-conditioned predictor on a frozen encoder plans pick-and-place zero-shot in a new lab via model-predictive control.

What it still costsShort-horizon tabletop manipulation, not open-world autonomy. The planner is a separate network fitted after pretraining - not the video encoder itself.

The story, animated

A tiny new predictor on a frozen encoder - then search, score, act, and replan toward a goal.

Where it sits

Worth having first: V-JEPA 2

Unlocks: LeWorldModel

Source-checked against Assran et al., V-JEPA 2 §3-4 (arXiv:2506.09985).