Robots learn from AI video: Praxis-1 and PAN 2 open up

Two open releases in two days, both about the same thing: a video model learning to predict action along with the picture. Here is what Runway and MBZUAI actually published, what the percentages show, and why anyone ordering ad spots should care.

A robot arm reaches for a glass while its trail stretches as a strip of video frames predicting motion
01

What landed on 10 and 11 November

On 11 November 2026 Runway pushed Praxis-1 out to everyone. Apache 2.0 weights, commercial use included, plus inference code and fine-tuning code, both full and LoRA. A day earlier the Institute of Foundation Models at MBZUAI announced PAN 2, also open: weights on Hugging Face, code on GitHub, Apache 2.0.

Two releases in one week, both pointed at action. The thing we used to call a video generator now answers a different question, what happens in the frame half a second after the button gets pressed.

02

Robots and AI video: the Praxis-1 numbers

Runway introduced Praxis-1 in September as its first world action model. Inside there are 27 billion parameters, a mixture-of-experts diffusion transformer predicting video and actions together, built on the same large scale video pretraining as Runway's world models.

The numbers in the post run like this. On NVIDIA's RoboLab-120 leaderboard, 1200 episodes: Praxis-1 at 46.2%, FLUX 3 Action at 42.9%, Cosmos3 Nano at 36.8%, π0.5 at 28.0%. The harder test sits in the next line. Fine-tuned on Google's RT-1 robot, which was absent from training, Praxis-1 scored 80.0% on SimplerEnv against 52.4% for the original RT-1 model.

I keep one caveat in mind. Both the leaderboard run and the percentages come from Runway's own publication. A four point lead over FLUX 3 Action looks plausible, still it deserves a rerun on your own hardware, which the open weights make possible from H100 and B200 down to RTX 5090 and Jetson Thor.

Diagram: teleoperation and web video feed one model that outputs a predicted robot action
03

Why ordinary web video was enough

The most interesting paragraph in the post is not about percentages. Runway states it plainly: web video as pretraining data is no worse than teleoperated robot footage. Internet clips can replace a good part of the hours at the control rig. The bottleneck moves, and the limit becomes how much ordinary video a model can absorb.

Teleoperation has always been the expensive link in robot training: an operator, a rig, repeatable motion. If web video closes that gap, robot training gets cheaper. Runway names Noble Machines, Standard Bots and Ultra as launch partners, with hosting on Reactor for teams without their own hardware.

04

PAN 2: a model that plays the scene forward

PAN 2 works differently. A language model plans, a video diffusion transformer with 29 billion parameters renders. The input is text describing a scene and an action, the output is video of the likely outcome, which the authors call simulative reasoning. Defaults: 704 by 1248 pixels, 24 frames per second, clips up to 20 seconds, 50 sampling steps, CFG 2.5. Diffusers and vLLM are supported, and the model does both text-to-video and image-to-video.

The model card also lists a separate guardrail filter. That makes sense for a model that draws the consequences of an action, including what happens when someone pushes a glass off a table. My facts here come from Writeble's 11 November writeup based on the model card, so exact numbers are worth checking on Hugging Face.

Unlabelled comparison chart: four descending bars with the tallest highlighted, next to a robot gripper
05

What this changes for ad video

The practical part is short. Money now flows into video models along two routes: clips and robots. Motion physics will improve faster than the ad market alone would ever pay for, and our side gets the spillover from someone else's race.

Second, the weights are open and commercial use is allowed, so a team doing AI video production can study these models with no contract and no API bill. They were built for robotics, and they show where physics in video models is heading.

Third, the less obvious one. A model that predicts action may eventually help in the edit too: if it knows what a person does in the next frame, cuts on movement get more predictable. For now that is my hypothesis. To compare with where this sat a year ago, our earlier piece on world models with generated audio is the reference point.

06

My conclusion

Two open releases in two days read as one line continuing. Video generation is drifting away from being a standalone entertainment business: the same pretraining now serves a manipulator arm.

My bet: by 2027 the word "videomodel" in job posts will sit next to "manipulation" more often than next to "spot". Personal position, and the numbers carry it: 46.2% on RoboLab-120 and 80.0% after fine-tuning on an unfamiliar robot stopped being a demo a while ago.

For anyone ordering ads the takeaway is duller and more useful. A model that understands physics slips less on movement, so the quality question moves toward believable falling objects and believable cloth. A shorter prompt will not fix that, a test fragment before the main job will surface it.

frequently asked

01What is Praxis-1
Runway's first world action model, opened on 11 November 2026 under Apache 2.0. 27 billion parameters, a mixture-of-experts diffusion transformer predicting video and actions together, trained on the same video pretraining as Runway's world models.
02What is RoboLab-120
An NVIDIA leaderboard with 1200 episodes of robotic tasks. In Runway's publication Praxis-1 scores 46.2% there, FLUX 3 Action 42.9%, Cosmos3 Nano 36.8%, π0.5 28.0%.
03How is PAN 2 different from Praxis-1
PAN 2 from MBZUAI is a world model with 29 billion parameters: a language model plans the action, a video diffusion transformer draws the likely result. Praxis-1 predicts video and actions and fine-tunes to a specific robot.
04Can these models run on my own machine
Yes. The weights are open under Apache 2.0, Praxis-1 runs from H100 and B200 down to RTX 5090 and Jetson Thor, PAN 2 supports Diffusers and vLLM.