What Video-Coder actually does
The paper is called Video-Coder: Reinforcing Agentic Coding for Video Generation. It landed on 9 November 2026 as arXiv 2611.09894, with a page on Hugging Face Papers and coverage in AI Weekly. The author list spans University of Central Florida, University of Würzburg, Microsoft Research, Tsinghua, UNC-Chapel Hill and UW-Madison. Data, models and code are open, sitting in the GitHub repo CYWang735/Video-Coder and on Hugging Face.
The pipeline has three steps. The agent writes and runs Python that drives Blender. That code builds a dynamic 3D scene out of grey clay objects, no textures, no materials. The scene renders into a draft clip. Then a video generator takes over, MiniMax H3 in this work, and receives the draft plus the prompt, and paints the final image.
The draft owns structure and motion. The generator owns light, colour and surface. The split of responsibility is the whole idea.
Why pixel models lose count
The cause is that these generators learn from pixels and build no explicit 3D model of the world. In a scene with several objects the count drifts: three balls become two, then four by the end. Positions slide, occlusions break, objects warp, some vanish mid-move.
Code offers no such freedom. Three balls falling at once from different heights is three coordinates and one shared start time. While the script runs, the physics stay identical on the first frame and on the last.
The authors' conclusion follows: move part of the reasoning out of pixels and into a layer where numbers can be checked.

How the agent was trained
The base model is Qwen3.5-9B. Supervised fine-tuning on Video-Coder-SFT-6k comes first, then agentic reinforcement learning on Video-Coder-RL-2k. The Setup section of the same paper lists different counts, 10k and 4k, so "several thousand examples" is the safe phrasing.
The reward is split in two. One part pays for code that executes and produces a scene. The other pays for how the resulting video looks. The weight on the first part decays over training, since the agent already writes working scripts late in the run and the interesting signal moves to the image.
Compute: 8 H100 GPUs for the agent and another 16 H100s behind MiniMax H3, about three days.
The numbers on PhyGenBench and VBench-2.0
On PhyGenBench the average score for MiniMax H3 rises from 0.61 to 0.67 when paired with Video-Coder. Inside that: mechanics 0.68, optics 0.79. On VBench-2.0 the physical measurements move from 54.58% to 78.19%, with mechanics at 88.67%.
Two caveats about the method. PhyGenBench is judged by GPT-4o, so the score comes from a language model and its notion of a plausible image. And every figure is the authors' own measurement, with no independent rerun.
For anyone comparing video models, that score works as a signal about physics and as a reason to look at specific shots. Treating it as a verdict is premature.
Where the draft loses
On materials the picture reverses. In the Material category Video-Coder scores 75.02% against 83.19% for CogVideoX-1.5, so the older model wins. The explanation is simple. A grey draft carries shape and motion, and no surface information at all. Skin, metal, water and fabric stay with the generator, which sees a mannequin and a text prompt.
The limits of the method show up right there. The more the surface matters, the less the draft contributes, because it says nothing about reflections in a shop window or the softness of cloth.
The blocking step, written by an agent
I find the approach familiar, because it copies the order of manual work. Block first: who stands where, where the camera travels, what collides with what. Then light, texture, detail. In a studio an animator does the blocking. Here an agent writes it, and that is an honest attempt at the dullest part of the job.
The authors mark their own boundaries. Faces and expressions are deliberately out of scope: a clay stand-in says nothing about a face, and on a person the draft is useless. There is no product either, just a repo and weights. So the careful version reads: the direction works, the customer-facing tool does not exist yet.
What this changes when you order video
For production it works as a way to choose an approach. When several objects in frame have to interact, motion gets set before generation: a storyboard, a 3D preview, or code. That is how we run AI video production, with scene structure held on the storyboard so the generator does not decide how many objects survived the shot. Where surface and performance carry the piece, the other route applies: handcrafted videos from $190, with a person building every shot.
The difference between routes shows in neighbouring work. Our piece on animation with code for Shorts works the other way around, since there the code produces the final graphics, while here the code stays a draft for a generator. For a broader map, our guide to AI for video animation covers how models behave on movement and where they typically break.
My conclusion
Betting on code as a planning layer is right, and the physics numbers back it. There is no breakthrough in surface quality, and there could not be: a clay draft does not carry any.
The practical takeaway is short. If objects in a shot have to interact, set the motion before generation. Hoping the model holds three objects and their trajectories costs more in rework. Once this layer reaches products, the first use case will be product ads with mechanical action: assembly, a fall, liquid pouring.




