Video-Coder: an agent blocks the scene in Blender so the video keeps its physics

In 3D work you block first: grey stand-ins, who stands where and where the camera goes, and only then do light and texture arrive. Video-Coder automates that first step. An agent writes Blender code, renders a draft clip, and a video generator paints the final frames on top of it. Here is what a team from the University of Central Florida, Microsoft Research and four other universities built, where the numbers stay flat, and why you still cannot order it as a tool.

Grey clay primitives on a 3D blockout stage become a painted frame: code sets the motion, the generator paints it
01

What Video-Coder actually does

The paper is called Video-Coder: Reinforcing Agentic Coding for Video Generation. It landed on 9 November 2026 as arXiv 2611.09894, with a page on Hugging Face Papers and coverage in AI Weekly. The author list spans University of Central Florida, University of Würzburg, Microsoft Research, Tsinghua, UNC-Chapel Hill and UW-Madison. Data, models and code are open, sitting in the GitHub repo CYWang735/Video-Coder and on Hugging Face.

The pipeline has three steps. The agent writes and runs Python that drives Blender. That code builds a dynamic 3D scene out of grey clay objects, no textures, no materials. The scene renders into a draft clip. Then a video generator takes over, MiniMax H3 in this work, and receives the draft plus the prompt, and paints the final image.

The draft owns structure and motion. The generator owns light, colour and surface. The split of responsibility is the whole idea.

02

Why pixel models lose count

The cause is that these generators learn from pixels and build no explicit 3D model of the world. In a scene with several objects the count drifts: three balls become two, then four by the end. Positions slide, occlusions break, objects warp, some vanish mid-move.

Code offers no such freedom. Three balls falling at once from different heights is three coordinates and one shared start time. While the script runs, the physics stay identical on the first frame and on the last.

The authors' conclusion follows: move part of the reasoning out of pixels and into a layer where numbers can be checked.

Pipeline diagram: code builds a grey scene, the draft fixes the motion, the video generator paints the final frame
03

How the agent was trained

The base model is Qwen3.5-9B. Supervised fine-tuning on Video-Coder-SFT-6k comes first, then agentic reinforcement learning on Video-Coder-RL-2k. The Setup section of the same paper lists different counts, 10k and 4k, so "several thousand examples" is the safe phrasing.

The reward is split in two. One part pays for code that executes and produces a scene. The other pays for how the resulting video looks. The weight on the first part decays over training, since the agent already writes working scripts late in the run and the interesting signal moves to the image.

Compute: 8 H100 GPUs for the agent and another 16 H100s behind MiniMax H3, about three days.

04

The numbers on PhyGenBench and VBench-2.0

On PhyGenBench the average score for MiniMax H3 rises from 0.61 to 0.67 when paired with Video-Coder. Inside that: mechanics 0.68, optics 0.79. On VBench-2.0 the physical measurements move from 54.58% to 78.19%, with mechanics at 88.67%.

Two caveats about the method. PhyGenBench is judged by GPT-4o, so the score comes from a language model and its notion of a plausible image. And every figure is the authors' own measurement, with no independent rerun.

For anyone comparing video models, that score works as a signal about physics and as a reason to look at specific shots. Treating it as a verdict is premature.

05

Where the draft loses

On materials the picture reverses. In the Material category Video-Coder scores 75.02% against 83.19% for CogVideoX-1.5, so the older model wins. The explanation is simple. A grey draft carries shape and motion, and no surface information at all. Skin, metal, water and fabric stay with the generator, which sees a mannequin and a text prompt.

The limits of the method show up right there. The more the surface matters, the less the draft contributes, because it says nothing about reflections in a shop window or the softness of cloth.

06

The blocking step, written by an agent

I find the approach familiar, because it copies the order of manual work. Block first: who stands where, where the camera travels, what collides with what. Then light, texture, detail. In a studio an animator does the blocking. Here an agent writes it, and that is an honest attempt at the dullest part of the job.

The authors mark their own boundaries. Faces and expressions are deliberately out of scope: a clay stand-in says nothing about a face, and on a person the draft is useless. There is no product either, just a repo and weights. So the careful version reads: the direction works, the customer-facing tool does not exist yet.

07

What this changes when you order video

For production it works as a way to choose an approach. When several objects in frame have to interact, motion gets set before generation: a storyboard, a 3D preview, or code. That is how we run AI video production, with scene structure held on the storyboard so the generator does not decide how many objects survived the shot. Where surface and performance carry the piece, the other route applies: handcrafted videos from $190, with a person building every shot.

The difference between routes shows in neighbouring work. Our piece on animation with code for Shorts works the other way around, since there the code produces the final graphics, while here the code stays a draft for a generator. For a broader map, our guide to AI for video animation covers how models behave on movement and where they typically break.

08

My conclusion

Betting on code as a planning layer is right, and the physics numbers back it. There is no breakthrough in surface quality, and there could not be: a clay draft does not carry any.

The practical takeaway is short. If objects in a shot have to interact, set the motion before generation. Hoping the model holds three objects and their trajectories costs more in rework. Once this layer reaches products, the first use case will be product ads with mechanical action: assembly, a fall, liquid pouring.

frequently asked

01What is Video-Coder
A method where a language agent writes Blender code, builds a grey 3D scene and renders a draft clip. A video generator turns that draft into the final frames.
02Why do video generators break physics
They learn from pixels and build no explicit 3D model of the scene. In a scene with several objects the object count drifts, positions and trajectories slide, objects warp or disappear.
03Can I use Video-Coder today
The code and weights are open, but there is a repository and no service around it. Running it takes a GPU, Blender and a generator that accepts a draft input.
04Does it work for videos with people
Faces and expressions are deliberately out of scope. A grey draft gives no useful anchors for a person, so the method does not fit those shots yet.