How an AI video model picks the next frame

In the browser a video model looks like one button: prompt in, ten seconds out. Inside, different models take steps of different size: the whole clip at once, a block of frames or a single frame. That choice sets the delay on your command, the drift by the end of the scene, and whether a shot can be fixed halfway through.

A strip of frames grouped by brackets, with one single frame stepping out of its group
01

Why the clip looks whole and is cut inside

You type a prompt, you get five seconds of motion. Inside, the model keeps the clip in compressed form and from there it depends on the design: some models compute the whole clip at once, others go in blocks of several frames, others one frame per step. Two things the client notices depend on that choice, the speed of reaction to an action and how far the scene drifts by the tenth second.

The word frame matters here. Streaming models predict the next fragment from the past alone, and the size of that fragment sets the whole mechanism.

02

How an AI video generator works when it paints the whole clip

Frontier models run inside a VAE latent space. An autoencoder is a compressor: it squeezes an image across space, and in time it learned to squeeze across time too. At first there was one latent per frame. Later 3D VAEs started merging several frames into a single latent, and that latent matches no individual frame. Training gets easier that way, since there are fewer tokens.

The bill arrives in three places. Doubling resolution quadruples the token count. Prompt conditions land on a group of frames, so an individual frame gets no correction of its own. And in interactive use every answer waits until its whole multi-frame block is computed. The full pipeline with noise and steps sits in our guide on how an AI video generator works, and here the size of the step is what counts.

Three step sizes: the whole clip as one block, groups of four frames, and single frames
03

How an AI video model builds one frame at a time

A group from Roblox, UC Merced, UPenn and UT Austin published "Back to Pixels: Lifting Latent Video Diffusion into Pixel Space" (arXiv 2611.17336) on 12 November, along with the PixVid model. They fine-tuned it from Wan2.1-14B on about 1.3 million clips so that each frame is born straight in pixels. In that mode the model reacts to an action in the very next frame. A latent model answers only in the next block, which means a delay of several frames.

The latent approach is still cheaper to train. What the paper shows is different: generating frame by frame in pixels, the model reaches twice the resolution at the same per-frame token count, keeps depth-guided edits aligned under fast motion, and answers an action in the next frame. For an ordinary clip from a prompt this barely matters, for interactive work (games, a camera steered in real time) it is the whole point.

04

How real-time AI video models work

inclusionAI Duo v1 (GitHub inclusionAI/Duo, 13 November) generates video and the audio of its answer in chunks on one timeline. Attention masks enforce causality: the model never sees future inputs, it has the past and its own generated frames. A separate Intent model decides when to speak. According to AI Weekly the chunks come out around 0.71 s.

lucidml/gta on Hugging Face (Apache-2.0, 11-12 November) leans toward games. It runs 630 million parameters on a Summaryformer architecture, and frames are generated one by one on a WASD press. That interface feels odd for an editor, yet it shows what real-time generation looks like without a queue of blocks.

05

Memory: why old frames get squeezed

A model that paints frame after frame runs into memory fast. In gta every frame is stored in two forms. A raw latent of 1008 tokens serves the nearest motion, while 96 summary tokens carry the distant context, and older frames are compressed into those summaries. The memory window reaches 24 latent frames, which is 96 ordinary frames, so one latent covers four frames. The attention budget with that compression is 6528 keys, against 26 208 for a full history.

The same logic runs through every streaming model: remember the last seconds in detail, keep a short note of everything before them.

06

Where drift comes from

When a model only ever sees the past and its own earlier frames, errors accumulate. A face slides, a hand changes shape, lettering loses characters. It is the drift people already know from ordinary models, except here it has an architectural cause. Duo fights it with double distillation on its own runs: the model trains on the histories it generated itself, and per the README that is part of the recipe. Whether the trick holds a long scene is hard to tell from the write-up, it needs viewing.

07

The hybrid: compressed motion, pixel detail

In LTX-2.5, judging by the ComfyUI blog, motion, composition and framing are generated in a latent compressed 8x across time, along with a set of high quality key frames. A separate pixel diffusion stage then finishes the video. A diffusion video decoder replaces the usual VAE decoder in this version. The logic is simple: cheap decisions happen at the compressed level, expensive detail lands where the eye can see it.

08

What it means for a brief and for an edit

For a client the difference shows up in two places. First, delay. A streaming block model reacts to a command only with the next block, a frame model in the very next frame. Second, control inside the shot. An ordinary generator computes the whole clip, you cannot interrupt it halfway, and a fix means a new run. Live models let you drive a scene like a game, and every mistake carries further.

For ads with an exact action on the second, AI video production makes sense in several passes with a human on the edit. When a shot rests on a talking AI avatar video, the same logic applies: first check how the model holds a face on a short piece, then roll it out over the whole film. The anchor frame and the first step of a shot have their own breakdown, on how an image becomes a video.

frequently asked

01Can a model redo a frame it has already drawn
In an ordinary generator, no: the clip is computed as a whole, and a fix means a new run. Frame-by-frame and streaming models compute a new frame each time, so the reaction arrives fast, and an error also travels into the history the next frame is built from.
02Why does the answer to an action arrive late
In a streaming latent model one step is a group of frames packed into one latent. Until the block is done there is nowhere to apply a new condition, so the bigger the block, the later the model hears you.
03What is a latent, in plain words
It is a compressed description of a frame or of several frames. The model works with numbers that describe pixels, and a decoder turns those numbers back into an image. Squeezing across time keeps more seconds in memory, and one such latent can cover four frames while matching none of them.
04Is real-time generation any good for advertising
Not yet: Duo v1 and lucidml/gta are research releases. Advertising needs repeatable frames and fixes after client notes, which is the job of ordinary generators and editing.