Why the clip looks whole and is cut inside
You type a prompt, you get five seconds of motion. Inside, the model keeps the clip in compressed form and from there it depends on the design: some models compute the whole clip at once, others go in blocks of several frames, others one frame per step. Two things the client notices depend on that choice, the speed of reaction to an action and how far the scene drifts by the tenth second.
The word frame matters here. Streaming models predict the next fragment from the past alone, and the size of that fragment sets the whole mechanism.
How an AI video generator works when it paints the whole clip
Frontier models run inside a VAE latent space. An autoencoder is a compressor: it squeezes an image across space, and in time it learned to squeeze across time too. At first there was one latent per frame. Later 3D VAEs started merging several frames into a single latent, and that latent matches no individual frame. Training gets easier that way, since there are fewer tokens.
The bill arrives in three places. Doubling resolution quadruples the token count. Prompt conditions land on a group of frames, so an individual frame gets no correction of its own. And in interactive use every answer waits until its whole multi-frame block is computed. The full pipeline with noise and steps sits in our guide on how an AI video generator works, and here the size of the step is what counts.

How an AI video model builds one frame at a time
A group from Roblox, UC Merced, UPenn and UT Austin published "Back to Pixels: Lifting Latent Video Diffusion into Pixel Space" (arXiv 2611.17336) on 12 November, along with the PixVid model. They fine-tuned it from Wan2.1-14B on about 1.3 million clips so that each frame is born straight in pixels. In that mode the model reacts to an action in the very next frame. A latent model answers only in the next block, which means a delay of several frames.
The latent approach is still cheaper to train. What the paper shows is different: generating frame by frame in pixels, the model reaches twice the resolution at the same per-frame token count, keeps depth-guided edits aligned under fast motion, and answers an action in the next frame. For an ordinary clip from a prompt this barely matters, for interactive work (games, a camera steered in real time) it is the whole point.
How real-time AI video models work
inclusionAI Duo v1 (GitHub inclusionAI/Duo, 13 November) generates video and the audio of its answer in chunks on one timeline. Attention masks enforce causality: the model never sees future inputs, it has the past and its own generated frames. A separate Intent model decides when to speak. According to AI Weekly the chunks come out around 0.71 s.
lucidml/gta on Hugging Face (Apache-2.0, 11-12 November) leans toward games. It runs 630 million parameters on a Summaryformer architecture, and frames are generated one by one on a WASD press. That interface feels odd for an editor, yet it shows what real-time generation looks like without a queue of blocks.
Memory: why old frames get squeezed
A model that paints frame after frame runs into memory fast. In gta every frame is stored in two forms. A raw latent of 1008 tokens serves the nearest motion, while 96 summary tokens carry the distant context, and older frames are compressed into those summaries. The memory window reaches 24 latent frames, which is 96 ordinary frames, so one latent covers four frames. The attention budget with that compression is 6528 keys, against 26 208 for a full history.
The same logic runs through every streaming model: remember the last seconds in detail, keep a short note of everything before them.
Where drift comes from
When a model only ever sees the past and its own earlier frames, errors accumulate. A face slides, a hand changes shape, lettering loses characters. It is the drift people already know from ordinary models, except here it has an architectural cause. Duo fights it with double distillation on its own runs: the model trains on the histories it generated itself, and per the README that is part of the recipe. Whether the trick holds a long scene is hard to tell from the write-up, it needs viewing.
The hybrid: compressed motion, pixel detail
In LTX-2.5, judging by the ComfyUI blog, motion, composition and framing are generated in a latent compressed 8x across time, along with a set of high quality key frames. A separate pixel diffusion stage then finishes the video. A diffusion video decoder replaces the usual VAE decoder in this version. The logic is simple: cheap decisions happen at the compressed level, expensive detail lands where the eye can see it.
What it means for a brief and for an edit
For a client the difference shows up in two places. First, delay. A streaming block model reacts to a command only with the next block, a frame model in the very next frame. Second, control inside the shot. An ordinary generator computes the whole clip, you cannot interrupt it halfway, and a fix means a new run. Live models let you drive a scene like a game, and every mistake carries further.
For ads with an exact action on the second, AI video production makes sense in several passes with a human on the edit. When a shot rests on a talking AI avatar video, the same logic applies: first check how the model holds a face on a short piece, then roll it out over the whole film. The anchor frame and the first step of a shot have their own breakdown, on how an image becomes a video.




