Latent space in plain words
A finished clip looks like an image: millions of colored points. The model sees something else. Before generation the footage passes through a separate network, an autoencoder, and that network writes it down in a compact form. This record is called the latent. The whole generation process runs inside it, and only at the very end a decoder expands the latent back into frames.
The closest everyday analogy is a sketch instead of a painting: the structure and the composition survive, the fine texture is simplified. It matters that this compression is lossy. An archive unpacks without losses, a latent always comes back approximate, and the gap between the approximation and the original is what people usually call generation artifacts.
How the model actually builds the video
To see how the model actually builds the video, follow three entities. The encoder compresses a reference or a first frame into the latent. A diffusion model turns noise inside that record into meaningful motion, step by step. The decoder assembles the finished frames from the latent, and those frames are what you download. Two networks on the sides, one large model in the middle.
Almost every modern video model works like this. A good example is the open Wan family. Wan2.1 ships with Wan-VAE, which the authors describe as a 3D causal VAE: compression runs across the frame area and across time at once, neighbouring frames get merged into blocks. In Wan2.2, the smaller TI2V-5B model carries a VAE with a 16×16×4 compression ratio, sixteen times fewer points on each side of the frame and four times fewer frames per block, which is exactly why it can run 720p on a single consumer GPU. Compression comes first: without it the math fits neither in memory nor in a project budget.

What compression throws away
The general principle: the encoder drops what it considers secondary, the decoder rebuilds the frame from the remainder. Small text and logos suffer first. A letter occupies a few dozen pixels, after compression fewer survive, and on expansion the caption drifts, letters fuse and change shape from frame to frame.
Fine details come next: hair, object edges, water spray. The encoder simplifies that texture and surfaces turn suspiciously smooth. Fast motion gets hit separately: when several frames are merged into one block, the movement inside the block is written as a single step, so the fast phase smears or jerks.
To be fair, the exact list of losses is specific to each model and its encoder, and nobody publishes percentages. The direction is common to everyone: the smaller the detail and the faster the motion, the smaller its chance of surviving to the decoder. Half of the complaints about faces and hands drifting in AI video trace back to the same place: a face is exactly the kind of fine detail compression simplifies. For a talking avatar this is the main part of the deal: the face has to stay identical from the first frame to the last.
Why 4K costs more and why upscaling is a separate step
The price grows non-linearly. According to the PixVid paper, doubling the resolution quadruples the number of tokens the model assembles the frame from, and attention grows even faster. Four times the computation on every pass, the GPU runs hotter, each second of video costs more. Promising everything in 4K therefore needs qualifiers.
Hence the usual order in production: generation runs at base resolution, then upscaling happens as a separate step, with a dedicated model that rebuilds details on the expanded frames. Two different steps, two different networks, two lines in the quote. For small captions and logos the cheaper version of the same logic is adding them in post over the finished clip.

Why PixVid tries to work without a latent
The November paper Back to Pixels (arXiv 2611.17336) from a team at Roblox, UC Merced, University of Pennsylvania and UT Austin attempts to break this deal. They kept the diffusion transformer and replaced the VAE decoder with a convolutional head that renders pixels directly. The model was finetuned from Wan2.1-14B and reaches quality competitive with the original latent model at roughly 1.3 million training clips.
From the abstract: at the same per-frame token count the resolution is twice as high, and each action gets its response at the next frame, without waiting for a block. This is research with no product behind it: no weights or code are listed on the project page, and promising that mode to clients would be early. We covered PixVid in a separate piece with more numbers and caveats.
My take
The latent is an honest trade. Compression makes training possible at all, and fine details and fast motion pay for it. In practice this means three things. Small on-screen text is added by a person in post, because that is cheaper than gambling on generation. 4K is assembled from generation plus upscaling, because honest 4K generation is disproportionately expensive. And when you commission AI video production for a specific task, the decision happens at the frame review, where you can see what survived compression and what did not.




