PixVid: an AI video model that draws pixels directly

A normal video model does not draw the frame. It works inside a compressed latent and the decoder turns that into an image afterwards. A team from Roblox and three universities removed the decoder and taught the model to work with pixels. Here is what that buys on resolution and latency, and where the story is still only a promise.

Pixel tiles run straight into a grid frame, bypassing an empty compression funnel
01

What the paper claims

The paper is called Back to Pixels: Lifting Latent Video Diffusion into Pixel Space and sits on arXiv as 2611.17336. The authors come from Roblox, UC Merced, the University of Pennsylvania and UT Austin, with Xun Huang and Ming-Hsuan Yang on the list. The short version: they take a pretrained latent video model, keep the diffusion transformer, remove the VAE and let the model predict the pixels of each frame.

For someone who commissions video, that sounds like kitchen detail. In practice it is about the price of high resolution and about how fast a model answers a command. Below is the walkthrough, with no formulas and no promises the paper never made.

02

What a latent space is, in plain words

A 1664x960 frame is roughly 1.6 million points, each with a colour to predict. Computing that whole grid at once is expensive, so a VAE steps in: an encoder squeezes the video into a compact record, the latent, and a decoder unpacks the result back into an image. The model inside works with the compact record.

Compression runs along two axes at once. Across the frame, where several neighbouring pixels become one vector, and across time, where several neighbouring frames are glued into one block. Every trade-off follows from that: fine detail is lost, and motion inside a block is settled in a single step.

If you want the whole chain from latent to finished file, we have a separate explainer on how an AI video generator works.

Two grids with the same cell count, the right one larger: same tokens, bigger frame
03

Why latent compression becomes the ceiling

The authors list three places where the latent starts to hurt. First, doubling resolution quadruples the token count, and attention grows faster still. Second, a condition or a command applies to a group of frames, so a single frame cannot be steered. Third, a response waits until its whole multi-frame block is computed, and that shows up as delay.

The numbers in the work show the scale. A 960p frame in a latent model needs four times the tokens of 480p and sixteen times the attention. Attention scales quadratically with token count, and that is the wall high resolution hits inside a compressed representation. The same arithmetic applies to vertical video resolution, the frame is simply turned on its side.

04

How PixVid gets around it

The move is simple, which is why it is interesting. The pixels of a frame go into tokens through a linear patch embedding, and a convolutional head, which the authors call the detailer head, renders those tokens back into pixels. The VAE drops out of the scheme entirely, and so does the decoder.

The second decision is about training. A model normally predicts velocity, the direction from noise towards an image. Here the model predicts the clean video directly and the noise is shifted much further. The authors say that without this pair of moves quality does not come together. The practical reading: pixel mode needs different maths, and ripping the VAE out of a pretrained model will not give you a good frame by itself.

The third decision is about time. Every convolution in the head is causal in time with a window of three frames, so frame t reads only frames t-2, t-1 and t. The head caches the previous two frames, so a frame is decoded the moment its tokens are ready, with no wait for the rest of a block.

The third frame reads only the two before it, so it is built at once without waiting for a block
05

1664x960 at the same token count

The token grid stays at 30x52, the same as 832x480, while each token gets bigger: 32x32 pixels instead of 16x16. Same compute, twice the points in the frame. The trade-off is obvious. A larger token means a coarser grid, and the detailer head has to recover the fine texture.

The model is finetuned from Wan2.1-14B. Quality comparable to the original latent model arrives in about 20K iterations, and the abstract mentions around 1.3 million training clips. That part matters: pixel mode was reached by transplanting a ready model, without building a new one from scratch.

06

Interaction and editing: what changes in practice

There is a hard number on latency. If a model compresses time by four frames, then at 30 fps each step commits four frames to one action, which adds up to 133 ms of waiting. PixVid generates in single-frame blocks, so an action lands on the next frame. For a button-press generation a hundred milliseconds is invisible. For a live scene where the camera follows a mouse, it decides the feel.

The authors also call out depth-conditioned editing. Under fast motion that edit holds up better than in the latent counterpart. A similar story sits next door: PI-H3 also drops the decoder and takes a step in pixel space, with the focus on generating in two to four steps. The idea is shared, the implementations differ, and that is a good sign, since several teams are testing the direction.

07

What is not proven, and what reaches a client

The abstract carries no numeric comparison against the base model and no cost per step, which AI Weekly also flagged. We found no statement on the project page about releasing weights or code. So today this is research, and it deserves to be treated as research.

We have no hands-on experience with this model and have not run it. The direction is practical though. If the approach holds, high resolution stops costing four times the compute, and scenes that obey frame-by-frame commands become realistic for advertising and interactive work. When a job needs a model trained on your own material, the outcome still depends on a person approving frames, and over long series that matters more than the architecture.

08

My conclusion

I read papers like this against one question: does the idea remove a specific trade-off. Here it does. Latent compression was an honest deal, training speed in exchange for a ceiling on resolution and responsiveness. PixVid shows the ceiling can be lifted while keeping a pretrained model, and that is cheaper than it looks at first glance.

For a client the consequence matters more than the arXiv number: 960p at the compute cost of 480p, and a live scene that answers a command on the following frame. No product uses this approach yet, so promising it to a client is early. Once models like this reach an API, the first question on a project will move to who signs off each frame.

frequently asked

01What is a VAE in a video model
It is an encoder and decoder pair that squeezes a frame into a compact record and unpacks it back. The model computes in the compressed form, the decoder returns the image, and part of the fine detail is lost at that step.
02Why is high resolution expensive for AI video
Because doubling resolution quadruples the token count, and attention grows faster still. In a latent model a 960p frame needs four times the tokens and sixteen times the attention of 480p.
03Can I already generate video at 1664x960
That mode exists in a research paper, without a product behind it. Weights and code are not announced on the project page, so the option is not available in a working pipeline yet.
04When will models like this be usable
The authors give no date. Since the model comes from finetuning a ready one, the road to an API looks shorter than for a model built from scratch, though that is my read and not a claim from the paper.