How image-to-video AI works: inside the anchor frame

You upload a still, type the camera pushes in slowly, and get a clip where the subject is looking somewhere else. Between the photo and the finished file there are a few steps nobody explains to the client. Here is what the model takes from the image, what comes from the prompt, why the first frame can differ from your picture and why everything drifts by the end of a long clip.

A photo acts as an anchor while a dashed path of frames drifts away from it
01

What the model is given

In image-to-video the picture works as a condition for the whole scene. Runway says it plainly in the official Gen-4 Video Prompting Guide: the input image sets the visual starting point of the entire process, subjects, composition, colours, light and style included. The model then builds motion on top of that condition.

A simple way to hold it: the image is an anchor the model is tied to at the start, the text is the direction it gets pulled in. The further from the anchor, the more the model invents on its own.

Two terms worth knowing. The anchor frame is the reference image the generation starts from. The latent is a compressed representation of frames that the model works with internally, and it becomes pixels only on the way out.

02

What the model takes from the picture, and what it leaves to the prompt

From the still it takes the subject's look, the arrangement of objects, the palette, the direction of light, the texture. All of that already counts as given, which is why Runway advises describing motion in the text. The guide adds a detail clients rarely enjoy: repeating image details in the prompt can reduce motion. The harder you work at listing what the subject is wearing, the less the subject moves.

Runway treats the image and the text as one prompt. A working split follows from that: the photo owns how the frame looks, the text owns what happens inside it. Two paragraphs about a jacket and half a line about the action give you a sluggish clip.

Diagram: a photo is compressed into a latent grid and decoded into frames that gradually shift
03

Inside: compressed frames instead of pixels

The open model Vimoo 1.0 (documentation and model card as retold by llms.blog on 9 November 2026) shows a layout you meet elsewhere too. A separate image encoder, SigLIP, encodes the picture, while the video is compressed and reconstructed by a Wan 2.2 VAE. In plain terms: the model does not paint millions of pixels per frame, it works with a compressed representation, the latent, and full frames appear at the end. The model thinks in notes.

One practical consequence. A small detail that made it poorly into those notes can show up late or change during the clip. A still frame hides it, motion shows it.

04

Why frame one can differ from the picture

The Vimoo 1.0 documentation names two reasons. The image is resized and centre cropped to the video format. The first frame does not necessarily match the picture pixel for pixel. A landscape photo and a vertical clip mean the sides go away and the subject ends up closer to the edge than in the original.

The practical takeaway: check small details such as a print on a shirt, a piece of jewellery or a word on packaging in frame one separately, since that is where a gap shows most.

The first frame is not the only handle. The Google Gemini Omni guide describes a mode where you set the first and the last frame and the model builds continuous video between them. In that setup you describe what is in the first frame, the path the camera takes and what is in the last. The open Pika H3 Flex in Diffusers 0.43, per its card from 12 November 2026, supports a first frame, a last frame or both, plus multimodal references. Models are built differently, so you test on your own material.

A camera frames a wide horizontal scene into a vertical frame, side edges left outside
05

Why a long clip drifts

In many designs the model keeps recent frames in more detail than early ones. A telling example from world models: lucidml published a GTA world model on Hugging Face (11-12 November 2026, retold by ArtRealmAI) where the starting image stays in memory throughout and never gets evicted, the last two frames are kept at full resolution and older ones remain as compressed notes. On long sessions the image drifts anyway.

That is the argument for anchors and short segments. Kandinsky S2V 1.0 in Diffusers 0.43 animates a person from a picture and a speech recording, generates in 5 second chunks and passes memory between chunks through FramePack. The shorter the gap between reference frames, the less room drift has. On a long clip with no mid-points the error accumulates, and faces, hands and background give it away first.

06

Which picture to hand over

Format. The image should match the ratio of the clip. For vertical that is 9:16, otherwise the centre crop eats the edges and shifts the composition.

Edges. Keep the subject away from the border. A crop and a moving camera will lose it first.

Cleanliness. No artefacts in the source. Small text, extra fingers and blurry patches get read as part of the scene and repeated.

One centre. Three equal subjects in frame can lead the model to move all three and the viewer's attention spreads.

07

What to put in the prompt

The text owns motion: what the subject does, how the camera moves, what changes in the scene. One idea per shot. Repeating the subject's appearance is wasted work, the model already has it from the image.

A last frame earns its place when the clip has an ending: the product lands in a hand, the subject turns to the shop window, the car pulls up at the entrance. Then you prepare and describe it separately, the same way as the first. Without an ending, motion and one anchor are enough.

This is how a set gets built. Our AI video production runs as a stream, each clip with its own anchor frame and its own short text about motion. When you need one careful clip for a specific job, that is our handcrafted AI videos, where a person finishes the frame. The overall order of a generation, from text to a finished file, I covered in how an AI video generator works step by step. Control over reference points has its own piece, first and last frame in AI video.

08

My conclusion

The model does not bring a photograph to life in the everyday sense. It extends a scene from the image, holds it as support and slowly loses it. The practice that follows is plain: short segments, a check that frame one matches the source, a last frame wherever the ending matters.

If after the first pass the subject looks nothing like the photo, I change the input and leave the prompt alone. Another frame, another ratio, another moment. Text steers motion, appearance arrives from the picture.

frequently asked

01Can one photo become a long clip
It depends on the model: the length allowed per pass differs. The longer a segment runs, the further it drifts from the starting frame, so a long clip is safer when assembled from segments, each with its own anchor frame.
02Why does the subject not look like the photo
Most often it is format and crop: the image gets resized and centre cropped to the clip's ratio. A second possible reason: the Vimoo docs say outright that frame one is not guaranteed to match the picture pixel for pixel.
03Do I still need a prompt if I supply a picture
Yes. Image and text count as one prompt, and motion comes from the text. With no described action the model may leave the scene almost still.
04What if the clip drifts towards the end
Cut it into short segments with reference frames and check the joins. If the ending matters, set a last frame and describe the camera path between the two.