How an AI model moves lips and gestures to sound

A client watches a video with a talking character and spots it in three seconds: the mouth is doing its own thing while the voice rolls on. Then he points at a hand waving for the second time in ten seconds. Here is what sits inside lip sync, why the context window matters, where the sync falls apart and which frames to check.

Slices of sound feed arrows into an avatar head's mouth, a hand gesture floats above
01

What people mean by lips that land on the sound

The phrase describes a narrow moment: a person says a word, and the mouth closes on the same frame where the voice stops. The mechanics are plain. Audio is cut into short slices, fractions of a second. Each slice has a matching position of lips, tongue and jaw. A model learns to tie a sound to a mouth shape, then paints the face along a new audio track so the shape lands.

No formulas needed, though the boundary does matter. Matching the mouth to the voice is the floor. The finish sits higher. After that the viewer reads the whole face: blinking, the tilt of the head, the shoulders, a hand that lands on a stressed syllable. When all of that lives apart from the speech, the clip feels off even with perfect sync.

02

From a pasted-on mouth to video painted from sound

The older approach worked like a patch. Take a finished frame, find the mouth area, replace it with a new shape. The rest of the face stayed close to frozen. That survives a short line. On a long phrase you see it: cheeks do not take part, the jaw sits still, the seam of the patch shows in the lighting.

The new approach rebuilds the frame from the audio. A video model draws the face, the head and the hands while listening. HeyGen says so directly in the Avatar 6 release of November 17: motion is built from audio, the model reads the script and the delivery and picks gestures to match. Gemini Omni 1.2 Flash on Google Cloud promised on the same day a more natural dialogue, cleaner lip sync and several speakers inside one clip. Suno Visuals picks up the words of a song and syncs the lips in the music video.

Runway Characters pushes the refresh rate: the model produces new video in chunks of four frames at 24 fps every 166.7 ms, so speech and face keep running without gaps. Four frames is about a sixth of a second, and a step that size keeps the motion from freezing between blocks.

Lip-sync diagram: a row of short sound slices, each with its own mouth shape below
03

Why the context window decides the gestures

A short context window means a short memory. The model sees a few seconds of audio and invents motion while holding only those seconds. Then it restarts the segment, and the gesture comes back: the same raised hand, the same nod, the same turn of the head. It sounds clean and reads like a stuck record.

HeyGen grew the window in Avatar 6 from roughly 3 seconds to 30. The model remembers which gesture already happened and does not serve it again. On a thirty second take the difference is obvious compared with a five second insert. The practical conclusion for a buyer: check a long talking shot for repeated movement, since lip sync is usually fine by second ten while the gestures start spinning in circles.

04

Where lip sync breaks

The causes repeat. Noise and music on the track: the model hears consonants badly and places the wrong mouth shape. A translated language: the articulation of the new speech does not sit on the sounds of the old one. Fast delivery: at speed the model cannot keep up with the syllables and the lips start chewing air. A covered mouth: a hand at the face, a microphone, the edge of the frame.

Profile is a separate case. From the side there is little to see, and the model fills the lips by guesswork. Hence the shooting rule: full face or three quarters, mouth open, nothing across it. For vertical video that also means headroom, otherwise a raised hand leaves the frame.

A long window holds varied gestures; a short one loops the same gesture again
05

How to check a clip in five minutes

A check costs less time than a re-render. Watch a long phrase, five seconds or more without a pause, because that is where drift accumulates. Next the consonants P, B and M: the lips have to close on them, and a still frame shows it.

Third stop is silence. In a pause the mouth rests closed, and it should not keep articulating. Fourth, watch with sound on a phone, since social compression eats soft motion and exposes the jerks. Fifth, run the same script twice and compare the gestures. When they match frame for frame, the model is working with a short window.

06

Consent, identity checks and watermarks

A personal avatar learns from recordings of a person, and HeyGen requires an identity check for it. For a client that means a borrowed face does not clear the service paperwork. About 30 seconds of video is enough to build one, and those thirty seconds are handed over on purpose, with the person knowing the model will speak in their voice and wear their face.

The second part is labelling. Gemini Omni stamps its output with SynthID, an invisible mark that lets anyone trace where the clip came from. For advertising that is mostly good news: disclosure stops being a question of honesty and turns into a technical detail.

07

Bottom line

The routine for a new project is short. First a test on the worst line available: long, with P, B, M and a pause, five seconds of it. Then a decision about the whole clip. If the model repeats a gesture or smears the lip closure on that test, the full render does the same, and a second run changes nothing.

I do not treat 2K as proof of anything. A crisp picture with a drifting mouth looks worse than a soft picture with exact articulation. Lip sync is checked on still frames, and the resolution line in a spec sheet proves little.

For a run of videos with a talking character this is already a working tool. AI video production ships in batches, and the avatar covers the cheap part of the job: a talking head for a product page, for an explainer of a service, for inserts inside vertical stories. AI video ads gain the most, because the face fills the frame and any drift shows at once.

The open topic is emotion. Calm delivery holds up, and shouting or whispering still needs careful recording and a separate check. A neighbouring topic is how an AI video model picks the next frame, since lip sync is really a special case of predicting the next frame from sound.

Related reading: real-time talking avatar.

frequently asked

01What is lip sync in plain terms
Matching the mouth to the voice: the shape of the mouth on screen lines up with the sound. Audio is split into short slices, and the model picks lips, tongue and jaw positions for each slice.
02How much footage does a personal avatar need
HeyGen cites about 30 seconds of video for a personal avatar. The model learns from a person's recordings and then drives that face along a new audio track.
03Why does a gesture repeat inside one clip
Usually because of a short context window: the model remembers a few seconds and restarts the movement. Avatar 6 raised the window from about 3 seconds to 30, and repeats became rarer.
04Does a person have to agree to an avatar
Yes. HeyGen requires an identity check for a personal avatar, and someone else's face should not end up in a clip without consent. Gemini Omni output additionally carries the SynthID mark.