What people mean by lips that land on the sound
The phrase describes a narrow moment: a person says a word, and the mouth closes on the same frame where the voice stops. The mechanics are plain. Audio is cut into short slices, fractions of a second. Each slice has a matching position of lips, tongue and jaw. A model learns to tie a sound to a mouth shape, then paints the face along a new audio track so the shape lands.
No formulas needed, though the boundary does matter. Matching the mouth to the voice is the floor. The finish sits higher. After that the viewer reads the whole face: blinking, the tilt of the head, the shoulders, a hand that lands on a stressed syllable. When all of that lives apart from the speech, the clip feels off even with perfect sync.
From a pasted-on mouth to video painted from sound
The older approach worked like a patch. Take a finished frame, find the mouth area, replace it with a new shape. The rest of the face stayed close to frozen. That survives a short line. On a long phrase you see it: cheeks do not take part, the jaw sits still, the seam of the patch shows in the lighting.
The new approach rebuilds the frame from the audio. A video model draws the face, the head and the hands while listening. HeyGen says so directly in the Avatar 6 release of November 17: motion is built from audio, the model reads the script and the delivery and picks gestures to match. Gemini Omni 1.2 Flash on Google Cloud promised on the same day a more natural dialogue, cleaner lip sync and several speakers inside one clip. Suno Visuals picks up the words of a song and syncs the lips in the music video.
Runway Characters pushes the refresh rate: the model produces new video in chunks of four frames at 24 fps every 166.7 ms, so speech and face keep running without gaps. Four frames is about a sixth of a second, and a step that size keeps the motion from freezing between blocks.

Why the context window decides the gestures
A short context window means a short memory. The model sees a few seconds of audio and invents motion while holding only those seconds. Then it restarts the segment, and the gesture comes back: the same raised hand, the same nod, the same turn of the head. It sounds clean and reads like a stuck record.
HeyGen grew the window in Avatar 6 from roughly 3 seconds to 30. The model remembers which gesture already happened and does not serve it again. On a thirty second take the difference is obvious compared with a five second insert. The practical conclusion for a buyer: check a long talking shot for repeated movement, since lip sync is usually fine by second ten while the gestures start spinning in circles.
Where lip sync breaks
The causes repeat. Noise and music on the track: the model hears consonants badly and places the wrong mouth shape. A translated language: the articulation of the new speech does not sit on the sounds of the old one. Fast delivery: at speed the model cannot keep up with the syllables and the lips start chewing air. A covered mouth: a hand at the face, a microphone, the edge of the frame.
Profile is a separate case. From the side there is little to see, and the model fills the lips by guesswork. Hence the shooting rule: full face or three quarters, mouth open, nothing across it. For vertical video that also means headroom, otherwise a raised hand leaves the frame.

How to check a clip in five minutes
A check costs less time than a re-render. Watch a long phrase, five seconds or more without a pause, because that is where drift accumulates. Next the consonants P, B and M: the lips have to close on them, and a still frame shows it.
Third stop is silence. In a pause the mouth rests closed, and it should not keep articulating. Fourth, watch with sound on a phone, since social compression eats soft motion and exposes the jerks. Fifth, run the same script twice and compare the gestures. When they match frame for frame, the model is working with a short window.
Consent, identity checks and watermarks
A personal avatar learns from recordings of a person, and HeyGen requires an identity check for it. For a client that means a borrowed face does not clear the service paperwork. About 30 seconds of video is enough to build one, and those thirty seconds are handed over on purpose, with the person knowing the model will speak in their voice and wear their face.
The second part is labelling. Gemini Omni stamps its output with SynthID, an invisible mark that lets anyone trace where the clip came from. For advertising that is mostly good news: disclosure stops being a question of honesty and turns into a technical detail.
Bottom line
The routine for a new project is short. First a test on the worst line available: long, with P, B, M and a pause, five seconds of it. Then a decision about the whole clip. If the model repeats a gesture or smears the lip closure on that test, the full render does the same, and a second run changes nothing.
I do not treat 2K as proof of anything. A crisp picture with a drifting mouth looks worse than a soft picture with exact articulation. Lip sync is checked on still frames, and the resolution line in a spec sheet proves little.
For a run of videos with a talking character this is already a working tool. AI video production ships in batches, and the avatar covers the cheap part of the job: a talking head for a product page, for an explainer of a service, for inserts inside vertical stories. AI video ads gain the most, because the face fills the frame and any drift shows at once.
The open topic is emotion. Calm delivery holds up, and shouting or whispering still needs careful recording and a separate check. A neighbouring topic is how an AI video model picks the next frame, since lip sync is really a special case of predicting the next frame from sound.
Related reading: real-time talking avatar.



