Lip Sync Battle – I put three leading AI video tools to the test

One of the fastest-improving areas of artificial intelligence video is lip syncing – that is, the ability to have an AI character speak and appear as if they are speaking the words they are saying.

There are a number of companies offering lip syncing, including Pika Labs, Synchlabs, and character-based platforms like Hey Gen and Synthesia. The latter two are possibly the best examples of lip syncing I’ve ever seen, but they focus more on avatars than animation.

In this story, I focused on platforms working in the AI video space, rather than avatar creation. Kling and Runway are the most similar, offering complete video creation platforms with lip syncing as a feature. Hedra is currently focused on the character, but is creating a more comprehensive, controllable video model that starts with the character. That’s why I chose these three for this test.

Design the fight

This will be a five round competition between the three models, three rounds using an image I gave them and two rounds using their own image/video generation capabilities. (I will explain how many laps I ran at the end.)

We will use the same image with each tool, but use their own built-in voices and monologue script. I focused on 10 second snippets, although Hedra can be up to a minute long. This is intended to ensure the consistency of all three models.

Hedra works a little differently than Kling and Runway. The latter two start with a video and record the lip movement within the video; Hedra begins with a picture. The end results are similar.

Round 1: The static face test

(Image: © Midjourney/Future AI)

This should be easiest. We commissioned Midjourney: “A neutral close-up portrait of a person with minimal expression, well lit in a natural studio setting, showing a front view of the face.” The background is a soft, blurred gradient with no distractions. The skin tones should be natural and the character should appear calm and without significant emotions.”

We then selected a custom voice for each of the three models and decided to have it say, “Hello, welcome to the future of AI video generation.” I don’t actually exist, but I can still talk to you thanks to the wonders of lip sync. “

This first test should have taken 20 minutes despite the added complexity of lip syncing, but Kling, as good as it is in terms of visual and motion realism, is by far the slowest AI video model. Runway runs almost in real time thanks to Turbo and Hedra animates an image, so it’s quick.

This was a close round between Hedra with the more realistic voice and mouth movement and Kling with the more impressive movement. The flickering didn’t convince me, so I am and gave it to Hedra on this occasion.

Round 2: The Expression Challenge

In the middle of the journey

(Image: © Midjourney/Future AI)

In this test we have an ultra-close-up image from Midjourney: “A close-up portrait of a person with an expressive, happy face, showing teeth in a wide smile.” The lighting is bright and warm, creating a happy and energetic mood. The background is a soft, light pastel color that does not distract from the facial expression.”

Each of the three models was asked to say the sentence: “Life can be strange sometimes, but it’s a good, strange, happy way of being. Something to smile about.” This tests the ability to grasp emotional context.

All three were nightmarish depictions. It is clear that if you want a good lip sync, you should start with your mouth closed. I can’t crown a winner, but I reluctantly will Give it to Hedra for the least terrible mouth movement.

Round 3: The action scene

In the middle of the journey

Finally, we’ll see how well each participant can animate someone’s lips in the middle of a conversation without looking directly at the camera. We use the Midjourney Prompt: “A mid-action shot of a person turned slightly to the side and speaking with their hand raised, as if gesturing during an intense conversation. The face shows determination and concentration. The background is a dynamic, slightly blurred urban street scene with movements that suggest the person is talking while moving.”

I gave the character the script: “So I told him if he wants to buy the car, he has to come back with a better price. Never heard from him again.”

None of them were perfect, but I think Hedra and Runway did a better job than Kling. Overall I think Runway won this round for the most realistic lip sync.

The winner: Hedra

I originally planned five rounds, but Kling took so long to create each video that it was impossible to complete in enough time. The last two tests looked at the text-to-video capabilities without the splash image, but the results were too sporadic to be useful.

Hedra’s Character-2 emerged victorious, and to some extent unavoidably so. It starts with an image and an animation, with the other two depicting the mouth movement in a video and having to synchronize the lips with the sound. Of the video models, I think Kling was better overall, but this was a first after the post-test, so Runway technically came in second place.

If I were to repeat this experiment, I would use external noise. Create more consistency, always use generated images, and run a wider range of tests. I just wish Kling was faster.

Recent Posts