YouTube adding AI dub technology to translate videos for creators

A new feature for YouTube will allow creators to dub their videos into other languages with an upcoming AI integration.

At VidCon 2023, YouTube announced that it would begin letting creators dub their videos into other languages via the use of AI. The software, developed by Aloud and part of Google’s Area 120 incubator, will open up content to a much wider audience once implemented.

Speaking with The Verge, YouTube’s Amjad Hanif said that the tool is already being tested with hundreds of creators and that the AI program supports just a few languages right now.

Right now, you’ll find Spanish, Portuguese, and English available for translation purposes.

Currently, YouTube allows viewers to use automatic captions if the original uploader hasn’t provided subtitles. However, this can be very limited depending on the accents, and translating subtitles between languages is spotty or leads to worse accusations.

You can already try out the function yourself, as a science channel, Amoeba Sisters, seems to already have a video using it. To do so, just click the little gear, and audio tracks should be an option.

The results are very clearly AI-generated, with a robotic undertone as the Spanish voice tries to keep up with the English one.

YouTube introduces new AI dubbing tool for creators

Hanif mentions that the audio should improve in 2024, as Aloud continues to develop it. This will begin to mimic inflections, expressions, and lip sync.

While it’s unclear how YouTube would implement lip-synced video, we’ve already seen Nvidia alter the human body on the fly to keep eyes looking directly at the camera.

However, we suspect that it’ll function similarly to how YouTube is implementing the dubbing process. Aloud takes the audio, similar to the automatic captions. Once it has been processed and transcribed into text, you can edit and make adjustments. Finally, the AI voice is overlayed and integrated into the content.