Create an AI Dubbing Tool for Video Content

If you’ve ever needed to localize a video, add a narration track, or simply replace a shaky on‑camera voice, doing it manually can be a nightmare. Traditional dubbing requires hiring voice actors, booking studios, and spending hours syncing audio to video. With modern text‑to‑speech (TTS) and voice‑cloning tech, you can automate most of that workflow in a few lines of code.

In this post I’ll walk you through a practical, end‑to‑end pipeline for creating an AI dubbing tool:

The heavy lifting for the voice generation will be done by ElevenLabs, a high‑quality, low‑latency TTS platform that also offers voice cloning. Let’s dive in.

You can install the Python dependencies with:

pip install pydub requests openai-whisper Enter fullscreen mode Exit fullscreen mode Tip: If you’re on Windows, make sure ffmpeg.exe is on your PATH. On macOS/Linux you can usually install it via Homebrew or apt.

First, we need the raw audio track from the source video. ffmpeg makes this a one‑liner:

ffmpeg -i input_video.mp4 -vn -acodec pcm_s16le -ar 22050 -ac 1 original.wav Enter fullscreen mode Exit fullscreen mode -vn – ignore the video stream -acodec pcm_s16le – uncompressed WAV (easier for later processing) -ar 22050 – 22.05 kHz sample rate (ElevenLabs works well with this) -ac 1 – mono channel (reduces file size) You now have original.wav ready for transcription or direct voice cloning.

If you need subtitles or want to dub into a different language, run the audio through a speech‑to‑text model. Here’s a quick Whisper example:

import whisper model = whisper. load_model ( " base " ) result = model. transcribe ( " original.wav " ) text = result [ " text " ] print ( " Transcribed text: ", text ) Enter fullscreen mode Exit fullscreen mode For translation, Whisper can output directly in another language:

result = model. transcribe ( " original.wav ", language = " en ", task = " translate " ) translated = result [ " text " ] print ( " Translated to English: ", translated ) Enter fullscreen mode Exit fullscreen mode Save the final script you want to feed into the TTS engine into a variable called script.

ElevenLabs offers a simple REST API. Below is a Python wrapper that sends the script and receives an audio file.

import requests ELEVEN_API_KEY = " YOUR_ELEVENLABS_API_KEY " VOICE_ID = " EXAMPLE_VOICE_ID " # you can use a default voice or a cloned one def synthesize ( text: str, output_path: str ): url = f " https://api.elevenlabs.io/v1/text-to-speech/ { VOICE_ID } " headers = { " xi-api-key ": ELEVEN_API_KEY, " Content-Type ": " application/json " } payload = { " text ": text, " model_id ": " eleven_monolingual_v1 ", " voice_settings ": { " stability ": 0.75, " similarity_boost ": 0.85 } } response = requests. post ( url, json = payload, headers = headers ) response. raise_for_status () with open ( output_path, " wb " ) as f: f. write ( response. content ) print ( f " Saved synthesized audio to { output_path } " ) # Example usage: script = " Hello, world! This is an AI‑generated dub. " synthesize ( script, " dub.wav " ) Enter fullscreen mode Exit fullscreen mode Using the API from curl If you prefer a quick terminal test, here's the equivalent curl command:

curl -X POST "https://api.elevenlabs.io/v1/text-to-speech/EXAMPLE_VOICE_ID" \ -H "xi-api-key: YOUR_ELEVENLABS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "text": "Hello, world! This is an AI‑generated dub.", "model_id": "eleven_monolingual_v1", "voice_settings": { "stability": 0.75, "similarity_boost": 0.85 } }' \ --output dub.wav Enter fullscreen mode Exit fullscreen mode Note: Replace EXAMPLE_VOICE_ID with the ID of the voice you want to use. ElevenLabs lets you clone your own voice with just a few minutes of audio – perfect for brand‑consistent dubbing.

The synthesized audio will usually be slightly shorter or longer than the original. A simple way to keep sync is to stretch/compress the track to match the original duration using pydub.

from pydub import AudioSegment original = AudioSegment. from_wav ( " original.wav " ) dub = AudioSegment. from_wav ( " dub.wav " ) # Match length dub = dub. set_frame_rate ( original. frame_rate ) dub = dub. set_channels ( original. channels ) dub = dub. set_sample_width ( original. sample_width ) # Speed up or slow down to fit the original duration ratio = len ( original ) / len ( dub ) dub = dub. speedup ( playback_speed = ratio, chunk_size = 150, crossfade = 25 ) dub. export ( " dub_aligned.wav ", format = " wav " ) Enter fullscreen mode Exit fullscreen mode Now we can replace the original audio track in the video:

ffmpeg -i input_video.mp4 -i dub_aligned.wav -c:v copy -map 0:v:0 -map 1:a:0 -shortest final_dubbed.mp4 Enter fullscreen mode Exit fullscreen mode -c:v copy – keep the original video codec (no re‑encoding) -map – select streams (video from input 0, audio from input 1) -shortest – stops encoding when the shortest stream ends (prevents extra silence) You now have final_dubbed.mp4 with an AI‑generated voice track!

Below is a minimal script that ties the steps together. Feel free to expand it with error handling, batch processing, or a web UI.

import os import subprocess from pathlib import Path import whisper import requests from pydub import AudioSegment # ---- Config ---- VIDEO_IN = " input_video.mp4 " VIDEO_OUT = " final_dubbed.mp4 " ELEVEN_API_KEY = " YOUR_ELEVENLABS_API_KEY " VOICE_ID = " EXAMPLE_VOICE_ID " # 1. Extract audio subprocess. run ([ " ffmpeg ", " -i ", VIDEO_IN, " -vn ", " -acodec ", " pcm_s16le ", " -ar ", " 22050 ", " -ac ", " 1 ", " original.wav " ], check = True ) # 2. Transcribe (or just set a script) model = whisper. load_model ( " base " ) result = model. transcribe ( " original.wav " ) script = result [ " text " ] print ( " Script to dub: ", script ) # 3.

Synthesize with ElevenLabs def synthesize ( text, out_path ): url = f " https://api.elevenlabs.io/v1/text-to-speech/ { VOICE_ID } " headers = { " xi-api-key ": ELEVEN_API_KEY, " Content-Type ": " application/json " } payload = { " text ": text, " model_id ": " eleven_monolingual_v1 " } resp = requests. post ( url, json = payload, headers = headers ) resp. raise_for_status () Path ( out_path ). write_bytes ( resp. content ) synthesize ( script, " dub.wav " ) # 4. Align audio length orig = AudioSegment. from_wav ( " original.wav " ) dub = AudioSegment. from_wav ( " dub.wav " ) ratio = len ( orig ) / len ( dub ) dub = dub. speedup ( playback_speed = ratio ) dub. export ( " dub_aligned.wav ", format = " wav " ) # 5. Merge back into video subprocess. run ([ " ffmpeg ", " -i ", VIDEO_IN, " -i ", " dub_aligned.wav ", " -c:v ", " copy ", " -map ", " 0:v:0 ", " -map ", " 1:a:0 ", " -shortest ", VIDEO_OUT ], check = True ) print ( " ✅ Dubbing complete!

Check: ", VIDEO_OUT ) Enter fullscreen mode Exit fullscreen mode You can run this script with python dub_tool.py (after filling in your ElevenLabs API key and voice ID). The result is a fully dubbed video with just a few seconds of processing time.

All of these extensions build on the same core steps we covered, so you can adapt the pipeline to your product’s needs.

Building an AI dubbing tool is now a weekend project thanks to modern TTS services. By leveraging ElevenLabs for high‑quality voice synthesis, you get natural‑sounding output without the overhead of recording studios.

Ready to give it a spin? Grab your free trial and API key from ElevenLabs, plug it into the snippets above, and start dubbing your videos in minutes. Happy coding!