How to Build a Voice AI Pipeline from Scratch

The first step is always to capture the audio. If you’re building a web app, you can use the Web Audio API; for a desktop or server‑side tool, you can rely on pyaudio or sounddevice. The goal is a clean, mono, 16‑bit PCM file at 16 kHz or 24 kHz.

import sounddevice as sd import numpy as np import scipy.io.wavfile as wav def record ( duration = 5, fs = 16000 ): print ( f " Recording { duration } s … " ) audio = sd. rec ( int ( duration * fs ), samplerate = fs, channels = 1, dtype = ' int16 ' ) sd. wait () wav. write ( ' input.wav ', fs, audio ) print ( " Done. Saved to input.wav " ) Enter fullscreen mode Exit fullscreen mode Run record() and you’ll have a WAV file ready for ASR.

Open‑source ASR models such as Whisper or Vosk are great for local inference, but if you want instant results without worrying about GPU, a hosted API can shave hours off your dev cycle. For this tutorial, we’ll use Whisper’s API via openai-whisper for demo purposes.

import openai openai. api_key = ' YOUR_API_KEY ' def transcribe ( file_path ): with open ( file_path, ' rb ' ) as f: audio_data = f. read () transcript = openai. Audio. transcribe ( model = " whisper-1 ", file = audio_data ) return transcript [ " text " ] text = transcribe ( ' input.wav ' ) print ( " Transcribed text: ", text ) Enter fullscreen mode Exit fullscreen mode You’ll get a clean string that you can feed into your NLP engine or pass straight to a TTS service.

If your application needs to understand user intent, you can plug in a lightweight NLP library like spaCy or send the text to a cloud NLP endpoint (Google NLP, Azure Text Analytics). For the sake of brevity, we’ll skip this step and go straight to speech synthesis.

ElevenLabs offers a state‑of‑the‑art TTS engine that supports realistic voice cloning, expressive prosody, and dozens of voices. Their API is simple, well‑documented, and has a generous free tier—perfect for prototyping.

Sign up at ElevenLabs and grab your API key from the dashboard.

import requests API_KEY = ' YOUR_ELEVENLABS_KEY ' headers = { ' Accept ': ' application/json ', ' xi-api-key ': API_KEY, ' Content-Type ': ' application/json ' } def tts ( text, voice_id = ' en-US-Standard-B ' ): data = { " text ": text, " voice_settings ": { " stability ": 0.5, " similarity_boost ": 0.75 } } response = requests. post ( f " https://api.elevenlabs.io/v1/text-to-speech/ { voice_id } ", headers = headers, json = data ) response. raise_for_status () return response. content audio_bytes = tts ( text ) with open ( ' output.mp3 ', ' wb ' ) as f: f. write ( audio_bytes ) print ( " Saved TTS to output.mp3 " ) Enter fullscreen mode Exit fullscreen mode That’s it—no machine learning tricks, just a single POST request and an MP3 file.

ElevenLabs also lets you upload a short sample (≈5 seconds) of a target voice and generate a custom voice model. The endpoint is POST /v1/voices. After you create the voice, you can pass the returned voice_id to the TTS call.

def create_clone ( sample_path, name = " My Clone " ): with open ( sample_path, ' rb ' ) as f: sample = f. read () response = requests. post ( " https://api.elevenlabs.io/v1/voices ", headers = headers, files = { ' sample_file ': sample }, data = { ' name ': name } ) response. raise_for_status () return response. json ()[ ' voice_id ' ] clone_id = create_clone ( ' sample.wav ' ) print ( " Clone ID: ", clone_id ) Enter fullscreen mode Exit fullscreen mode Now call tts(text, voice_id=clone_id) to speak in that cloned voice.

def main (): record ( duration = 5 ) text = transcribe ( ' input.wav ' ) print ( " You said: ", text ) # Optional: create a cloned voice # clone_id = create_clone('sample.wav') # voice = clone_id voice = ' en-US-Standard-B ' # replace with clone_id if you created one audio = tts ( text, voice ) with open ( ' final.mp3 ', ' wb ' ) as f: f. write ( audio ) print ( " Final audio written to final.mp3 " ) if __name__ == " __main__ ": main () Enter fullscreen mode Exit fullscreen mode Run this script, speak into your microphone, and you’ll hear the same words spoken back in a realistic, cloned voice.

If you’re building a web app, here’s a minimal fetch‑based TTS snippet:

const fetch = require ( ' node-fetch ' ); const API_KEY = ' YOUR_ELEVENLABS_KEY '; const text = ' Hello, world! '; fetch ( `https://api.elevenlabs.io/v1/text-to-speech/en-US-Standard-B`, { method: ' POST ', headers: { ' Accept ': ' application/json ', ' xi-api-key ': API_KEY, ' Content-Type ': ' application/json ' }, body: JSON. stringify ({ text: text, voice_settings: { stability: 0.5, similarity_boost: 0.75 } }) }). then ( res => res. arrayBuffer ()). then ( buffer => { const fs = require ( ' fs ' ); fs. writeFileSync ( ' output.mp3 ', Buffer. from ( buffer )); console. log ( ' Saved to output.mp3 ' ); }). catch ( console. error ); Enter fullscreen mode Exit fullscreen mode 7. curl One‑liner curl -X POST "https://api.elevenlabs.io/v1/text-to-speech/en-US-Standard-B" \ -H "xi-api-key: YOUR_ELEVENLABS_KEY" \ -H "Content-Type: application/json" \ -d '{"text":"Hello, this is a test","voice_settings":{"stability":0.5,"similarity_boost":0.75}}' \ --output output.mp3 Enter fullscreen mode Exit fullscreen mode 8. Tips & Best Practices Tip Why Use short audio samples ElevenLabs’ cloning works best with a 5‑second clip that’s clear and free of background noise. Normalize volume ASR and TTS engines expect consistent amplitude; use a quick equalizer or a library like pydub.

Cache voice models If you’re cloning many users, store the voice_id so you don’t hit the API every request. Handle latency TTS can take 200‑300 ms per 10 words; consider streaming the audio or pre‑generating responses for interactive bots. Respect privacy Store or transmit audio only over HTTPS; delete raw recordings after processing if you don’t need them.

9. Common Pitfalls Problem Fix “Invalid API key” Double‑check you’re using the key from your ElevenLabs dashboard. “Voice not found” Make sure you’re using the correct voice_id. Audio distortion Ensure your microphone is set to the same sample rate you’re sending (16 kHz or 24 kHz).

Long processing time If you’re using a local ASR, upgrade to a GPU or switch to a hosted API for faster inference. 10. Wrap‑Up You now have a complete, end‑to‑end voice AI pipeline:

All the heavy lifting is handled by the APIs, so you can focus on the unique logic of your application.

Ready to take your prototype to the next level? Sign up at ElevenLabs, grab your API key, and start building a voice‑powered experience that feels genuinely human. Happy coding!