Switching an OpenAI Realtime voice agent to ElevenLabs: the 2-line change, the real cost per call, and the WebRTC catch

Posted on Nov 3 • Originally published at tokenlean.wordpress.com

Disclosure: this post has affiliate links. If you sign up through them I may earn a commission, at no extra cost to you.

TL;DR: ElevenLabs shipped Eleven Realtime on Oct 20, 2026: two speech-to-speech models (v1 Mini and v1 Max) behind an API that speaks OpenAI's Realtime protocol. If your voice agent already uses @openai/agents-realtime over WebSocket, the switch is a new endpoint URL and a new model string. If you're on WebRTC, it's a transport rewrite first. Cost is $0.08 per connected minute plus LLM tokens, and the per-minute part dominates.

Browser mic ──PCM16 24kHz──► wss://api.elevenlabs.io/v1/convai/realtime │ ┌──────────┴───────────┐ │ ASR → turn-taking → │ one model ID: │ LLM → TTS (tuned as │ eleven_realtime_v1_mini │ a single system) │ eleven_realtime_v1_max └──────────┬───────────┘ Speaker ◄──PCM16 audio───────────┘ (+ text at every step for logs/guardrails) Enter fullscreen mode Exit fullscreen mode The numbers ElevenLabs published v1 Mini v1 Max Time to first audio (avg) 0.62 s 0.57 s Big Bench Audio (spoken reasoning) 75.9% 98.8% Full-Duplex-Bench (pauses, interruptions) 95.8% 94.1% Tool-use task completion 60.3% 87.1% Input tokens / 1M $0.20 $1.80 Output tokens / 1M $6.00 $9.00 Connected audio $0.08/min $0.08/min Benchmarks are from Artificial Analysis as reported in ElevenLabs' launch post; prices from the ElevenLabs API pricing page (checked Nov 3, 2026).

The one line that should drive your model choice: tool use, 60% vs 87%. If your agent calls functions (book a slot, look up an order), Mini will drop roughly 4 in 10 tasks on that benchmark. If it mostly chats, Mini is the better-sounding, cheaper pick.

ElevenLabs' docs say the API uses the same client and server events over the same WebSocket transport as OpenAI's Realtime API. In practice there are two changes in the browser, plus one on your server.

You'll need an ElevenLabs API key for step 1 ( create one here; the free tier is enough to test).

1. Server: mint a short-lived client secret (never ship your API key to the browser).

import os from elevenlabs.client import ElevenLabs from elevenlabs import ( CreateClientSecretRequest, ExpiresAfter, OpenAiSessionConfigInput, OpenAiAudioConfigInput, OpenAiAudioInputConfigInput, OpenAiAudioInputFormat, OpenAiAudioOutputConfigInput, OpenAiAudioOutputFormat, OpenAiTurnDetection ) el = ElevenLabs ( api_key = os. environ [ " ELEVENLABS_API_KEY " ]) secret = el. conversational_ai. realtime. create_client_secret ( request = CreateClientSecretRequest ( session = OpenAiSessionConfigInput ( type = " realtime ", model = " eleven_realtime_v1_mini ", # or eleven_realtime_v1_max instructions = " You are a concise, friendly voice assistant. ", output_modalities = [ " audio " ], audio = OpenAiAudioConfigInput ( input = OpenAiAudioInputConfigInput ( format = OpenAiAudioInputFormat ( type = " audio/pcm ", rate = 24000 ), turn_detection = OpenAiTurnDetection ( type = " semantic_vad ", eagerness = " medium " ), ), output = OpenAiAudioOutputConfigInput ( format = OpenAiAudioOutputFormat ( type = " audio/pcm " ), voice = " brian " ), ), ), expires_after = ExpiresAfter ( anchor = " created_at ", seconds = 600 ), ) ) # return only secret.value to the browser Enter fullscreen mode Exit fullscreen mode 2. Browser: same OpenAI SDK, new URL and model.

import { OpenAIRealtimeWebSocket, RealtimeAgent, RealtimeSession } from " @openai/agents-realtime "; const { value: clientSecret } = await fetch ( " /api/token " ). then ( r => r. json ()); const agent = new RealtimeAgent ({ name: " support ", instructions: " Keep replies short. " }); const transport = new OpenAIRealtimeWebSocket ({ url: " wss://api.elevenlabs.io/v1/convai/realtime ", // ← change #1 }); const session = new RealtimeSession ( agent, { transport, model: " eleven_realtime_v1_mini ", // ← change #2 config: { outputModalities: [ " audio " ], audio: { output: { voice: " brian " } } }, }); await session. connect ({ apiKey: clientSecret }); Enter fullscreen mode Exit fullscreen mode The catches nobody puts in the headline WebSocket only. If your app uses the SDK's WebRTC transport, you have to move to WebSocket first.

You handle the audio. The OpenAI SDK only wires mic and speaker automatically over WebRTC. Over WebSocket you capture and play PCM16 at 24 kHz yourself ( wavtools works; it's what OpenAI's own WebSocket examples use).

Browser flow only, for now. The docs say server-side OpenAI Realtime SDK docs aren't published yet, so phone-bridge setups that run the SDK on a server are untested territory. The API is separate from ElevenAgents.

Phone numbers, WhatsApp, knowledge base, guardrails and testing live in ElevenAgents. If you need those, build there and pick a Realtime model from the dropdown instead. Burst pricing.

Connected audio above your plan's concurrency limit bills at $0.16/min, double the normal rate. Concurrency runs from 4 calls (free/pay-as-you-go) to 40 (Business). What a 5-minute call costs (worked estimate) The connected-audio fee is fixed: 5 × $0.08 = $0.40.

Tokens depend on your prompt and turn count. Here's an illustrative support call with my assumptions spelled out: 30 turns, about 3,000 tokens of context re-sent per turn (90k input tokens total), and about 60 output tokens per reply (1.8k output).

You are {brand}'s phone assistant. Goal: resolve the caller's issue in under 3 minutes. Rules: - Answer in 1–2 sentences.

Never read lists longer than 3 items aloud. - Ask one question at a time. - If you need data, call the tool immediately; say "one sec" while it runs. - When the issue is resolved, confirm in one sentence and call end_call. - If the caller asks for a human or you fail twice, transfer. Enter fullscreen mode Exit fullscreen mode Which one should you pick? Your agent mostly…

Pick Chats, tutors, plays a character v1 Mini: cheapest, best turn-taking score Calls tools / follows a script / qualifies leads v1 Max: 87% vs 60% on tool tasks Needs phone, WhatsApp, a knowledge base ElevenAgents with a Realtime model selected Is an existing OpenAI Realtime app on WebSocket Realtime API: swap URL + model, test today If you want to try it, the free tier includes roughly 14 minutes of Realtime v1 Mini, enough to run your existing client against it this afternoon. Get an ElevenLabs API key →

Sources: ElevenLabs launch post (elevenlabs.io/blog/realtime), OpenAI Realtime compatibility guide (elevenlabs.io/docs/eleven-agents/guides/openai-realtime-compatibility), ElevenLabs API pricing (elevenlabs.io/pricing/api).

Affiliate disclosure: TokenLean participates in affiliate programs, including ElevenLabs'. Some links in this article are affiliate links, which means I may earn a commission if you click and make a purchase, at no additional cost to you. I only recommend tools I'd use in a real build, and the facts above come from the vendor's public docs and pricing pages. Prices and benchmarks change; check the official pages before you commit.