Building an AI Outbound Calling Agent on Google Voice with Python, Groq and FastAPI

Most AI voice-agent tutorials begin the same way: sign up for a telephony API, get a webhook when the call connects, then stream audio. google-voice-dispatch-agent works differently. It runs real outbound calls through Google Voice in a Chrome browser, and that changes which problems are hard.

This post walks through how it's built, based on the project's own README. Code: https://github.com/mafzalkalwardev/google-voice-dispatch-agent

A telephony API tells you when a call is answered. A browser doesn't. You read the page (the DOM), and the page is noisy: banners from the last call stay on screen, call controls can show up before anyone picks up, and voicemail cues can appear while the phone is still ringing.

So the AI is the easy part. The hard parts are:

The project is built around a few rules:

The most important logic is in src/google_voice.py. It's a conservative state gate:

Every decision is logged: current state, elapsed ringing time, DOM cues, answered controls, call-active status, voicemail cues, the audio classifier's result and the timeout reason. When something goes wrong, you can see exactly why the agent did what it did.

The session model and its CallState transitions are kept in their own file, call_session.py, so state handling can be tested apart from the browser.

Once a call is confirmed connected, conversation_loop.py takes over:

Two details cut down awkward silence. The opening line and the voicemail message are generated before dialing, so they're ready the moment the state gate allows them. And voicemail_detector.py runs an audio classifier in the first seconds of a connected call, listening for greeting, beep and live-call signals. When it confirms voicemail, the loop hands off to voicemail playback instead of trying to hold a conversation with a recording.

crm.py handles CRM enrichment and keeps separate archives for connected and failed calls. Contacts come from CSV or XLSX files and settings from JSON.

The agent is meant to run on a Windows desktop, so:

The tests/ folder covers state handling, mock dialing, the CRM, audio helpers, web routes and the realtime loop. It runs with pytest on GitHub Actions. With a browser-driven agent, mock dialing tests are what let you change the state gate without placing real calls.

Source: https://github.com/mafzalkalwardev/google-voice-dispatch-agent More about the call-state logic: https://dev.to/mafzalkalwardev/how-my-google-voice-ai-agent-knows-when-to-talk-call-state-detection-groq-and-tts-e5a

Muhammad Afzal Kalwar, Python Automation Engineer & Founder, INDUS Web Agency. Portfolio: https://mafzalkalwardev.github.io