How I Built a Voice-First Desktop AI Assistant with a 3D Holographic HUD (Sub-400ms Latency)

Most voice assistants on the desktop today feel sluggish. You press a button, wait 2 to 3 seconds for cloud processing, and get a robotic voice response while staring at a static icon.

I wanted something that felt like a true sci-fi command deck—fast, responsive, and visual.

So I spent the last few weeks building J.A.R.V.I.S., a voice-first desktop engineering assistant built with Electron, TypeScript, Three.js, and a low-latency local neural pipeline.

Here is how the architecture works, how I got the voice loop under 400ms, and what I learned along the way.

🔗 Live Website: www.sohan.cfd 📦 Beta Deck: www.sohan.cfd/beta

The biggest reason voice assistants feel unnatural is Turn-Taking Latency. Humans converse with an average pause of only 200–300ms.

Total delay: 2.5 to 3.5 seconds. That's far too slow for an engineering companion.

To get TTFT (Time to First Token) and response time below 400ms, I redesigned the pipeline:

Instead of waiting for the user to stop speaking before sending audio, audio is captured in 80ms PCM slices. Voice Activity Detection (VAD) monitors silence thresholds locally in the Electron main process.

As soon as the first sentence chunk completes from the reasoning model, it is immediately dispatched to the TTS buffer before the rest of the response finishes generating.

A voice assistant shouldn't just be an invisible background service. I wanted real visual telemetry to know the system's operational state at a glance:

By offloading the HUD animation to WebGL via Three.js in a separate renderer process, the UI maintains 60 FPS without blocking the audio processing thread in Node.js.

The project is currently in open beta. You can check out the live site and download the installer here:

I'd love to hear your thoughts: what features would you want in a desktop voice assistant for coding and system engineering?