How to Actually Build Your Own Local AI Assistant

Posted on Oct 29 • Originally published at garystow.co.uk

A few weeks ago I had nothing, no local AI setup, no idea where to even start, just a genuine question: could I build a proper voice assistant using only free tools. Here's the actual path, assuming you're starting from exactly nowhere.

This is the step everyone skips and it's the one that decides everything else. Local AI models don't run in the cloud, they run on your actual graphics card, and two numbers matter more than anything else.

Open Task Manager, go to Performance, click your GPU, and you'll see your dedicated VRAM figure right there. Then check whether you're on Nvidia or AMD. This matters because almost every tutorial online assumes Nvidia, using something called CUDA. If you're on AMD like me, that path isn't available, but it's not a dead end, it just means a different backend called Vulkan, which works properly and is built into the software you're about to install.

This is the actual engine everything else sits on top of.

This is where people freeze, so here's the actual rule of thumb I use. Stick to models published by the model's original maker, so names like Qwen or openai, or by a group called lmstudio-community, who package models properly for this exact use case. Before downloading, check for a green badge saying full GPU offload possible, that confirms it'll actually fit on your card. And on quantisation, the technical-sounding number and letters after the model name, Q4_K_M is the standard, sensible default, a solid balance of speed, quality, and file size, don't overthink it beyond that at the start.

On my own machine right now I've got a small model for quick general chat, a 14B all-rounder, a dedicated coding model, and a larger 20B model that's become my actual daily driver, all running through this same setup. Start with something smaller than that, get it working end to end, then upgrade once you trust the process.

Right now your model only talks to you inside LM Studio's own chat window. To let something like a voice assistant actually reach it, you need the Local Server.

Worth knowing honestly, if you switch between two different models regularly like I do, one for chat and one for code, only one properly fits in memory at a time on a 16GB card, so there's a genuine ten to thirty second wait the first time you switch. Not a bug, just physics.

Inside LM Studio there's a System Prompt field, this is a set of instructions the model reads before every single reply, and it's how a generic model becomes something that actually feels like your assistant rather than a chatbot. Mine's built in two modes, one for casual back and forth with real personality, and a completely separate one that drops that personality entirely and writes in my own voice instead whenever it's producing something meant to go out under my name. Worth deciding early which of those you actually need.

Start simple. Ask it to introduce itself, see if the personality you set actually comes through. Ask it something you already know the answer to, to confirm it's genuinely working. Then, and this is the one that matters most for trust, ask it something it has no way of actually knowing, and watch what it does. If it confidently makes something up rather than telling you it doesn't know, that's the very first thing worth fixing before you build anything else on top of it.

On LinkedIn I showed a live demo of this in action, worth a look if you want to see how it actually responds in real time.

Everything so far still needs you to type. Getting it to genuinely listen, the way Wade does, needs something called a wake word, a tool called openWakeWord handles the actual detection. Here's a real snag worth knowing before you get your hopes up, there's no pre-trained wake phrase for whatever name you've picked for your own assistant. You get a short, fixed list, Alexa, Hey Mycroft, Hey Jarvis, that's genuinely it unless you train your own from scratch. Mine ended up answering to Hey Jarvis, still makes me laugh given the name I actually gave him.

Worth knowing too, once it's wired up, if wake detection feels unreliable, check your actual microphone before you touch a line of code. Mine looked completely normal in every settings menu and was quietly too quiet to be picked up properly, swapping to a proper USB headset mic fixed detection completely, no amount of code tuning would have solved it.

Faster-Whisper handles turning your actual speech into text, running fully on your own machine, no cloud involved. Piper handles the reverse, turning replies back into a spoken voice, also entirely local. Worth setting expectations honestly here, free local voices have a real ceiling on how expressive and natural they sound compared to paid cloud options, it's a genuine trade-off for staying completely free, not a flaw you've caused.

And this is honestly where I have to stop holding your hand.

Everyone's setup from here genuinely differs, your hardware, your microphone, your exact file paths, your chosen wake phrase, enough that writing out generic code steps would do you more harm than good, you'd end up debugging my assumptions instead of building your own thing. This bit takes real trial and error, proper coding, and patience with things breaking in specific, findable ways rather than a big leap forward. I'm genuinely still learning as I go here myself, so I can't pretend I've got a clean, universal path from this point through to a finished voice loop.

If you've built something like this yourself, or you're further along than me, I'd genuinely love to hear how you approached it. If you're just getting started, you're already closer than you think. Good luck.

If getting a build like this actually found by anyone matters to you once it's live, I wrote separately about what makes a site readable to search engines and AI crawlers, worth a look once you're at the publishing stage.

Built something similar, or stuck on a step? Email me at garystow.hq@gmail.com — I read everything myself.

I write about practical AI for marketers and building with free tools at garystow.co.uk. This piece first appeared there.