Run a Capable LLM on Your Own Machine

Running a model locally sounds like a weekend project. It's actually ten minutes. This is the copy-paste version: pick a tool, pull a model, serve it over an OpenAI-compatible endpoint, and make your first tool call, with no hand-waving.

Treat this as a starting point for getting a model running, not a fine-tuning guide. The aim is a working endpoint in ten minutes; the deeper performance knobs (sampler settings, speculative decoding, GPU offload splits) are their own rabbit hole, and a separate post.

Why local, when hosted models are right there and genuinely good? Privacy is the obvious one: nothing leaves the machine. Then cost, because a long or repetitive job stops metering you by the token. And control, which matters more than it sounds. No rate limits, no refusal out of nowhere, and the version stays pinned instead of shifting under you one morning. You pay for all of it in quality and speed, and that price is real. The last section is about when it's worth paying.

Three tools cover almost everyone. Pick by how you work:

I'll use LM Studio's CLI ( lms ) as the main path, because it gives you a clean OpenAI-compatible server and stays out of your way on model management, then show the Ollama and llama.cpp equivalents.

Before pulling anything, do the memory math. A quantized model's file size is roughly its RAM/VRAM footprint, plus headroom for the context (KV cache).

Rule of thumb: model file size + ~20% ≤ your available RAM/VRAM.

Quantization (Q4/Q5/Q8) trades quality for size. Q4_K_M is the usual sweet spot; go Q8 only if you have the memory to spare and want the extra fidelity.

Install LM Studio, then the CLI ships with it. List what you have, pull a model from the UI or CLI, and load it:

# what's available locally lms ls # load a model, with an explicit context length for headroom lms load <model-name> --context-length 16384 --yes # what's loaded right now lms ps # free the memory when you're done lms unload <model-name> Enter fullscreen mode Exit fullscreen mode LM Studio serves an OpenAI-compatible API (default http://localhost:1234/v1 ), and that compatibility is the whole point: anything that speaks the OpenAI API (your scripts, LiteLLM, an agent framework) points at the endpoint with a dummy key and works unchanged.

# confirm the server and see loaded models curl -s http://localhost:1234/v1/models | jq '.data[].id' Enter fullscreen mode Exit fullscreen mode A first chat completion:

curl -s http://localhost:1234/v1/chat/completions \ -H "Authorization: Bearer lm-studio" \ -H "Content-Type: application/json" \ -d '{ "model": "<model-name>", "messages": [{"role":"user","content":"Say hi in five words."}] }' | jq -r '.choices.message.content' Enter fullscreen mode Exit fullscreen mode The same thing in Ollama # pull and run (first run downloads) ollama run llama3.1:8b "Say hi in five words." # serve the API (runs on:11434) ollama serve Enter fullscreen mode Exit fullscreen mode Ollama also exposes an OpenAI-compatible endpoint at http://localhost:11434/v1, so the same curl as above works with the port swapped.

For headless servers or maximum control:

# build once (see the repo for your platform), then serve a GGUF file llama-server -m./models/your-model.Q4_K_M.gguf \ -c 16384 --host 0.0.0.0 --port 8080 Enter fullscreen mode Exit fullscreen mode llama-server also speaks the OpenAI API ( /v1/chat/completions ). One thing to watch on long sessions: the KV cache grows with context and RSS climbs over time; if latency degrades, restart the server to reset it.

If you want the model to do things (call functions, drive tools), you need tool-calling, and local models vary wildly here. Send the OpenAI tools array and see if it responds with a structured tool_calls:

curl -s http://localhost:1234/v1/chat/completions \ -H "Authorization: Bearer lm-studio" -H "Content-Type: application/json" \ -d '{ "model": "<model-name>", "messages": [{"role":"user","content":"What time is it in UTC?"}], "tools": [{ "type":"function", "function":{ "name":"get_time", "description":"Return the current time for a timezone", "parameters":{"type":"object","properties":{"tz":{"type":"string"}}} } }], "tool_choice":"auto" }' | jq '.choices.message.tool_calls' Enter fullscreen mode Exit fullscreen mode If you get a clean tool_calls array back, the model can drive tools. If you get prose describing the call instead, that model (or that quant) isn't reliable for agentic work; pick another, or keep the tool schema small and simple. This is the single biggest gotcha with local models, and it's worth its own article.

Local is not the default answer. Here's how I'd actually decide:

Pick a runner, do the memory math, load a model, hit the endpoint. That part really is ten minutes. The decisions that matter come after: how far you quantize, and whether the model can hold tool-calling, because that is what makes or breaks anything agentic you put on top of it.

Next: why local models fail at tool-calling, and how to design around it.