How to serve an OpenAI-compatible local API with llama.cpp

Posted on Oct 25 • Originally published at dedupedai.com

Your app already talks to the OpenAI SDK. You want to test it against a free local model without rewriting anything. llama.cpp is a dependency-free C/C++ engine for running LLMs and vision language models, and its server exposes OpenAI-compatible chat completions, responses and embeddings routes. Point your client's base URL at it and most code just works.

This guide installs llama.cpp, starts a server with a model pulled straight from Hugging Face, and calls it with curl and the OpenAI Python library.

# macOS / Linux curl -LsSf https://llama.app/install.sh | sh # Windows (PowerShell) irm https://llama.app/install.ps1 | iex Enter fullscreen mode Exit fullscreen mode The quick install pipes a script into your shell. If you would rather not, download pre-built binaries from the releases page, run it with Docker, or build from source. llama.cpp runs on macOS, Linux and Windows, with backends for CUDA, Metal, Vulkan, HIP, SYCL and more.

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF Enter fullscreen mode Exit fullscreen mode -hf downloads a GGUF model directly from Hugging Face. The quant is optional and defaults to Q4_K_M (for example ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M ).

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF Enter fullscreen mode Exit fullscreen mode According to the server README, it listens on 127.0.0.1:9931 by default. Use --host and --port to change that. The built-in web UI is served at the same address, so open http://localhost:9931 in a browser to test the model before writing any code.

Check readiness with the health endpoint. It returns 503 while the model is loading and {"status": "ok"} once it is ready:

curl http://localhost:9931/health Enter fullscreen mode Exit fullscreen mode Call it with curl curl http://localhost:9931/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer no-key" \ -d '{ "model": "gpt-3.5-turbo", "messages": [ {"role": "user", "content": "Write a limerick about python exceptions"} ] }' Enter fullscreen mode Exit fullscreen mode Call it with the OpenAI Python library import openai client = openai. OpenAI ( base_url = " http://localhost:9931/v1 ", api_key = " sk-no-key-required " ) completion = client. chat. completions. create ( model = " gpt-3.5-turbo ", messages = [{ " role ": " user ", " content ": " Write a limerick about python exceptions " }] ) print ( completion. choices [ 0 ]. message ) Enter fullscreen mode Exit fullscreen mode That is the only change in your app: the base URL. GET /v1/models returns the loaded model.

By default the model id is the path to the model file; set a friendlier one with --alias, for example --alias gpt-4o-mini. The README is careful to say it makes no strong claims of full OpenAI spec compatibility, but in their experience it is enough for many apps.

/v1/chat/completions accepts response_format, both plain {"type": "json_object"} and schema-constrained JSON with {"type": "json_schema", "schema": {...}}. That is handy when your app expects structured output.

docker run -p 8080:8080 -v /path/to/models:/models ghcr.io/ggml-org/llama.cpp:server -m models/7B/ggml-model.gguf -c 512 --host 0.0.0.0 --port 8080 Enter fullscreen mode Exit fullscreen mode There is also a server-cuda image; add --gpus all and --n-gpu-layers 99.

llama.cpp supports 1.5-bit to 8-bit integer quantization and CPU plus GPU hybrid inference, so models larger than your VRAM can still be partially accelerated. Pick a 4-bit or lower GGUF, let layers spill to CPU, and see the performance troubleshooting doc if it is slow. Lower-bit quantization saves memory but can reduce output quality.

The 0.8B example model is fast and great for wiring things up, but it is much weaker than hosted frontier models. Swap in a larger GGUF once your plumbing works.

Get one email every Monday with the week's hottest AI repos and exactly how to use them: subscribe free at dedupedai.com