I Tried to Build an Offline AI Assistant for an 8GB Laptop, and the CPU Said No

I kept waiting for the memory to fill up. A question sat there for three seconds, and 13 GB was still free.

I am Savindu Thathsara. I live in London, and this is a Hearthlen note. Hearthlen is an offline AI assistant on a lightweight Linux laptop: you say “hey Hearthlen,” ask something, and a model on that machine answers. Nothing in the question goes to a cloud API. The plan was written for an 8GB laptop. The ThinkPad I am actually building on has 16 GB. That extra room did not save the reply.

The early budget was a pile of RAM. Leave most of an 8 GB machine for the person and the browser. Keep a wake word always on. Borrow one heavy model while a request is in progress, then let it go. One model at a time, because two of them would spend the same memory twice.

That story is about fitting. The failure I hit was about waiting.

On 25 October 2026 the dashboard read 13.1 GB free, 2.4 GB used, 15.5 GB total. The models on disk were 3.5 GB: Qwen2.5-3B at 2.0 GB, a 1.5B coder at 1.0 GB, SmolLM2 at 0.3 GB, and the Lessac voice at 0.1 GB. The language model was not even loaded until someone spoke. The listener, which is the process holding the microphone and the speech model, was 1507 MB. After I dropped a second language and restarted it, that process was 1331 MB, and 13.0 GB was still available. The machine had room. The answer was still late.

The first voice model was SmolLM2, 360 million parameters. It was quick. “How are you?” came back in 437 ms. “What should we do today?” came back in 356 ms, as a picnic. A joke took 346 ms. A later pass, “Hi, how about you?”, took 857 ms to generate and still sounded like a line from a list.

A model that size can finish a short sentence. Give it “I have two hours free and it is raining” and it does not keep the rain. It picks a pleasant errand and stops. I wrote that down in the small model was the stock answer. Speed was fine. The answer was not.

The voice model is now Qwen2.5-3B-Instruct, 4-bit, 2,104,932,768 bytes, running through llama.cpp on the CPU. There is no discrete GPU on this laptop.

I sent the sentence the small model had been failing: “I have two hours free and it is raining. What should I do?” The reply used both facts. It suggested reading a book, because it was raining and there were two free hours. That was 39 tokens in 3017 ms. About thirteen tokens a second. You feel that gap. The face sits on Thinking the whole time, which at least shows the wait is the model and not a frozen window.

The same CPU shows up when the assistant writes a page. Qwen2.5-Coder-1.5B produced a small HTML animation in 8220 ms, inside a turn of 19233 ms. A circle animation took 14347 ms to write, and the whole turn was 26338 ms. A form from one sentence took 48881 ms. None of those runs reported a full memory. They reported a processor chewing through a local model while the person watched.

Speech itself is cheap when the line is short. “How are you?” was transcribed in 373 ms. A short YouTube command was 364 ms. The spoken reply from Piper, including playback of “Hi, Savindu,” was 1939 ms. The long pause is the language model, after the words are already text.

I added Sinhala beside English. The speech model had to carry both, and the 3B model had to answer in Sinhala. One question, “ආයුබෝවන්. ඔයා කොහොමද?”, came back in Sinhala in 9564 ms. The grammar was rough, and nine seconds is too long for a greeting. Memory was still the same 13 GB free. I took Sinhala out. English speech is the smaller English-only model again. The note on that measurement is the slow reply was CPU time.

Taking the second language out removes one bill. It does not make the 3B model instant. A question still waits on those tokens. I would rather wait three seconds for an answer that uses the sentence than get a picnic in a third of a second.

The 8 GB budget is still the target, because this ThinkPad is luckier than the laptop the project is for. Weights of 3.5 GB and a listener over a gigabyte would sit much closer to the edge on 8 GB, and the rule stays: one heavy model at a time, then it leaves. That rule is about room.

The rule the CPU added is about the clock. A local assistant feels intelligent when the model is big enough to hold the sentence, and it feels broken when that model takes long enough that you wonder if it heard you. On this machine those two things arrived together. More RAM was already there. It did not move the 3017 ms.

Next I stay on Qwen2.5-3B for questions and on the small English speech model for the microphone. The budget those numbers still have to fit is an 8GB laptop running YouTube and a local assistant.

Savindu Thathsara builds practical AI and automation in London. This series is the public log of an offline AI operating system he is writing and learning at the same time.