Building My Own Local AI Coding Assistant on an 8GB Celeron Laptop

I've started working on a project I've wanted to experiment with for a while: building my own local AI assistant, with the eventual goal of having it help me with development work.

My development laptop isn't exactly an AI workstation.

No dedicated GPU. No 32 GB or 64 GB of RAM. No monster Ryzen or Core i9 sitting under the desk.

I'm not training a multi-billion-parameter foundation model from scratch.

That would require a completely different level of hardware, data and infrastructure.

Instead, I'm building my own LLM-powered assistant around an existing open-weight model.

The model handles inference, while I'm building the software around it: the interface, conversation handling, developer tools, file access, safety boundaries, prompts, context management and eventually things such as retrieval and tool use.

That distinction is important because the model itself is only one part of an AI application.

«Build a local AI assistant that is actually useful to me as a developer.»

And importantly, build it in such a way that the underlying model can eventually be replaced without rebuilding the entire application.

The Hardware Constraint Is Part of the Project

Most local AI discussions quickly reach hardware specifications that make my laptop quietly hide in the corner.

Large models simply aren't realistic on this machine.

With only 8 GB of system RAM and a 2-core/2-thread Celeron N4000, every decision has to take the available resources into account.

I'm currently targeting the sort of 1B–3B model range that has a realistic chance of running on the hardware.

In fact, I think getting useful local AI inference running on this hardware is an interesting engineering problem in itself.

Instead of throwing more compute at the problem, I have to ask:

How much useful AI can I squeeze out of the computer I already have?

The project uses "llama.cpp" as the local inference runtime.

One of the first major milestones was simply getting from:

My laptop ↓ llama.cpp ↓ quantised GGUF model ↓ response

without sending that prompt to a cloud AI service.

It sounds like a tiny milestone compared with everything I eventually want to build.

There is something rather satisfying about looking at a Celeron N4000 generating tokens locally and thinking:

The machine definitely isn't going to break any inference speed records, but it doesn't need to.

That gave me the foundation for everything else.

Running a model from a terminal is interesting, but I don't just want a terminal demo.

So the architecture is gradually becoming something like:

Frontend ↓ My backend ↓ AI provider layer ↓ llama.cpp ↓ Local model

I don't want the whole application tightly coupled to one particular model or runtime.

The frontend shouldn't care whether the response eventually comes from the tiny local model I'm using today, a better local model on future hardware, or even an optional cloud provider.

The surrounding application is the part I'm really building.

If I replace the model later, those parts shouldn't suddenly become useless.

I've also moved beyond interacting with the model exclusively through the command line.

The project now has the beginnings of its own chat interface, including user and assistant messages, loading and error states, Enter-to-send behaviour and protection against accidental duplicate submissions.

I've also started working on conversation context, so follow-up messages can include previous messages instead of treating every prompt as an entirely new conversation.

«"Okay... this is beginning to behave like an actual application."»

And that transition is probably one of my favourite parts of building it.

The long-term reason I'm particularly interested in this project is development.

So I don't need my local assistant to know absolutely everything.

I'd rather have it become useful at the relatively small jobs I repeatedly encounter.

Inspect these files and propose a small change.

That last one is where things become especially interesting.

I don't want an AI model to have unrestricted access to my filesystem and simply change whatever it wants.

The model's output should be treated as a proposal, not authority.

So I'm experimenting with controlled file access and human approval.

Developer request ↓ AI inspects allowed context ↓ AI proposes a change ↓ Human reviews it ↙ ↘ Reject Approve ↓ ↓ Nothing Apply exact changes proposed edit

I've already reached the point where a coding test can make it through the workflow to the human approval gate.

That's a really important boundary for me.

The goal isn't an AI that gets free rein over my projects.

«"I think this is the change you need. Shall I apply it?"»

That is much closer to the developer tool I actually want.

Small Models Also Force Better Engineering

One thing I'm beginning to appreciate is that weak hardware exposes inefficiencies very quickly.

On a powerful system, an unnecessary extra inference pass might just be mildly annoying.

If a tool error causes the model to reread a file, regenerate context, retry something unnecessarily and go through several inference passes, that becomes expensive very quickly.

«Make the model generate more tokens per second.»

«Make the application ask the model to do less unnecessary work.»

I'm looking at things such as reducing repeated file reads, keeping tool errors concise, avoiding unnecessary context, controlling context size and measuring how many inference passes a coding task actually requires.

Sometimes the best optimisation is simply not asking the LLM to think about something twice.

I'm also putting together a fixed benchmark for the assistant.

The plan is to test the same set of tasks against different configurations, including things such as:

Each test can then record things such as response quality, runtime, tokens per second, tool calls, retries and errors.

Then I can change one variable at a time:

Baseline ↓ Change one thing ↓ Run the same benchmark ↓ Compare quality + performance ↓ Keep it or revert it

That feels much more useful than endlessly changing model parameters until something feels faster.

And on hardware this constrained, I don't necessarily want the fastest configuration anyway.

If configuration A gives me 2 tokens per second but consistently useful coding answers, while configuration B gives me 3 tokens per second and writes nonsense, configuration A wins.

I think that's probably the biggest thing this project is teaching me so far.

When people talk about running LLMs locally, it's easy to focus entirely on parameter counts, GPUs and increasingly enormous models.

But a useful AI tool doesn't necessarily need to be enormous.

It needs to be good enough at the task you're giving it.

My Celeron isn't going to run the same models as an RTX-equipped workstation.

I'm trying to find the point where a small local model plus carefully designed software becomes useful.

And honestly, seeing an 8 GB, 2-core Celeron laptop go from:

"There's no way I'm running useful AI on this."

"Wait... it just inspected that and reached my approval screen."

The immediate work is less glamorous than shouting "AI!" at everything.

I need to improve error recovery, create a proper benchmark baseline, finish the provider abstraction, improve conversation handling and continue tightening the controlled developer workflow.

Later, I can experiment with retrieval, better models, more developer tools and perhaps dedicated coding models when the hardware allows it.

But I'm deliberately not making those requirements for version one.

V1 doesn't need autonomous agents, image generation, voice, enormous context windows or a giant model.

If this little machine can eventually help me debug code, understand files, generate regex, explain errors and safely propose edits to my projects, I'll consider that a success.

And if I eventually move the application onto significantly better hardware?

I can swap out the little model for something considerably more capable while keeping the architecture I've already built.

For now, though, I'm rather enjoying finding out just how far I can push an 8 GB Celeron N4000.

The little CPU that probably never expected "run an AI coding assistant" to appear in its job description. 😅