I Built an Android AI Assistant That Doesn't Need the Cloud

Running language models locally on Android, keeping conversations on-device, and exploring what happens when AI doesn't depend on a server.

Most AI applications have something in common.

You type a message, send it somewhere, wait for a server to process it, and receive a response.

It's convenient. It's also an architecture we've become so accustomed to that we rarely question it.

What if the model didn't need to run on someone else's infrastructure?

What if your phone could handle the inference itself?

That question is what led me to build Snor, an Android application designed around local AI inference.

I wanted to experiment with something different from the typical AI chatbot architecture.

Instead of making a remote API the center of the application, I wanted the phone itself to be capable of running language models.

That introduces an entirely different set of engineering problems.

You're no longer just building a chat interface around an API.

The interesting part is that all these constraints become part of the application design.

Snor is built around local model inference.

One of the technologies involved is llama.cpp, which makes it possible to run compatible quantized language models using GGUF files.

The application also explores LiteRT-LM for on-device inference.

This means model execution can happen directly on the device rather than requiring every prompt to be sent to an external AI provider.

For smaller devices, this distinction matters.

A model that works comfortably on a desktop might be completely impractical on a phone with limited available memory.

That's why model selection, quantization, and resource management are important parts of the project.

One of the more interesting directions I've been exploring is lightweight models.

Instead of assuming that bigger models are always better, I've been focusing on the SmolLM2 family, particularly:

These models aren't intended to compete with large frontier models.

Their value is in exploring what can be accomplished with relatively small models under mobile hardware constraints.

Smaller models have limited reasoning capabilities, and they aren't suitable for every task. They can also produce incorrect or incomplete answers.

But that limitation is part of the engineering problem.

A useful local AI application shouldn't pretend that a tiny model can do everything.

It should manage expectations, handle output carefully, and acknowledge when it cannot provide a reliable answer.

One of the main reasons for building Snor around local inference is data control.

When inference happens locally, conversations don't need to be transmitted to a remote model provider.

Privacy shouldn't just be a marketing statement added to an application description. It should influence how the software is architected.

Local inference provides a different relationship between the user, the application, and the model.

The difficult work happens underneath it.

Running native inference engines inside a Flutter application requires communication between Dart and native code.

Snor uses native integration to connect its application layer with the inference engines.

This creates additional responsibilities around model lifecycle management, memory handling, and safe execution.

Native inference isn't something you want to manage casually.

Loading a model, generating a response, clearing context, and disposing of native resources need predictable execution.

Concurrent operations can cause serious problems if the underlying native resources aren't protected.

This is why synchronization and controlled model lifecycle management are important parts of the implementation.

Android devices don't have unlimited memory or desktop-class cooling systems.

Model size, available RAM, context length, and inference settings all affect the experience.

A mobile AI application needs to account for those constraints rather than assuming every device behaves the same way.

A smaller model shouldn't be given an unlimited conversation history and expected to perform like a much larger model.

Snor explores bounded conversation history, model-specific context limits, and output constraints to keep interactions manageable.

The goal is to build a more predictable experience around the capabilities of the model actually running.

I don't think local AI and cloud AI need to be competing philosophies.

Local models offer more control over data and can work without an internet connection.

Cloud models offer access to significantly larger models and more demanding capabilities.

Snor explores a local-first approach while allowing optional cloud integrations.

The important distinction is that cloud usage should be a deliberate choice rather than an invisible requirement.

Snor is an ongoing experiment in making local AI practical on Android.

Some areas I'm interested in improving include:

I find this space particularly interesting because mobile hardware keeps evolving, while efficient inference techniques and smaller models continue to make local AI more practical.

If you're interested in running language models locally on Android, you can explore the project here:

I'm particularly interested in hearing from developers experimenting with llama.cpp, GGUF models, Flutter, and on-device inference.

The bigger question I'm exploring is simple:

How much useful AI can we actually fit inside a phone, without depending on someone else's server?