Phi-3-mini on Linux ARM: Run LLMs Locally with MLC LLM
Running large language models often feels like a cloud-only affair, requiring powerful GPUs and substantial budgets. But what if you want to run a capable LLM like Microsoft's Phi-3-mini on a low-power Linux ARM device, say, a Raspberry Pi 5 or an ARM laptop, without relying on cloud APIs or a beef
Running large language models often feels like a cloud-only affair, requiring powerful GPUs and substantial budgets. But what if you want to run a capable LLM like Microsoft's Phi-3-mini on a low-power Linux ARM device, say, a Raspberry Pi 5 or an ARM laptop, without relying on cloud APIs or a beefy discrete GPU? This is where projects like MLC LLM come into play, offering a path to efficient local inference.
What You'll Be Running and What You Need We're going to set up and run Phi-3-mini-4k-instruct, a compact yet powerful model from Microsoft, directly on your ARM-based Linux machine. Phi-3-mini is notable for its small size (3.8 billion parameters) and strong performance for its scale, making it ideal for edge deployments. To make this happen, we'll use MLC LLM (Machine Learning Compilation for Large Language Models), an open-source universal deployment framework that compiles LLMs to run efficiently on various hardware, including ARM CPUs.
First, ensure you have Python and pip ready. For most Linux distributions, you can install these via your package manager. Once that's covered, installing MLC LLM is relatively straightforward using pip. MLC LLM provides pre-built wheels for common architectures, which simplifies the process.
Let's install the mlc_llm and mlc_chat packages:
pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly mlc-chat-nightly Enter fullscreen mode Exit fullscreen mode This command installs the nightly builds, which often include the latest model support and optimizations. After installation, the mlc_chat command-line interface becomes available, allowing you to download models and start conversations.
With MLC LLM installed, you can now download the Phi-3-mini model. MLC LLM manages models in a structured way, often downloading them into a local mlc-chat-config directory.
To start the chat and automatically download Phi-3-mini-4k-instruct (using the q4f16_1 quantization, which offers a good balance of size and performance), use the following command:
mlc_chat chat Phi-3-mini-4k-instruct-q4f16_1 Enter fullscreen mode Exit fullscreen mode This command will first check for the model. If it's not present, it will download the necessary model weights and compiled library for your ARM architecture. This might take some time depending on your internet connection. Once downloaded, the chat interface will launch in your terminal.
After the model loads, you'll see a prompt similar to a regular chat interface. You can type your query, and Phi-3-mini will respond.
[system] user: Tell me a short story about a brave knight and a dragon. Enter fullscreen mode Exit fullscreen mode Phi-3-mini will then generate a response:
[assistant] In a land of ancient forests and towering peaks, Sir Kaelin, known not for his strength but his unwavering heart, faced the formidable task of confronting Ignis, the fire-breathing dragon. The beast had slumbered for decades, but now its shadow fell upon the kingdom. Kaelin, armed with only a shield bearing his family's crest and a rusty sword, ventured into the dragon's lair.
He found Ignis not in a rage, but guarding a single, unhatched egg. Kaelin, understanding the dragon's sorrow, chose not to fight, but to speak. He promised the kingdom would protect the egg if Ignis would retreat to the highest mountain.
A silent understanding passed between them. Ignis, with a final, mournful glance at the egg, took to the sky, and Kaelin returned, not with a dragon's head, but with peace. Enter fullscreen mode Exit fullscreen mode The exact output will vary, of course, but you'll experience Phi-3-mini generating text directly on your ARM device.
One important limitation here is that while MLC LLM enables CPU inference on ARM, for real-time applications or heavy batch processing, a dedicated GPU is still the gold standard. If your use case demands sub-second response times for complex queries or needs to process many requests concurrently, even a small LLM like Phi-3-mini will likely struggle on a low-power ARM CPU alone. For such scenarios, investing in hardware with GPU acceleration or utilizing cloud-based services becomes more practical.
This article was generated with AI (Google Gemini + web search). Please check important details against the sources above.
Daily AI notes on LinkedIn — Kamal Kishor