Wiring Android's NNAPI to a Quantized On-Device LLM for Real-Time Function Calling
--- title : " On-Device LLM Function Calling on Android: Schema Injection, Token Budgets, and Fallback Parsing" published : true description : " Wire a quantized on-device LLM to Android's NPU for real-time function calling. Covers schema injection, token budgets, fallback parsing, and 3B vs 7B tra
Posted on Oct 14 • Originally published at mvpfactory.io
--- title: " On-Device LLM Function Calling on Android: Schema Injection, Token Budgets, and Fallback Parsing" published: true description: " Wire a quantized on-device LLM to Android's NPU for real-time function calling. Covers schema injection, token budgets, fallback parsing, and 3B vs 7B tradeoffs." tags: [ kotlin, android, mobile, architecture ] canonical_url: https://mvpfactory.co/blog/android-nnapi-on-device-llm-function-calling --- ## What We Are Building Let me show you a pattern I use in every on-device agent project: structured output generation — function calling — from a quantized LLM running directly on Android's NPU. You will wire together three layers that must all work: a schema-injected prompt prefix, a hard token budget that enforces sub-300ms tool-call latency, and a two-pass fallback parser for when the model drifts.
Get all three right and you have a viable on-device agent tier that keeps sensitive user data off the wire. ## Prerequisites - Android project targeting API 26+ - LiteRT (Google's rebrand of TensorFlow Lite) or MediaPipe LLM Inference API — **do not build on the raw `ANeuralNetworks*` NDK APIs**, which were deprecated in Android 13 (API 33). The patterns here apply to either runtime; where I say "NPU-delegated inference," I mean the hardware path, not the deprecated surface. - A quantized 3B or 7B model (INT4 or INT8) — Llama 2, Gemma, Phi, or Mistral all work - Baseline familiarity with Android inference sessions ## The Architecture Enter fullscreen mode Exit fullscreen mode Prompt Builder → [Schema Prefix + User Turn] ↓ NPU-Delegated Inference (INT4/INT8 quantized weights, via LiteRT) ↓ Token Budget Gate (hard stop at N tokens) ↓ Structured Output Parser → JSON tool call ↓ Fallback Heuristic Parser (regex + key extraction)
### Step 1 — Inject the Schema at the Prompt Prefix The model does not know your tool schema unless you tell it — every time. Inject it as a system-level prefix before the user turn, serialized as compact JSON. The docs do not mention this, but verbose descriptions will eat your latency ceiling before the model generates a single output token.
Keep the schema under 80 tokens. The example below uses Llama 2 chat format. If you are on Gemma, Phi, or Mistral, swap in the correct special tokens for your model family — the structural principle is identical, the delimiters are not interchangeable.
Enter fullscreen mode Exit fullscreen mode kotlin // Llama 2 chat format — replace [INST]/[/INST] with your model's special tokens val schemaPrefix = """ [TOOLS] {"name":"search_products","params":{"query":"string","category":"string|null"}} [/TOOLS] [INST] """.trimIndent()
val fullPrompt = schemaPrefix + userMessage + "\n[/INST]"
### Step 2 — Enforce a Token Budget as a Latency Contract Here is the gotcha that will save you hours. Most teams treat `maxNewTokens` as a model configuration detail. It is a latency contract.
Measure tokens-per-second on your target NPU tier, then compute the budget directly: | Model Size | Quantization | NPU Tier | Tokens/sec | Budget for <300ms | |---|---|---|---|---| | 3B | INT4 | Mid-range NPU | ~40–60 | ~12–18 tokens | | 3B | INT8 | High-end NPU | ~55–80 | ~16–24 tokens | | 7B | INT4 | Flagship NPU | ~25–40 | ~7–12 tokens | | 7B | INT8 | Flagship NPU | ~30–50 | ~9–15 tokens | Enter fullscreen mode Exit fullscreen mode kotlin // Inline the budget math directly — no abstraction needed val tokensPerSec = 50.0 // measure on your target device val maxTokens = (targetLatencyMs / 1000.0 * tokensPerSec).toInt() inferenceSession.generate(prompt, maxNewTokens = maxTokens)