Choosing an Edge LLM Runtime: Start with Devices and Model Artifacts

Suppose a team wants local question answering on a Linux gateway, then similar capability on an RK3588 device and in a phone app. A tempting plan is to run four runtimes, rank their tokens/s, and standardize on the fastest. That comparison skips an earlier question: can the team's model become an artifact the target runtime can load? It also skips backend support and who will rebuild, package, and release the model when it changes.

These are not four interchangeable model players, and none has a universal speed rank independent of hardware, model, and version.

This guide uses public project documentation to compare delivery contracts, not benchmark results. It makes no claim about decoding speed, time to first token, peak memory, power, or production reliability. Backend lists can change; confirm every route against the version and device you intend to ship.

The first difference is what leaves the build machine.

Use the table to eliminate routes that cannot ship, not to score performance. A model-family name does not make four artifacts equivalent: quantization, tokenizer, context settings, and generation code can differ. If procurement asks for one unchanged model file across all devices, test whether that requirement itself is realistic. A product may need several validated artifacts from one model source, each recorded with its own target and build provenance.

Consider a gateway that answers questions about equipment manuals on site. No NPU has been selected yet. The team knows only that the network may be unavailable, a local API is needed, and the model must work with the retrieval flow. Starting with a supported model and a llama.cpp service can separate functional validation from later resource measurements. Once the hardware is fixed, measure CPU or a supported GPU backend on that target.

If the eventual product requires a Rockchip NPU, the prototype cannot simply be "moved to the NPU." It needs an RKLLM conversion path, driver compatibility check, and answer-consistency tests. Whether that extra chain pays off is a question for the target board, not an inference from a desktop benchmark.

Now consider a team embedding offline summarization in Android and iOS apps. It already has PyTorch training and evaluation, and wants to release model and app versions together. An ExecuTorch minimum path is worth testing: export one supported model, assign ownership for tokenizer and generation code, and run it on target phones with CPU or a chosen delegate. If the team instead plans to build target GPU model libraries and can maintain the TVM/MLC compilation and packaging infrastructure, MLC LLM is another candidate. Both routes create app-size, first-download, offline-start, weight-update, and rollback questions. A support badge for "Android/iOS" does not settle those choices.

Even "the same model" can cease to be a comparable test object after different quantization or operator handling. Export can fail; a backend can fall back to CPU; or a model architecture may have to change. Record each candidate as a chain:

model source/version → conversion/export version → quantization → target backend → Runtime/driver → tokenizer and generation config

A ticket that says only "use RKLLM" cannot identify which layer should be rolled back when an update fails.

For OTA products, attach that chain to the release record. Weights, Runtime, driver, and app interface may not be independently upgradeable. Replay a fixed question set on a small target-device cohort before widening rollout. The point is to make artifact dependencies visible before choosing the runtime.

Before a comparison, define "same conditions": actual device, power and thermal conditions, model source and weights, acceptable quantization differences, tokenizer, prompt, context length, concurrency, and output limit. Then record cold start, time to first token, sustained generation, peak memory, temperature, power, and answer quality. If conversion changes operators or quantization, describe the difference instead of presenting the numbers as a pure runtime race.

A fast demo does not make a production route sound if any of those three checks fails.

Include user experience in the measurement plan. A field manual assistant needs a timely answer with traceable sources. A mobile app needs a controlled first download and useful offline behavior. An industrial terminal may share memory with camera, UI, and networking tasks. Average decode speed cannot stand in for first response, memory peaks, or long-running thermal stability.

If a GGUF runs on the Linux prototype but the product contract specifies a Rockchip NPU, do not promise that llama.cpp will use that NPU merely because it has several general backends. Verify the actual board's backend; if the official Rockchip chain is required, assess RKLLM separately. In the opposite situation — hardware still uncertain, models changing frequently, and no capacity to maintain conversion matrices — binding to RKLLM early can turn model exploration into a firmware compatibility project.

ExecuTorch is a concrete candidate for a mobile team with a PyTorch export and native-app workflow. If the target model cannot export reliably or important operators have no suitable delegate coverage, "mobile-friendly" is not a reason to keep investing. MLC LLM's multi-target compilation can be attractive, but without ownership of compiler, model-library, and weight-package versions it is not a low-maintenance shortcut. The team can validate the product task with a simpler route while hardware and update frequency are still fluid.

The output of this selection should be a short responsibility record: target device, shippable model artifact, intended backend, rollback version set, and owner of conversion and app integration. If a field remains unknown, run a minimal test on one target device and record the unsatisfied condition.

These sources support claims about documented interfaces and delivery paths. They do not establish target-device benchmark results. Before a production decision, recheck exact project versions, model licenses, target OS and backend constraints, and keep failed test cases and rollback outcomes with the release record.