Wiring Android's MediaPipe LLM Inference API to a Streaming UI

Posted on Oct 28 • Originally published at mvpfactory.io

--- title: " Streaming LLM Tokens in Android with Jetpack Compose: Back-Pressure and the Frame Budget" published: true description: " Wire MediaPipe's LLM Inference API to Jetpack Compose using Kotlin Flow back-pressure, correct buffer sizing, and a 16ms recomposition budget that keeps mid-range devices jank-free at 20+ tokens/sec." tags: kotlin, android, mobile, architecture canonical_url: https://mvpfactory.co/blog/streaming-llm-tokens-android-compose --- Enter fullscreen mode Exit fullscreen mode What You Will Build By the end of this tutorial you will have a production-ready pattern for streaming on-device LLM token output into a Compose Text component — without dropped tokens, without jank on mid-range hardware, and without the recomposition storm that trips up most first implementations.

Here is the gotcha that will save you hours: MediaPipe does not give you a suspend function. It gives you a callback on a thread you do not own. The naive bridge will work fine on your Pixel 9 and silently drop frames on your users' Helio G88.

MediaPipe exposes token output like this:

inferenceSession. addQueryChunk ( prompt ) inferenceSession. generateAsync ( { partialResult, done -> onToken ( partialResult, done ) } ) Enter fullscreen mode Exit fullscreen mode Two constraints that matter:

Ignore these two constraints and you get frame drops that only appear under thermal throttling on mid-range hardware. That is exactly the device your users own.

The cleanest bridge is a Channel with a bounded buffer. Here is the minimal setup to get this working:

fun LlmInferenceSession. tokenFlow ( prompt: String ): Flow < String > = channelFlow { val buffer = Channel < String >( capacity = 64, onBufferOverflow = BufferOverflow. SUSPEND ) addQueryChunk ( prompt ) generateAsync { partial, done -> runBlocking { buffer. send ( partial ) } if ( done ) buffer. close () } for ( token in buffer ) send ( token ) }. flowOn ( Dispatchers. Default ) Enter fullscreen mode Exit fullscreen mode A capacity of 64 covers roughly 2–3 seconds of burst at 25 tok/sec before SUSPEND kicks in. runBlocking inside the callback is intentional — it blocks the producer when the consumer falls behind, without dropping tokens. The docs do not mention this, but this is the only safe primitive here.

Let me show you a pattern I use in every project. Compose does not batch state writes from collectAsState the same way it batches layout passes. Each emission can schedule an independent recomposition.

At 25 tokens/sec on a budget device, one recomposition per token misses frames. The fix is chunked emission with a time window:

fun Flow < String >. chunkByTime ( windowMs: Long = 80L ): Flow < String > = flow { val accumulator = StringBuilder () var lastEmit = System. currentTimeMillis () collect { token -> accumulator. append ( token ) val now = System. currentTimeMillis () if ( now - lastEmit >= windowMs ) { emit ( accumulator. toString ()) accumulator. clear () lastEmit = now } } if ( accumulator. isNotEmpty ()) emit ( accumulator. toString ()) } Enter fullscreen mode Exit fullscreen mode An 80ms window caps recompositions at ~12/sec — safely below the budget-device ceiling.

@Composable fun StreamingResponse ( session: LlmInferenceSession, prompt: String ) { val text by remember ( prompt ) { session. tokenFlow ( prompt ). chunkByTime ( windowMs = 80L ). runningReduce { acc, chunk -> acc + chunk } }. collectAsState ( initial = "" ) Text ( text = text, modifier = Modifier. fillMaxWidth ()) } Enter fullscreen mode Exit fullscreen mode runningReduce accumulates the full response string without holding a mutable StringBuilder in composition scope. remember(prompt) restarts the flow cleanly on new prompts, avoiding stale emissions.

Do not use MutableStateFlow to bridge the MediaPipe callback. Only a Channel with BufferOverflow.SUSPEND prevents dropped tokens from a native producer thread.

Tune to your minimum supported device, not your desk Pixel. 80ms is a safe default. Tighten to 33ms only if your floor is mid-range 60Hz and you have profiler data to back it up.

Before shipping, verify in Android Studio's Compose Layout Inspector: skipped frames should be zero during active streaming; recomposition count on Text should be ≤15/sec on your lowest-supported device.

The full pattern is three composable pieces: a Channel -based callback bridge, a time-windowed chunking operator, and a stateless composable that folds chunks with runningReduce. Each piece solves one specific constraint — thread ownership, back-pressure, and frame budget — and they stay decoupled enough to test independently.

Profile on your lowest-supported hardware before shipping. Flagship benchmarks will lie to you.