When LLM Inference on CPU Beats Waiting for the GPU

mini-vllm-rs is a single-node LLM inference engine I built in Rust on top of Candle for macOS. It can run CPU and Metal GPU backends concurrently, routing whole requests between them. My first request routing benchmark favored sending almost everything to GPU: the fastest policies assigned it 11 of 12 requests.

That left a more useful question unanswered: could selective CPU offload improve completion time compared with serving the entire workload on GPU alone? Separating prefill from decode suggested where to look: requests with warm prefixes, arriving behind a growing GPU backlog.

A follow-up experiment tested that idea. Routing 1 of 8 requests to CPU reduced the burst's completion time by 9.8% compared with forcing all 8 onto GPU. With 48 requests, offloading 6 to CPU increased aggregate output throughput by 12.4%.

The experiments used a quantized Qwen2.5 0.5B model with F32 activations on an Apple M2. For a 512-input/128-output request, isolated backend profiles showed:

A long, uncached prompt exposed the CPU's largest disadvantage. Even when GPU already had work, avoiding CPU prefill could justify waiting for it. A warm prefix could remove most of that cost, leaving the smaller decode disadvantage for GPU backlog to outweigh.

Each backend owns its own scheduler and KV cache, and a request stays on the backend selected for it. I seeded an approximately 512-token prefix into both caches. Each measured request used the same 516-token prompt and generated exactly 128 tokens. Warming both sides gave CPU a chance to avoid expensive prefill.

I compared forced-GPU routing against mixed routing, where the router chose between CPU and Metal using estimated pending and incoming work. Both cases used the same scheduler settings, with up to 4 active requests per backend. Requests arrived 50 ms apart without waiting for earlier responses. Each case started with a fresh server and seeded caches, and ran 3 times.

The router priced prefill and decode separately:

score = remaining prefill tokens / prefill rate + remaining decode-token budget / decode rate Enter fullscreen mode Exit fullscreen mode The totals included existing work plus the incoming request. Decode work was approximated using each existing request's unused max_new_tokens budget and the incoming request's full budget, so the router did not predict actual output lengths or early stopping. The rates came from isolated backend profiles, making this a work-based ranking rather than an exact prediction of batched completion time.

The candidate diagnostic logs show the boundary in run 1:

Estimated routing costs from mixed-routing run 1. All 3 runs made the same choices, with slightly different GPU scores.

Both candidates advertised a 496-token reusable prefix for the incoming 516-token prompt. The idle CPU therefore had only 20 estimated prefill tokens and a 128-token decode budget to account for. Its score stayed at 20/73 + 128/45 = 3.118 seconds.

At request 7, GPU already had 4 active requests and 2 queued requests. Each queued prompt also carried a 496-token cache estimate, leaving 20 prefill tokens apiece. Including the incoming request, GPU's estimated prefill work was therefore 2 × 20 + 20 = 60 tokens. Its decode budget was 256 queued + 496 active + 128 incoming = 880 tokens. The score was 60/2551 + 880/293 = 3.027 seconds, still below CPU's. Occupied GPU slots alone were not enough to justify offload.

About 50 ms later, request 8 reached the router. Request 7 had joined the GPU queue, while the active requests’ published remaining decode budget had fallen by 8 tokens. Including the incoming request, the GPU now had 80 estimated prefill tokens and a combined decode budget of 384 queued + 488 active + 128 incoming = 1,000 tokens. Its score rose to 3.444 seconds, above the idle CPU’s unchanged 3.118 seconds, so the router chose CPU. The warm prefixes kept prefill work small, and the accumulated decode work tipped the decision.

While helping me organize the earlier benchmark data, AI surfaced an accounting bug: queued requests lost their cache discount, inflating GPU prefill work. I decided to fix it and rerun the benchmark. The first CPU assignment moved from request 7 to request 8, and the capacity results here use the corrected runs. Queued requests now retain their routing-time cache estimates, though eviction can make them stale.

Output-length estimation remains unfinished. For comparison, SGLang's scheduler scales remaining token budgets with an adjustable new_token_ratio to estimate future KV-cache demand for running requests, a heuristic rather than an exact output-length prediction.

Run 1 produced the following measurements for request 8 and the burst as a whole:

Whole-burst time starts at the first submission, while request latency starts at that request's submission.

The offloaded request finished well before the mixed burst ended. CPU could start useful work while GPU continued serving its assigned requests, and the entire burst also finished sooner. Across all 3 runs, mean completion time fell from 11.47 to 10.35 seconds, a 9.8% reduction.

Mixed routing assigned 0, 1, and 6 requests to CPU, respectively. Wall time excludes startup and prefix seeding.

The benefit continued under a longer GPU backlog. In a 48-request burst, CPU received just 6 requests in every run. Mean completion time fell from 68.43 to 60.88 seconds, and aggregate output throughput rose by 12.4%.

Cells show routing estimates, not measured latencies.

Latency gains were not uniform. In the 8-request case, median TTFT improved, but p95 TTFT increased by 3.5%. Under the longer 48-request backlog, both median and p95 TTFT improved, alongside median and p95 end-to-end latency. Selective offload helped capacity without guaranteeing a win for every latency statistic.

A GPU concurrency sweep remains the next baseline check, since these gains were measured against the fixed 4-active-request GPU configuration. Sustained arrivals and varied cache-hit rates would test how far the result carries.

The first benchmark suggested where to look: the CPU’s disadvantage was much smaller in decode than in prefill. The follow-up showed that warm prefixes and GPU backlog could make selective CPU offload useful. Backend speed alone did not determine placement: the router also needed to account for the work already waiting and the prefill that cache reuse could avoid. For a broader discussion of the tradeoffs behind heterogeneous execution, see When Does Heterogeneous Inference Pay Off?.

The benchmark report contains the complete aggregate measurements and reproduction details.