What metrics matter for LLM inference observability (TTFT, TPOT, ITL)?

Track time to first token (TTFT), time per output token (TPOT), and inter-token latency (ITL) together, because each exposes a different aspect of inference effectiveness. TTFT reflects queueing and prompt processing, TPOT summarizes streaming speed, and ITL reveals stutter inside the stream.

TTFT lands on the prefill side, since it includes queue wait and prompt processing. TPOT and ITL land on decode.

With N output tokens, the math ties them together:

TPOT = (E2E latency - TTFT) / (N - 1) E2E latency ≈ network overhead + TTFT + (TPOT × N) Enter fullscreen mode Exit fullscreen mode On long responses, TPOT dominates total time even when the first token is fast.

ITL measures the gap between two consecutive tokens, so it catches jitter that average TPOT smooths over. Our inference benchmarking guide lists common causes: large prefills monopolizing compute, PagedAttention overhead, and poor hardware topology—such as cross-NUMA traffic.

Averages hide the slow tail. Track P95 and P99 for each metric, then define goodput: the share of requests that meet every target at once, such as a TTFT ceiling, an ITL ceiling, and an end-to-end budget. Set thresholds per use case.

Definitions vary. NVIDIA's benchmarking docs, for example, treat ITL as the average gap between tokens, also called TPOT, and warn that tools differ on whether TTFT is included. Confirm what each tool calculates before comparing numbers. On the DigitalOcean Inference Engine, Serverless Inference metrics cover TTFT, end-to-end latency, error rates, and per-request cost with token counts. You can derive TPOT from those with the formula shared earlier.

What is disaggregated prefill and decode in LLM inference?

It runs prompt processing (prefill, which is compute-bound) and token generation (decode, which is memory-bound) on separate workers. The KV cache must move between them, and depending on the implementation, that transfer can delay TTFT or the second token’s ITL. On the DigitalOcean Inference Engine, Serverless Inference metrics report TTFT and end-to-end latency, so you can check whether an architecture change affects first-token time.

What is the difference between TPOT and ITL?

TPOT is the average time per token after the first token, calculated across the entire response. ITL, in the stricter sense, is the gap between two specific consecutive tokens, so it exposes stutter that a healthy TPOT hides. You can derive it from TTFT and end-to-end latency with TPOT = (E2E latency - TTFT) / (N - 1).

Why track P95 and P99 instead of averages?

Averages hide outliers. P95 and P99 show what your slowest requests experience, which is what you compare against latency targets and use to define goodput. Serverless Inference metrics on DigitalOcean cover TTFT, end-to-end latency, error rates, and per-request cost with token counts, so you can set those targets against both speed and spend.