The path to AI FDE #01: Anatomy of an LLM app request

Posted on Oct 19 • Originally published at dhanushkumarsivaji.substack.com

I've spent 8+ years building React/TypeScript front ends. Now I'm learning applied AI in public, working toward AI Engineer, then AI Solution Architect, and (I hope) AI Forward Deployed Engineer. Before tokens, RAG or agents, I wanted one map: what actually happens when a user clicks "Summarize"?

Each step below gets its own figure first, then the short version in words.

1. The client sends a request. From the browser's side, an LLM feature is just an HTTP call to your backend: POST /summarize with the user's text, plus a conversation id if it's a chat. OpenAI's docs say never to expose an API key in client-side code, so the key lives on your server and only the server talks to the provider.

2. Your API validates the input. FastAPI checks the body against a Pydantic model. Invalid data never reaches your handler: FastAPI raises a RequestValidationError, and its default handler returns HTTP 422. Once validation passes, Pydantic guarantees the fields match their declared types. It's the cheapest place to stop bad input.

3. You assemble the prompt. The model sees only what you send. Anthropic's Messages API is stateless, so you resend the full history every time. OpenAI also calls each request independent and stateless, and offers optional server-side conversation state, but notes that earlier input tokens in the chain are still billed on each turn. Later in this series, retrieved documents (RAG) join the prompt here.

4. Count tokens and check the budget. Models read tokens, not characters, and tokenizers are tied to the model, so counts differ between models. System prompt, tool definitions, history and the output all share the context window. If the input alone is over the limit, the request fails (Anthropic returns a 400, "prompt is too long"). Estimate tokens before sending and set a maximum output length.

5. Prefill: the model reads all the input at once. Inference has two phases. In prefill, the model processes every input token in parallel. It's compute-heavy, and it's the main model-side driver of time to first token (TTFT). Network time and provider queueing add to that pause.

6. Decode: one token at a time. Then the model generates autoregressively: score every token in the vocabulary, pick one, append it, repeat. A KV cache keeps earlier work so each step doesn't start over, and the per-step pace is the time per output token. It stops at an end-of-sequence token, a stop sequence or the token limit.

How it picks is a sampling setting: greedy, or sampled with temperature, top-k or top-p. Don't assume those settings exist on every model: Anthropic's docs say Claude 4.7 and later return a 400 if you set them to non-default values.

7. Stream the tokens back. OpenAI and Anthropic both stream tokens as server-sent events (SSE) while decoding is still running, so steps 6 and 7 overlap. Two catches from the docs: an error can arrive mid-stream, after a 200 status, and partial output is harder to moderate than a finished response.

8. Validate the output too. Structured outputs constrain the model to a JSON Schema you supply (OpenAI supports a subset), but OpenAI's guide says they can still contain mistakes, and documents two cases to handle: refusals and responses cut off at the token limit. If you stream, validate once the stream completes. FastAPI's response_model is a second check: a mismatch becomes a server error, not wrong data. A valid shape isn't a correct answer; evals cover correctness later in this series.

9. Retry safely, then record. The same status can mean different things: a 429 can be a rate limit, but Anthropic also uses it when you hit your monthly spend cap, so read the error body before retrying. Never retry quota or billing errors. For transient errors, honor Retry-After when present, otherwise use exponential backoff with jitter, and cap attempts and total time. Official SDKs already retry some errors (Anthropic's twice by default), so count that in your limits rather than nesting loops. Then log input/output token usage, latency and the provider's request id. Those records feed debugging, cost tracking and, eventually, evaluation.

As a front-end engineer, I found this oddly familiar: it's a request lifecycle with an unusual backend in the middle. Two of the nine steps are the model. The other seven are contracts, budgets, streams and failure handling, the parts an engineer owns. My working rule from here on: treat the model as an unreliable, metered, remote dependency, and put validation on both sides of it.

Where this fits in my roadmap: this post is the map I'm following into the foundations block: Python typing and async, FastAPI and Pydantic, and LLM mechanics (tokens, context, inference). Next post: Python for AI backends: a TypeScript engineer's refresher.

Over to you: which of these nine steps broke first in the first LLM feature you built?

Sources checked Oct 18, 2026. Provider APIs change quickly. Re-check model-specific details (status codes, sampling support) right before publishing.

Originally published on The Path to AI FDE on Substack. Subscribe for one diagram-led post every weekday: https://dhanushkumarsivaji.substack.com/p/the-path-to-ai-fde-01-anatomy-of