LLM Apps in Python: A Production Stack Without Framework Bloat

Posted on Nov 5 • Originally published at lazar-milicevic.com

The first LLM feature I shipped that actually paid for itself was about 200 lines of Python and one SDK. The one that cost me a weekend of debugging was a framework-based pipeline where I could not tell which layer had swallowed a retry. Since then I have built agents, RAG pipelines and scheduled workers that run unattended, and my default stack has gotten smaller every year. This post is the stack I use now, where I still reach for a framework, and what I do about retries, cost and deployment.

Use the provider SDK directly until you can name the specific problem a framework would solve for you. Most LLM apps are an HTTP call with structured input, a loop over tool calls, and some state. That is a small amount of code, and owning it means every failure is yours to read.

Here is the whole core of an agent, which is the part frameworks like to hide:

import anthropic client = anthropic. Anthropic ( max_retries = 2, timeout = 60.0 ) def run_agent ( messages, tools, tool_impls, max_turns = 8 ): for _ in range ( max_turns ): resp = client. messages. create ( model = MODEL, max_tokens = 2048, tools = tools, messages = messages ) messages. append ({ " role ": " assistant ", " content ": resp. content }) if resp. stop_reason!= " tool_use ": return resp results = [] for block in resp. content: if block. type == " tool_use ": out = tool_impls [ block. name ]( ** block. input ) results. append ({ " type ": " tool_result ", " tool_use_id ": block. id, " content ": str ( out ), }) messages. append ({ " role ": " user ", " content ": results }) raise RuntimeError ( " agent exceeded max_turns " ) Enter fullscreen mode Exit fullscreen mode The max_turns cap is not decoration. An agent without a turn budget is an unbounded spend loop waiting for the day a tool returns something confusing.

Where I do reach for a framework: when I need a durable graph with checkpoints and human-in-the-loop pauses, or when a team of five people needs a shared vocabulary for orchestration. What I avoid is adopting one for the first prototype. The abstraction you pay for on day one is the one you fight on day ninety, when you need to change how a single request is retried or how one tool's output is truncated.

Treat the prompt as a versioned artifact with a typed contract, not a string you edit in place. Most "the prompt worked in testing" failures are really contract failures: the input shape drifted, a field went missing, or the output was parsed with a regex.

from pydantic import BaseModel, Field class SearchArgs ( BaseModel ): query: str = Field ( min_length = 3, max_length = 200 ) limit: int = Field ( default = 5, ge = 1, le = 10 ) def search_docs ( ** raw ): try: args = SearchArgs ( ** raw ) except Exception as e: return f " INVALID_INPUT: { e } " return run_search ( args. query, args. limit ) Enter fullscreen mode Exit fullscreen mode Add RAG as a pipeline, not a feature RAG fails in retrieval far more often than in generation, so build and measure retrieval on its own. If the right chunk is not in the top results, no model will save you. I keep retrieval as a plain function with a clear signature, separate from the LLM call, so I can test it without spending a token.

My production default is Postgres with pgvector, which keeps vectors next to the relational data they describe, plus a hybrid search that fuses keyword and vector results with Reciprocal Rank Fusion (RRF). Pure vector search is bad at exact identifiers, product codes and rare names. Pure keyword search is bad at paraphrase. Fusing them is cheap and fixes both.

The constant in the formula is a smoothing parameter. The original RRF paper (Cormack et al., 2009) introduced the method, and a commonly used default for that constant is 60, which is what I start from and then tune against my own test questions.

def rrf ( rank_lists, k = 60 ): scores = {} for ranks in rank_lists: for pos, doc_id in enumerate ( ranks ): scores [ doc_id ] = scores. get ( doc_id, 0.0 ) + 1.0 / ( k + pos + 1 ) return sorted ( scores, key = scores. get, reverse = True ) Enter fullscreen mode Exit fullscreen mode The gotchas I have actually hit:

The default SDK settings are fine for a script and dangerous for a worker. According to Anthropic's docs, the Python SDK retries connection errors, 408, 409, 429 and 5xx responses twice by default with short exponential backoff, configurable through max_retries. The same docs list a default request timeout of 10 minutes, and a timed-out request raises APITimeoutError and is retried. Put those together and one stuck call can quietly consume 30 minutes of wall-clock time. On a serverless function with a hard ceiling, that is an outage. The SDK reference is at platform.claude.com/docs/en/cli-sdks-libraries/sdks/python.

So I always set an explicit timeout and decide retries per call:

resp = client. with_options ( max_retries = 1, timeout = 45.0 ). messages. create Enter fullscreen mode Exit fullscreen mode Then I add an application-level layer on top, because SDK retries only handle transient transport errors, not logical failures:

For cost, the biggest single lever is prompt caching. Per Anthropic's pricing docs, cache writes cost a premium over the base input price, while cache reads cost only a fraction of it. Their docs describe when each cache duration pays off. The current multipliers and break-even points are on the pricing page and in the prompt caching docs. That arithmetic is why prompt layout matters: long, stable content (system prompt, tool definitions, reference documents) goes first, volatile content goes last, so the prefix is reusable.

Model choice is the second lever. Prices per million tokens differ by roughly an order of magnitude across tiers, and you can compare the current ones on the pricing page linked above. I route by task: a small model for classification, extraction and routing, a larger one only for the steps that need real reasoning. In an agent that makes eight calls per run, moving six of them to the cheap tier changes the monthly bill more than any prompt trick.

One more habit: log token counts and cost per run, per step. You cannot optimize what only shows up as a monthly invoice.

Lambda plus API Gateway scales to zero and suits bursty LLM workloads, but several hard limits shape the design. The AWS documentation lists a maximum invocation duration, payload size caps for synchronous and asynchronous invocations, a memory range with CPU scaling alongside it, and default concurrency and throttle quotas. I do not quote the numbers from memory because they change and differ by region. Check the current values in the Lambda limits documentation (and the equivalent page for your region) before you design around them. One consequence is worth stating plainly: API Gateway's default throttle and Lambda's default concurrency limit are different numbers, so a burst can exceed Lambda capacity unless you raise the concurrency quota.

EventBridge Scheduler plus Lambda is also how I run unattended workers: a cron-like trigger, a short-lived function, state in the database. Nothing idles, so the system costs almost nothing when there is nothing to do.

If I were starting a new LLM application tomorrow, I would do this:

A framework earns a place only after one of these pieces hurts. At that point you will know exactly what you need it to do, which is the best moment to choose one.

If you are weighing a build like this and want a second pair of eyes on the architecture, you can reach me at lazar-milicevic.com/#contact, or keep reading the other engineering notes on the blog.