Your AI Demo Works. Will Your LLM Architecture Survive 15 Seconds?

Imagine you are building an AI customer support assistant for an e-commerce company.

"Where is my order? It was supposed to arrive yesterday."

Your application retrieves the order status, checks the shipping information, sends the relevant details to an LLM, and returns a helpful answer.

Then a major sale begins. Thousands of customers start using the assistant. The LLM provider becomes slower, some requests receive HTTP 429 responses, and a few requests time out.

Now customers are waiting for answers about their orders. Support tickets begin to pile up.

The model is still capable of generating an answer. Your application is struggling to deliver it reliably.

This is where production LLM architecture matters.

Let's define a simple target: the application should return a response within 15 seconds.

That is an illustrative service objective for this example, not a universal recommendation. The appropriate deadline depends on the user experience and business requirements.

A typical request might follow this path:

Customer | v Support Chat API | v Authentication + Order Lookup | v LLM Gateway | +------> Primary LLM Provider | +------> Approved Fallback | v Validate Answer | v Return to Customer Enter fullscreen mode Exit fullscreen mode Notice that the LLM is not responsible for looking up the order by itself.

Your backend should retrieve authoritative order and shipping information from the relevant systems. The model can explain those facts in natural language, but it should not invent an order status or guess a delivery date.

This separation is the first important design decision.

Suppose the request takes this much time:

This request succeeds within the 15-second budget.

Now imagine the LLM takes 12 seconds and returns a rate-limit error. Your application retries, and the second request takes another 8 seconds.

The customer has now waited more than 20 seconds.

The lesson is simple: setting a timeout on each API call is not the same as setting a deadline for the entire business operation.

Start with a deadline for the complete request.

const REQUEST_BUDGET_MS = 15 _000; function createDeadline () { return Date. now () + REQUEST_BUDGET_MS; } function remainingTimeMs ( deadline: number ) { return Math. max ( 0, deadline - Date. now ()); } Enter fullscreen mode Exit fullscreen mode When the order lookup finishes, calculate the remaining time before calling the model.

const deadline = createDeadline (); const order = await getOrderForCustomer ( customerId, orderId ); const remaining = remainingTimeMs ( deadline ); if ( remaining <= 0 ) { throw new Error ( " Request deadline exceeded " ); } const answer = await generateSupportAnswer ({ order, timeoutMs: remaining }); Enter fullscreen mode Exit fullscreen mode The generateSupportAnswer function must pass timeoutMs to the actual SDK or HTTP client. The snippet illustrates the design, not a complete implementation.

In a real application, you should also account for authentication, database timeouts, cancellation, and any other work performed before or after the model call.

Important: a timeout means your application stopped waiting. It does not always mean the provider stopped processing the request.

That distinction becomes especially important when a request can trigger a real-world action.

During the sale, your provider starts returning HTTP 429 responses.

A tempting implementation is to retry every failed request immediately.

That can make the situation worse. If 1,000 requests fail and each retries three times, the provider could receive up to 3,000 additional attempts from that batch.

function retryDelayMs ( attempt: number ): number { const baseMs = 250; const maxMs = 2 _000; const cap = Math. min ( maxMs, baseMs * 2 ** attempt ); return Math. random () * cap; } Enter fullscreen mode Exit fullscreen mode This function calculates a randomized backoff delay. It is only one part of a retry policy.

For example, if only 700 milliseconds remain and the next retry would require a longer delay plus a model response, the application should stop trying.

It is better to return a controlled response than to keep a customer waiting for a deadline the application has already missed.

What should happen when the primary LLM provider is unavailable?

For this support assistant, there are two reasonable paths.

This can work when the alternative model has been evaluated for the task and meets your latency, quality, privacy, and cost requirements.

Path B: Return a deterministic response.

If no suitable model is available, the backend can return a simple message based on the verified order data:

"Your order is currently marked as in transit. We cannot generate a detailed explanation right now. You can check the latest tracking status here."

This response may be less conversational, but it is based on real data and still helps the customer.

The application should never fabricate a delivery date just to make the conversation feel complete.

Fallbacks should be designed around the business outcome, not around the assumption that another model will always work.

The backend must verify that the authenticated customer is authorized to access that order before including any order information in the model context.

Authenticated Customer | v Authorize Order Access | v Fetch Verified Order Data | v Send Minimum Required Context | v Generate Answer | v Validate and Return Enter fullscreen mode Exit fullscreen mode Do not let the model decide whether the customer has permission to view an order. That decision belongs to your application's authorization layer.

Likewise, if the assistant can cancel orders or issue refunds, those actions should use explicit backend authorization, business rules, and appropriate confirmation steps. The LLM can help interpret the customer's intent, but it should not bypass the controls that protect the transaction.

During a large sale, request volume can increase sharply.

Suppose the assistant sends the customer's entire conversation history and a large amount of order documentation with every request. Token usage can grow even when the actual question is simple.

For "Where is my order?", the model probably needs only the relevant order status, tracking information, and a small amount of conversational context.

Track the cost per successfully resolved support interaction, not just the average cost of an LLM request.

A request that is cheap but repeatedly fails may still increase total support costs if customers abandon the assistant and contact a human agent.

A dashboard showing successful HTTP responses is not enough.

Imagine that the model returns HTTP 200 for 99% of requests, but many answers are too slow or fail to address the customer's question.

The infrastructure may appear healthy while the support experience is deteriorating.

For answer quality, evaluate whether the response matches the verified order data. For business impact, measure whether the customer receives a useful answer or still needs a support agent.

Avoid logging sensitive order details or personal information unnecessarily. Use request IDs for tracing, and apply appropriate access controls and retention rules to logs.

Check that the application stops retrying when the budget is exhausted, falls back only when allowed, and never exposes another customer's order information.

If the assistant can initiate transactions, test the case where the provider times out after the backend may already have completed an action. A retry must not accidentally cancel an order twice or create duplicate refunds.

Before releasing this assistant to real customers, I would want these controls in place:

You do not need to build a complex microservice architecture immediately. A well-structured module inside your existing backend can enforce many of these controls. Extract it into a separate gateway when operational needs justify the added complexity.

The real test of an AI support assistant is not whether it can answer a question when everything is working.

It is whether the business can still serve the customer when the model is slow, the provider is rate-limiting requests, or the request budget is nearly exhausted.

The LLM generates the language. Your architecture determines whether the feature is reliable enough to use.

If your primary LLM provider became unavailable during your busiest hour, would your application still be able to help the customer?