Building Amit.AI, Part 2: The Backend

Posted on Oct 19 • Originally published at amitmondal.in

Part 1 was the browser. This is the other half: a small FastAPI service that answers questions about me, as me, from facts I wrote down.

Same rules as before: no keys, no limits, no secrets, just the interesting decisions.

Here is the whole trip of one question again. This post covers everything from the BACKEND box down:

1. Python, for the TypeScript brain I write Python and TypeScript, and this service sits right on the overlap. If you come from JS/TS, here's the map I'd hand you for the pieces that show up in this project:

FastAPI plus Pydantic is a great fit for this kind of service. A route is a typed function, request validation reads almost like a Zod schema, and async streaming is a generator that yield s chunks. The framework does the rest.

Every AI tutorial wants you to chunk your documents, embed them and store them in a vector DB. I came very close to building all of it.

Then I counted. My whole professional background, as Markdown, comes to about 3.3k tokens. That fits in a prompt with room to spare.

So the "retrieval" step is: read every file at startup, glue them together, and put them in the system prompt on every request. The model sees everything, which makes it more accurate than retrieval would be, because there are no retrieval misses.

knowledge/ ├── profile.md ├── experience.md ├── projects.md ├── skills.md └── education.md Enter fullscreen mode Exit fullscreen mode A few details I'm happy with:

Cutting scope felt like cheating at first. It wasn't. Every component had to earn its place.

The system prompt does the job a lot of code usually does. The knowledge goes first in a delimited block, then the rules. Without quoting it, here is what it covers:

A prompt is a polite request. I'd rather not build security on politeness.

The server runs its own checks before the model is ever called:

None of this is exotic. The point is layering: input checks, then guards, then the prompt, with the model as the last line of defence rather than the only one.

I'm running on free tiers. The thing nobody tells you is that the daily token limit, not the request limit, is what you hit first.

The math was sobering. Each request carries the full knowledge base plus instructions plus history, so it's a few thousand tokens. A free model's daily token allowance divided by that is a surprisingly small number of conversations per day. This is also why Tier 0 matters so much: every FAQ answer served locally is a call I didn't spend.

So I built a small gateway that holds an ordered list of models and tries them in sequence:

async def stream ( self, messages ): for provider in self. providers: if self. is_cooling ( provider ): continue started = False try: async for chunk in provider. stream ( messages ): if not started: started = True yield chunk return except UpstreamError as error: if started: raise # tokens already reached the user, can't switch now self. record_failure ( provider, error ) raise UpstreamError ( " All LLM providers failed or are cooling down " ) Enter fullscreen mode Exit fullscreen mode Three ideas in there:

Repeat questions are common ("what's your stack?"), so there's a small in-memory cache. It's a bounded LRU with an expiry, keyed on the lowercased, whitespace-collapsed question.

The rule that matters: only first-turn questions are cached. Later turns depend on the conversation, so "tell me more" must never get a stale answer from someone else's chat. Only complete answers are stored, and guard replies and errors never are.

One tiny detail I like: the answer is stored before the final "done" event is sent. If a client disconnects right after the last token, the generator can be closed before the next line runs, and the cache write would be lost.

Each request ends with a single structured log line: outcome, which model answered, total latency, time to first token. Visitor IPs are hashed, so I can spot one noisy client without storing who they are.

Which model answered a given request is tracked with a ContextVar. The gateway is shared by every request, so it can't just store that on itself. It's the same problem AsyncLocalStorage solves in Node, so if you've met one, you've met the other.

Still on the list: evals for the answers, tracing, and shared rate limits. Probably the next post.

Go try it. Ask it something about me. Then try to break it. I'd genuinely like to know what you find.