Day 02: An LLM is a stateless function. Everything else is you

Posted on Oct 14 • Originally published at harshakatkam.dev

I pointed the forty-line version of cartographer at a second file and asked the obvious follow-up: how does this one relate to the last one?

My first instinct was that I had broken something. A flag I had dropped, a session id I had forgotten to thread through. I went looking for it for longer than I am comfortable admitting. There is no session id. There was never going to be one. The endpoint had no idea I had called it ninety seconds earlier, and no amount of configuration was going to give it one.

Every piece of memory you have ever seen in an AI product is somebody, somewhere, deciding what goes into an array before a POST. That is the entire mechanism. Once you have seen it you cannot unsee it, and a surprising amount of vocabulary stops being mysterious.

Code for this day: cartographer at day-02

Here is the version I wrote first, and it is worth staring at because it looks correct.

def turn ( messages: list [ dict [ str, str ]], content: str ) -> str: # Broken. Do not ship this. return ask ( messages + [{ " role ": " user ", " content ": content }]) Enter fullscreen mode Exit fullscreen mode A list of messages goes in. A reply comes out. The signature reads like a conversation.

But messages + builds a throwaway list, sends it, and drops it. The caller's list is never touched, so every call starts from the same place. This survives a read-through precisely because the type annotation implies history — you have to notice that nothing ever grows.

The endpoint this series uses is the OpenAI-shaped chat completions endpoint, and on that shape a request is self-contained. Whatever you did not put in the array did not happen.

The fix is three lines and it is the whole idea of the day.

def turn ( messages: list [ dict [ str, str ]], content: str ) -> str: messages. append ({ " role ": " user ", " content ": content }) reply = ask ( messages ) messages. append ({ " role ": " assistant ", " content ": reply }) return reply Enter fullscreen mode Exit fullscreen mode It mutates the list it was handed. That is deliberate: there is one transcript, the caller holds it, and every append happens in one function you can put a breakpoint in.

The second append is the line people drop, and its failure mode is worse than no memory at all. Leave it out and the model receives a list of questions with no answers attached — a transcript in which it apparently never spoke. So it re-answers things it already answered, contradicts a conclusion it reached two turns ago, and generally behaves like a model that is bad at reasoning. You will spend an evening rewriting the system prompt, the standing instructions in the first message, before you find it.

Making both appends the responsibility of one function is not style. It removes the state where only half the exchange got recorded.

Around turn, the rest of src/cartographer/agent.py changes less than you would expect. Yesterday's template goes, replaced by SYSTEM, the system message that opens every list and tells the model it is mapping a repository one file at a time and should refer back to earlier files by name. ask keeps its body but now takes the whole list and sends it as the messages field, so it no longer decides what the model sees. main turns every path on the command line into one turn, then asks a closing question that only a model holding every earlier file can answer: which would you read first? tests/test_agent.py is replaced by a test that swaps ask for a fake and asserts both the order of roles and that the second call saw four messages. Today's dev notes walk through every line.

Because the model is a pure function of that array, the array is the complete state of your program. So write it down:

TRANSCRIPT. write_text ( json. dumps ( messages, indent = 2 )) Enter fullscreen mode Exit fullscreen mode That line ends main, and transcript.json goes into.gitignore, because it holds whole source files and is rebuilt on every run. This is the highest-value five minutes of the day, and almost nobody's tutorial does it. When the output is wrong, the question you actually need answered is not why did the model say that. It is what did the model see. There is exactly one artifact that answers it, and now it is on disk.

Nearly every bad answer I have chased at this stage has been visible in that file within thirty seconds: a file that was never appended, a source read as an empty string because the path was relative to the wrong directory, a system message clobbered by a later assignment. None of those are model problems, and all of them look like model problems until you open the transcript.

It also makes runs replayable. The input is a JSON array; you can re-send it byte for byte, change one message, and send it again.

Now the part that gets expensive. Every call carries the whole array, so you pay for turn one again on turn two, and again on turn three.

If each turn adds roughly T tokens, the chunks of text a model reads and bills by, the n*th request carries about *nT input tokens, and the run as a whole costs T·n(n+1)/2. Ten turns is fifty-five turns' worth of input. Twenty turns is two hundred and ten. The curve is a triangle, and you are billed for its area.

For a chat assistant that is survivable, because a turn is a sentence. For an agent reading source files, each turn is an entire file, so T is enormous and the triangle gets frightening at single-digit n. That is a measurement problem before it is a design problem, which is what tomorrow is about.

There are three honest strategies, and they trade against each other:

For a codebase the third column is clearly right — the repository is still sitting there, and re-reading a file costs a disk read. That is the direction this project goes, and it is why the agent eventually has to be able to fetch things for itself rather than being handed them.

If this was useful, a clap helps other people find it. I am curious about the other side of this one: what is the longest-running agent conversation you have kept alive, and what did you eventually have to throw away to keep it affordable?

All views in this post are my own. The scenarios are illustrative composites, and code examples are not drawn from any specific codebase.