An Experimental Comparison of Six AI Agent Session Reset Policies

Signup Write New Story An Experimental Comparison of Six AI Agent Session Reset Policies by Evgenii Arsentev by Evgenii Arsentev | @ arsentev PhD, CEO of AskDocDoc (telehealth). Our software is built by AI agents, so I measure what they cost and publish the data with DOIs. arsentev.ai

PhD, CEO of AskDocDoc (telehealth). Our software is built by AI agents, so I measure what they cost and publish the data with DOIs. arsentev.ai

At AskDocDoc (telehealth, I'm the CEO) the software development is done by AI agents. In engineering we have only our CTO, so the agent bill is a real line in my budget and not a hobby number.

I wanted to know one simple thing - when should I clear the agent's context? After every task feels clean. New session, nothing old in the window. But I didn't know what it costs, so I measured it.

Twelve programming tasks, same agent (claude-sonnet-5 in Claude Code), six session policies. Fresh session for every task, then every two tasks, every three, four, six, and one session for all twelve. Each policy I ran six times - 36 runs in total.

The cost I computed from token counts and the list price ($2 per million input, $10 output, cache write $4 at the 1-hour rate, cache read $0.20). So it's modeled cost, not an invoice. Token counts are real though.

What I got is a U-shape. Cheapest was clearing every third task - $1.62 per run on average. Clearing after every task was $2.17 per run, it's 33.5% more (p = 0.002, and 0.011 after Holm correction).

The other side of the U is weaker. One session for all twelve tasks came out 9.7% above the minimum, but after the correction it's not significant. So I don't say "never clear" is bad - only that it wasn't better. From three to six tasks per session numbers stay close (more a plateau than a sharp bottom).

Why the clean option costs more - it's the cache. Every new session loads the context again (system prompt, project files, instructions) and this load is a cache write. On this price list a cache write is 20 times more expensive than a cache read.

In my runs cache writes went from about 253k tokens to 88k as sessions got longer. At the same time context per call grew from about 38k to 71k tokens (longer session, bigger window, more to re-read on every call). So short sessions pay a lot of writes and long ones pay for re-reading a fat context. For my tasks they met around three tasks per session.

It matches my bigger log study too. I looked at 722 sessions of my own agents (150,902 model calls) and more than 85% of modeled cost was work with context - cache reads plus cache writes. Writing new code is a small part of the bill.

And they are real. I picked "every three tasks" as a comparison point after the runs, not before with six conditions that matters (Holm helps but doesn't fully fix it).

The agent wrote its own tests inside each task. Minimum was set, maximum wasn't. In the "after every task" branch it wrote more tests - so part of the extra cost can be extra work and not only extra context. All tests passed in all runs, but that doesn't prove the conditions did equal work. I had to correct my own report on this.

And it's twelve tasks, one model, one setup. Temperature wasn't recorded. One careful measurement - not a rule for everyone.

I don't clear after every small task. I clear between unrelated tasks (new feature, other part of the code, different kind of job). If the next task is close to the last one, I keep the session because the cache is warm.

I keep my instruction file short, under 200 lines. It's loaded at every fresh start, so a long file makes every cache write bigger. For each line I ask: if I remove it, will the agent start making mistakes? If no, it goes.

Heavy reading I give to sub-agents. They read big logs or docs in their own window and bring back a short answer, and the main session stays small.

The data is open. Run data is on Hugging Face ( DOI 10.57967/hf/10366 ) and the report with the method is on Zenodo ( DOI 10.5281/zenodo.22759217, version 1.2). It's a technical report, not peer-reviewed. If you run the same comparison on your tasks and get a different curve, I'd really like to see it.