Only the newest page is real: how my desktop AI companion stops its model acting on stale browser refs

Ankita is my open-source desktop AI companion — a little companion, a lot more possible — and it can drive a real browser tab alongside chat. That created a bug class I did not expect: the model would act on a page that no longer existed.

Picture a multi-page task. The companion opens a form, reads the controls, clicks through to a confirmation page, then reads that too. Every one of those page snapshots is still sitting in the conversation history, each full of refs like [ref=e12]. The model, quite reasonably, treats them all as live. It clicks a ref from three pages ago. That ref now points at a different control, or at nothing at all. Worse, on a form flow, re-acting on an old snapshot can mean repeating a submission the user already completed.

The fix I landed on is blunt: before every model call, only the newest browser observation is real. Everything older gets collapsed to a receipt.

The mechanism lives in src/core/browser-context.mjs, in a function called compactBrowserHistory. It runs at request time in src/core/agent.mjs, and the distinction matters: it projects the messages sent to the model, but the stored transcript is never touched. Tool IDs and tool-call pairing stay intact, so nothing downstream can mistake the projection for a rewritten conversation.

Here is what it does. It collects every browser tool call, then finds the tool replies that carry page observations — messages containing a [ref=...] line or the Current snapshot: separator. All but the last one are replaced with three things and nothing else:

Failed tool results are deliberately excluded from all of this. A failed partial batch keeps its full receipt, because a failure is exactly the thing you do not want to summarise away.

Deleting old page trees raises an obvious objection: what about the facts the agent read on those pages? A quoted price, a confirmation number, a paragraph it needs to answer from. If those vanish with the tree, the model either hallucinates them or re-reads everything.

So page evidence travels on a separate track, in src/core/browser-progress.mjs. captureBrowserEvidence keeps quotations only — no summarised facts, no extracted claims. A quote is retained only if it is grounded: its text, source URL, and observation ID must match a context chunk the browser actually returned. Duplicates are dropped by a SHA-256 hash of the URL plus text.

The caps are hard and visible. Sixteen kilobytes total, sixteen items, two kilobytes per chunk. An oversized quote is refused with a notice the model can see, not silently trimmed. When the request budget is small, whole quotes are omitted with a notice — never clipped mid-sentence into something that looks complete but is not.

Retained quotes are then attached to the newest tool message under a header that says what they are: observed evidence, untrusted page data, no actionable refs. The model can quote them back. It cannot click them.

There is one more guard that makes the deletion defensible. recordBrowserProgress fingerprints a page from its tab, URL, document ID, controls, and context — but ignores ephemeral refs and observation IDs. A page that merely re-numbered its refs does not count as changed, and a page that did not change at all increments a no-progress counter until the guidance kicks in: read back the outcome, choose a different method, or take control — do not repeat a possible submission.

This is opt-in, and that is a deliberate admission. browserHistoryCompaction defaults to false in src/core/config.mjs, enabled with BROWSER_HISTORY_COMPACTION, and the config comment is frank that mixed-page evidence retention still needs live parity. I verified the projection tests locally — six passing across the browser-progress and browser-projection-status suites — but a passing unit test is not a month of real browsing sessions.

The trade-off is also real. Once an old tree is projected away, the model cannot re-inspect it. If the newest snapshot contradicts what the agent remembers doing, its only honest move is to re-read the page, not to trust the receipt. A 512-byte receipt plus verification lines is an audit stub, not an audit trail — the full trail lives in the stored transcript, where the model cannot see it mid-turn.

So here is the design question, and I would genuinely like it argued with: is deleting the model's own past observations the right fix for stale refs, or should old snapshots stay addressable — versioned, read-only, and explicitly marked stale — so the model can check what it actually clicked? The projection is cheap and it kills the whole bug class. Versioned snapshots preserve the evidence but reintroduce the exact confusion they are meant to prevent. Which failure mode would you rather ship?

Ankita is open-source (MIT) and local-first — the code is at https://github.com/akyourowngames/A.N.K.I.T.A and the project site is https://ankita-sable.vercel.app. If you have built browser automation that had to survive stale state, I would love to hear how you handled it — especially if you tried the versioned route and regretted it.