How I Built a Shared AI Memory System Using Markdown and GitHub

Signup Write New Story How I Built a Shared AI Memory System Using Markdown and GitHub by Geoff Cox by Geoff Cox | @ redgeoff I’m Geoff Cox, a hands-on engineering leader in Seattle. I lead engineering teams, and I still write code.

I’m Geoff Cox, a hands-on engineering leader in Seattle. I lead engineering teams, and I still write code.

For a while, every new AI session started with me re-explaining things: who I am, what I’m working on and which documents matter, often by uploading the same PDFs again. Now any AI I use starts with all of that.

This is a story about how I moved my notes out of Google Drive and into a folder of plain text files that every AI I use can read and update, and about the handful of rules that keep that folder from turning into a mess. I call it my memex.

If you read my last post, you’ve already met it. The bot that reads our kids’ school email keeps its rules there. This post is about the memex itself: what’s in it, how it stays accurate when several AIs are writing to it, and how you could start one.

In short, AIs kept tripping over my documents. When I pointed Claude at my Google Drive and asked it to edit a Google Doc, it would create a new copy instead of changing the one I had. Claude has its own documents that it can edit, but other AIs can’t easily read them. And most AIs couldn’t search my Drive well enough to find things, so I often ended up exporting documents to PDF and uploading them by hand.

I wanted one place that any AI could read, search and change, without me copying things around. A note-taking app wasn’t the answer either, since you’re limited to the structure and features the app gives you.

The name comes from Vannevar Bush’s 1945 essay “As We May Think”, where a memex is a desk that stores a person’s books, records and communications so they can be consulted quickly. Mine is a git repo of markdown files. Markdown is plain text with a little formatting, e.g. a # in front of a heading. A git repo is a folder that records every change, so you can see what changed, when and why, and undo any of it. Mine lives on my laptop and in a private repo on GitHub.

The files are organized by what they’re about:

I created the repo on March 1, 2026 from a template and then mostly forgot about it. The real work started on October 13, 2026. As of October 31, it has 131 notes and about 157,000 words.

On October 13, I downloaded my Google Drive to my laptop and had Claude read it. It extracted the text from about 780 files: PDFs, Word documents, slide decks, spreadsheets and plain text. Google Docs were the one snag. They sync to a laptop as pointer files with no text in them, so I downloaded those folders again as Word and Excel files. Claude then read the converted files too and summarized everything it had read into notes, mostly about my career and past projects, rather than copying the documents in.

Some things I didn’t want in the memex at all, so before anything was summarized we agreed on what to leave out: tax and financial records, anything that looked like it held a password, medical and identity documents, and the folders about our kids. The memex keeps a ledger of the import that lists what was read, what was skipped and why. That way a later session doesn’t redo the work, or worse, import something I had already decided to exclude.

The import surfaced something I hadn’t expected. When you read 25 years of someone’s documents side by side, they disagree with each other. For example, one of my old self-reviews said our hotfixes had dropped from 8 a week to 1.5, while my other records said 7. Rather than pick one, Claude listed every conflict like that and asked me. I answered them in batches that day, and each answer became an entry in a decisions log.

A pile of notes that several AIs write to will drift. One session restates a fact a little differently, another copies an old number from a document, and a month later nobody knows which version is right. So the memex has a few rules, and a program checks most of them on every change.

An example header. The index is built from these lines.

None of this is clever. The rules are simple enough that any AI can follow them, and the checker catches it when one doesn’t.

The checker can tell you a link is broken, but it can’t tell you that two notes say the same thing in different words, or that a plan from three weeks ago is stale. For that, there is a cleanup pass called a “dream”, loosely after the way sleep is thought to consolidate memories.

I didn’t come up with the idea. Anthropic added dreaming to Claude Managed Agents in May 2026: a scheduled process that reviews an agent’s past sessions and memory, pulls out patterns and tidies the memory, either on its own or with you reviewing the changes first. Their Code with Claude talk, “Memory and dreaming for self-learning agents”, covers how they got there. OpenClaw has its own version that runs in light, REM and deep phases and writes a diary of what it changed, and Hermes Agent has a community plugin modeled on OpenClaw’s. Mine borrows from all of them. The difference is that my memory is a git repo, so a dream’s changes arrive as a pull request I can read.

A dream has two halves. The mechanical half runs the checker and rebuilds the index. The reflective half is an AI working through a written checklist: promote anything durable from the daily log into the right note, find facts stated in more than one place and keep one owner for each, flag notes that have gone stale, and write down the questions only I can answer. It lands as a pull request with a short report.

What the AI is allowed to do on its own depends on the change:

There have been four dreams so far: October 13, October 15, October 25 and November 1. The last two were run by a bot on its own weekly schedule. On October 25, lint was already clean, so the main results were notes for the two people mentioned most often who didn’t have one yet (notes on people are private and stay in the memex), and one place where the repo’s instructions contradicted a newer decision. The November 1 dream was lighter: one lint warning fixed, the bot’s own log brought up to date and no questions for me.

More than one AI, which was the whole idea. From October 13 to 31, the memex had 190 commits and 122 merged pull requests. Most came from Claude Code sessions on my laptop. Others came from Grok Bot, the agent I use to run bots on a cloud computer, and from the bots themselves, e.g. the school bot updating its profile and a memex bot that does the weekly upkeep.

Two things fall out of that. First, I’m not tied to any one AI. If I switch tools tomorrow, the notes, the rules and the history come with me, because they’re just files. Second, every session starts with context. I don’t paste my background into a prompt or explain how something works, because the AI reads the index and finds it.

In practice, I start most things with that same memex bot. When I want to add to something or work on it, like a new project idea, a blog post I should write or a book I should read, I ask the memex bot to add the details. It already has the context of many things in my life, both current and old. And when I’m working in another AI and I think my memex would help, I give that AI read and write access to the memex’s GitHub repo, and it has the same context right away. A bot gets its own access token that works only on that one repo, can only read and write files and open pull requests, and expires within 90 days.

Pull requests turned out to be useful for more than catching an AI’s mistakes. When I want feedback on a document, I open it as a pull request and leave comments on specific lines in GitHub, the way you’d review code. The AI answers each comment and pushes a new version. This post was reviewed that way.

I think it’s relatively safe, as long as you don’t have a lot of automations that send out or post content unattended. The only bot of mine that emails anyone sends to one family list, from its own account; the others open pull requests or message me in a private Slack channel. That said, there are risks, and you certainly don’t want to store anything sensitive in your memex, e.g. credit card numbers. The checker flags anything that looks like a password or an access key, but it can’t catch everything, so the better habit is to leave those things out in the first place.

Not much, and I wouldn’t try to design it all up front. Everyone’s preferences differ, and mine weren’t clear until I’d used it for a while. A memex changes shape as you use it, and the goal is that over time it holds all of your context and preferences, written down.

That said, three things would have saved me some cleanup:

You don’t need my scripts to get most of the benefit. A folder of markdown files in a git repo, a short header on every note and an AI that can read files will get you a long way. I’d add the rest in roughly the order it paid off for me:

If you’d like a head start, I’ve put the memex’s structure and tooling on GitHub as a template, with none of my notes in it: memex-template. It has the folders, the note header, the index builder, the checker, a decisions log with an example rule, and the dream protocol with its /dream command. The bot template from my last post lives there too, since the bots deliver into the memex.

How an AI Bot Turned 30 School Emails a Week Into One Daily Digest