Building a Personal AI Brain: Qdrant, Embeddings, and the Illusion of Memory

Posted on Oct 16 • Originally published at ayraix.com

Vector databases sound magical until you actually use one. The story of my Qdrant setup, the grocery list that broke cosine similarity, and the domain filters that fixed it.

I was debugging my Qdrant setup on a Saturday afternoon, feeling very clever about my self-hosted AI memory system, when I asked it something reasonable: "What's the status of my recent project follow-ups?" I got back a note I'd stored about needing to pick up eggs, Greek yogurt, and that specific brand of oat milk my wife insists on.

Not a hallucination. Not a bug. A mathematically perfect cosine similarity match. The grocery list had used words like "items," "pending," "need to check," and "follow up" — the exact same vocabulary fingerprint as a project status note. The embedding model found patterns. It found its nearest neighbor in vector space. It returned results with full confidence.

It was correct by the only measure it knew. That's the moment the illusion broke for me — and where this post starts.

thousandsvector points4collection pointsnomic- embed-textembed model768dimensionsThe query "status of recent project follow-ups" landed in the yellow zone. The grocery list was the nearest neighbor — high cosine similarity, zero relevance.

Before getting into what went wrong, here's why I built it: context initialization. When a new session starts, the first thing it does is query the knowledge base for relevant context based on what I'm working on that day. That 5-second query replaces 10 minutes of me re-explaining project history, decisions made, things tried, things abandoned.

Running in my system: Qdrant on the NAS, the nomic-embed-text model for generating embeddings (runs locally on Ollama — no API calls, no cost), two collections: a personal knowledge base with thousands of points of session notes, task outcomes, and project summaries, and a research archive for reading snapshots. Operating budget: modest monthly power beyond hardware already running.

The collection gets populated automatically. After each session, a script runs in the background. It pulls key facts and decisions from the session transcript, formats them into clean structured notes, generates embeddings via nomic-embed-text, and upserts them to Qdrant. No manual curation step. The curation happens upstream — during the session, when I'm explicit about what I decided and why. Garbage in, grocery lists out. Good structured notes in, useful memory out.

Session Endtranscript ready→summarizernode background→Groq extractllama-3.3-70b→chunk384t / 15% overlap→nomic-embed-textOllama · 768d→Qdrant upsert+ domain metadata→knowledge baseQdrant · local

AI doesn't remember. It retrieves. The quality of retrieval determines the quality of the memory illusion.

At thousands of points, it's genuinely useful. I can ask about past architecture decisions and get back the actual reasoning from months ago. That's the memory illusion working as intended.

An embedding model converts text into a list of numbers — a vector — where similar-meaning text produces similar vectors. Vector search finds the stored vectors nearest to your query vector. That's it. It's a nearest-neighbors problem, not memory. The system doesn't "know" your work tasks any more than Google "remembers" your search history — it finds what's mathematically most similar to what you asked.

Vector constellation personal memory brain.

Cosine similarity measures the angle between two vectors — how similar their direction is in high-dimensional space. It's excellent at capturing semantic similarity. It will also confidently return a grocery list when you ask about project follow-ups if those two things happen to live near each other in the model's concept space. Similarity is not relevance. This distinction breaks systems that don't account for it.

Similarity Search Result — The Grocery List IncidentRankDocumentScore1grocery-list-2024-11-15.md0.8472job-applications-november.md0.8313agent-memory-session-notes.md0.8024n8n-job-filter-config.md0.789

When you index documents for vector search, you split them into chunks first. The size of those chunks is one of the most consequential decisions you'll make — and almost nothing online tells you how to actually choose.

Too small: a 64-token chunk might be a single sentence. "The project is on hold." Nearly meaningless when retrieved — on hold because of what? Which project? Who decided? Too large: a 2048-token chunk contains 15 different topics. The embedding tries to represent all of them and ends up representing none of them well. The retrieval signal dilutes across everything crammed into the chunk.

Chunk Size Comparison256-token chunks (15% overlap = ~38 tokens)chunk 1chunk 2chunk 3chunk 44 chunks · tight context · fast retrieval512-token chunks (15% overlap = ~76 tokens)chunk 1chunk 22 chunks · richer context · more topic dilutionThe Goldilocks Range (My Tested Numbers)

For factual notes and project summaries: 256–512 tokens with 10% overlap. For conversational content: 512–1024 tokens. The overlap is not optional — it prevents relevant context from being split across chunk boundaries and lost. My knowledge base collection settled on 384 tokens with 15% overlap after testing with actual queries. Your content type will produce different numbers. Test with your real data, not synthetic examples. There is no universal answer and anyone who gives you one hasn't tested it.

The grocery list incident had a specific root cause: the knowledge base was a single flat namespace. Personal reminders, grocery lists, infrastructure decisions, and project notes all lived in the same vector space and competed on the same similarity metric. A grocery list that used task-tracking language was indistinguishable from a work note at query time.

The fix was adding a domain metadata field to every stored point — values like jobs, infra, personal, research — and rewriting all queries to filter by domain before running the vector search. Hybrid retrieval: metadata filter first, then vector search inside that filtered subset. The false positive rate dropped to near zero.

Before: Query "status of recent project tasks" against the full collection. Results ranked by cosine similarity across all points. Grocery list wins on vocabulary overlap.

After: Query pre-filtered to domain = "jobs", then vector search within that subset. Only job-related notes are in the candidate pool. The grocery list isn't even evaluated. The correct result returns in the top 3 every time.

The lesson isn't that vector search is flawed — it's that semantic similarity across unrelated domains is the wrong retrieval problem to solve. Namespace your data. Filter before you search. The vector similarity does its job well when it's operating on a coherent domain, not a soup of everything you've ever written down.

Before You Build a Vector Memory SystemYou've defined domain or namespace boundaries for your collections — grocery lists and work tasks must never compete for the same queryYou've tested chunk sizes with your actual content type and measured retrieval quality before indexing everythingYou've added metadata fields (domain, date, source) so you can filter before vector search, not only afterYou understand you're building a search index, not memory — and you've designed your storage format to be specific and structuredWhat Most Vector DB Tutorials Miss

They show you happy-path retrieval. "Ask about Paris, get the Paris document." Nobody shows you the grocery list problem — where similarity is high but relevance is zero. They don't cover the metadata filter pattern that fixes it. They don't mention that your embedding model's concept space was learned from internet text, meaning domain-specific vocabulary (SAP transaction codes, internal project names) may not embed well without fine-tuning. They don't tell you that nomic-embed-text outperforms OpenAI's ada-002 on several benchmarks and runs locally for free. Read the paper, not just the tutorial.

Qdrant is excellent software. The self-hosted setup on my NAS runs reliably, costs nothing extra, and has made every session more productive by eliminating the context re-establishment overhead that was silently eating 10-15 minutes of every conversation.

But the grocery list was the most useful thing that happened to this project. It broke my mental model before I'd built a production pipeline on a false assumption — that vector similarity equals relevance. Add a domain field. Filter before you search. Be specific in what you store. Do those three things and the memory illusion works well enough to be genuinely useful.

Don't skip them and you'll get grocery lists. Which, in retrospect, is a pretty good outcome for catching a fundamental design flaw early.

Related Posts AIRAG vs Fine-tuning: The Real Decision AIMulti-Agent Systems

Have you ever encountered a situation where an AI system misunderstood your query or provided incorrect information? What steps do you take to ensure the accuracy and reliability of your AI systems?

No account needed — pick a take, then keep reading. We rotate these prompts so each piece feels like a conversation, not a clone.

Building a personal AI brain with tools like Qdrant and embeddings is an intriguing journey into the intersection of technology and human memory. While these systems can provide impressive results in certain scenarios, they also highlight the limitations and potential pitfalls of relying on machine learning for complex cognitive tasks.

The example of a grocery list being incorrectly identified as a project status note serves as a stark reminder that even sophisticated AI models can struggle with context and relevance. This incident underscores the importance of continuously refining and curating the data used to train these systems, ensuring they align more closely with human intent and understanding.

Thought-provoking Ayraix publisher take titled conceptually 'What we're really seeing'. Challenge the reader. Original paraphrase — no third-party quotes.

Originally published on ayraix.com, practical AI for enterprise builders.