Should our LLM wiki use vector search? We benchmarked it before deciding

Every "add vector search" proposal comes with the same promise: users phrase questions their own way, keyword search misses, and embeddings fix it. A feature request on Synthadoc made that case for us ( issue #411 ):

Synthadoc uses BM25 exclusively. For wikis under ~200 pages, BM25 is fast and precise; at 500+ pages, paraphrase and concept queries increasingly miss without a semantic layer.

It also asked us to turn embeddings on by default. That's a big change for a pip-installed tool (more on the cost below), so before writing any retrieval code we built a benchmark to answer three questions:

Synthadoc is an open-source tool that compiles your sources — PDFs, web pages, notes — into a Markdown wiki (it works nicely as an Obsidian vault), then answers questions from that wiki with citations. Answer quality depends almost entirely on one step: finding the right pages to hand to the LLM.

Today that step is BM25, the classic keyword-ranking function behind most search engines before neural search. It scores pages by how many of the query's words they contain, weighting rare words more. It's fast, needs no model, and is excellent when the user uses the same words as the page.

We also ship an optional [search] vector = true mode. It takes BM25's top 20 results and re-sorts them by embedding similarity - how close the query and the page are in meaning, according to a small embedding model ( BAAI/bge-small-en-v1.5, run locally via fastembed ).

Each approach fails where the other is strong. That's the whole story of this post in one table - but we wanted numbers, not intuition.

Turning embeddings on by default isn't free for a pip-installed tool:

If the gain is small, or only shows up on wikis far bigger than our users run, that cost isn't worth paying by default.

We wanted something realistic, reproducible, cheap to run, and small enough to keep in the main repo.

A wiki-like corpus. We pinned 1,985 English Wikipedia articles from 24 overlapping computing categories - history of computing, AI, cryptography, databases, networking, and so on - each at a fixed revision ID, so the text never changes under us. Overlap matters: paraphrase misses happen when many similar pages compete. Each article is converted to a Synthadoc wiki page (lead plus sections, capped at 1,500 words, roughly the size of a compiled wiki page).

Three kinds of question per page. For 500 randomly sampled pages, an LLM (Claude Sonnet 5.5) wrote three questions that page answers:

That gave 1,430 questions (some paraphrases failed the check twice and were dropped rather than mislabelled).

A human-written cross-check. LLM-written paraphrases might suit embedding models unfairly well, so we also ran 300 real questions from HotpotQA over its ~3,000 Wikipedia paragraphs.

Wiki sizes. We scored the same questions on nested 200, 500, 1,000 and 1,985-page wikis to test the "it only matters past ~200 pages" claim.

Retrieval only, no LLM in the loop. Each setup is the real Synthadoc search code, called the way the query agent calls it. Scoring a run needs no API key and gives the same numbers every time.

Small in git. Only titles + revision IDs, the frozen questions, HotpotQA question IDs and the results are committed — about 0.5 MB. A script downloads and caches the text (both sources are CC BY-SA).

Metrics. Recall@5 - how often the right page is in the top 5 results we hand to the LLM. We also tracked Recall@10/20, MRR and nDCG (how high it ranks) and P95 latency.

Worth sharing, because both would have silently produced a wrong conclusion:

The issue was right, and now we know by how much: every doubling of the wiki costs roughly 7–8 points of paraphrase recall. Even Recall@ 20 for paraphrases falls from 0.80 to 0.65 - a third of those questions never get the right page into the candidate list at all.

Three reasons, all visible in the code once the benchmark pointed at them:

On HotpotQA's human-written questions the re-rank looked great (Recall@5 0.717 → 0.842), but HotpotQA's distractor paragraphs were chosen for word overlap with the question - it's built to be hard for keyword search. On our own kind of content, the picture is very different. This is exactly why we benchmark on our own use case.

It's also the slowest setup: every query reloads every stored embedding from SQLite.

If BM25 and embeddings each fail where the other is strong, don't make one the judge of the other. Our prototype runs both over the whole wiki, takes the top 20 from each, and merges them with reciprocal rank fusion (RRF): every page scores 1 / (60 + rank) in each list it appears in, and the scores are added. No score calibration - a page near the top of either list surfaces, and a page high in both wins.

That fixed paraphrases (Recall@5 0.479 → 0.694) but still cost exact-name lookups (0.994 → 0.936): an exact BM25 hit now shared the vote with pages that were merely similar in meaning.

Rather than guess at weights, we asked the data: when is BM25's top result right? One signal stood out — how far BM25's best score is ahead of its second. When the top score is at least 1.2× the next:

A dominant keyword match almost always means an exact name or term hit, and it rarely happens for paraphrases. So the final rule is simple: when BM25's best score is at least 1.05× its second, keep BM25's top 3 on top; otherwise let RRF decide. (We also tried simply weighting BM25's vote higher. It backfired: BM25's entries then always outrank the embedding ones, and fusion collapses back into keyword search.)

It's the only setup that beats BM25 on every overall measure, including how high the right page ranks (MRR). And the part re-ranking could never do - paraphrase recall within the top 20, i.e. whether the right page reaches the LLM's candidate list at all:

The gain grows with the wiki — paraphrase Recall@5 improves by 9 points at 200 pages, 11 at 500, 14 at 1,000 and 20 at 1,985 — which is exactly the shape the original issue predicted. On HotpotQA's human-written questions, fusion scores 0.815 at top 5 (BM25: 0.717) and 0.970 at top 20 (BM25: 0.920).

The handful of exact-name questions it still misses are mostly ambiguous labels: "A Mathematical Theory of Communication 1948" is cited on several pages about information theory, but only one is labelled correct.

BM25 latency grew from 77 ms at 200 pages to 362 ms at 1,985. Profiling one query on the full wiki:

Synthadoc cached the tokenised pages but rebuilt the index on every search. Caching the built index (per search scope, cleared whenever a page changes) took BM25's P95 from 362 ms to 20 ms at 1,985 pages with identical results - a speed-up every user gets, whether or not they turn on embeddings.

Fusion's P95 was 122–166 ms, and profiling showed why: the search ran BM25, then embedded the query, so the two costs added up. Embedding the query is about 20 ms on a laptop CPU; comparing it with every page embedding in memory takes under 2 ms. Both steps are independent, and the embedding model releases Python's GIL while it runs, so we now start the query embedding in a worker thread before BM25 runs. Same results, and P95 at 1,985 pages dropped from 166 ms to 62 ms (BM25 alone: 15 ms). We also keep the last 256 query embeddings - an embedding depends only on the text and the model, so a cache entry can never go stale. Either way, that is well under 1% of an answer that spends seconds in the LLM.

Our prototype embedded only each page's title, tags and first 512 characters, so the obvious next step was section-level embeddings: one per heading, so content deep in a long page is searchable. On its own, the embedding search got clearly better - paraphrase Recall@5 0.616 → 0.669, and exact-name lookups 0.305 → 0.659.

The more precise section embeddings mostly found the pages BM25 already found. Fusion gains from the two methods disagreeing - each catching what the other misses — and the coarser page embeddings disagreed with BM25 in more useful places. Twenty times the compute for a worse result was an easy call: we ship one embedding per page.

Every threshold above - RRF's k, the 1.05× pin, the top 3 - was chosen on our Wikipedia corpus. A fair test needs data the design has never seen, with questions people wrote and relevance people judged. We added three public retrieval sets:

English held up everywhere. Chinese didn't: recall@5 fell from 0.751 to 0.645. That is what an English-only model does with Chinese text - it turns both the question and the pages into embeddings that mostly reflect noise, and fusion dutifully mixes that noise into the results.

The fix didn't need a new model, just two rules applied automatically:

A mixed wiki gets semantic search for its English pages and keyword search for the rest. With the rules in place, DuRetrieval scores exactly what BM25 scores, and every English result is unchanged:

A multilingual model could turn that Chinese tie into a win; that's a separate experiment, with its own benchmark run.

Then a hand-built test wiki caught one more problem. We made a small mixed wiki - six pages in Chinese (Turing, the transistor, the von Neumann architecture, the origins of the internet, Chinese character input, China's first computers) and three in English (Grace Hopper, Unix, the Linux kernel) - and asked it questions through the real query path. Startup logged exactly what we wanted:

Vector search: embedding 3 page(s)… (6 mostly Chinese/Japanese/Korean page(s) left to keyword search) Enter fullscreen mode Exit fullscreen mode But every Chinese question came back with the English pages. The query agent translates CJK questions into English before searching - a feature built for English wikis asked in Chinese - and it searched only the translation. "Who invented the transistor" can't match a page written in Chinese, so the right page never reached the LLM. The benchmark missed this because it calls search directly, without the agent's translation step.

The fix: search the question as written and its translation, and merge the results. Now each language finds its own pages:

The last row is the honest limit of an English-only model, and the best case for trying a multilingual one.

The benefit for users is simple: ask in your own words and still get the page that answers you, without losing the precision you have today when you search for a name, a product code or an acronym.

Everything above now runs in Synthadoc's real search path, and the benchmark measures the production code, not a prototype - it reproduces the prototype's numbers exactly:

Search wasn't the only thing that had only ever met small wikis. Synthadoc's web UI has a knowledge graph view: every page is a dot, links between pages are lines, and pages are coloured by topic cluster (found with the Louvain community-detection algorithm). With the benchmark wiki at hand we opened it on bigger wikis, then timed it in a headless browser on generated wikis of up to 5,000 pages.

Before: all 5,000 pages and 100,801 links drawn at once. Once the layout settles, after about half a minute of a frozen browser, the clusters show as colour clumps. But none of them has a name, the legend just says "Cluster 0" to "Cluster 23", and the individual pages are too small to pick out.

At 5,000 pages the browser had to animate about 115,000 SVG elements, roughly 100,000 of them links. The clumps are pretty, but they don't tell you what the wiki is about. The server was slow too: building the graph response read and parsed every page's YAML front matter with the pure-Python parser.

The fix is the same idea as fusion: show the reader what they can actually use.

Opening the largest cluster (289 pages) on the 5,000-page wiki takes about 2 seconds, also at 60 fps.

After: the same 5,000 pages as 24 topic clusters. A bubble's size and the number after its name give the cluster's page count. Thicker lines mean more links between two clusters. The list below the graph names every cluster, largest first, and clicking a name opens that cluster. The overview draws about 100 shapes instead of 115,000.

Inside one cluster: 268 pages, drawn with each page's strongest 605 links rather than every link. Clicking a page opens the usual side panel with its type, its total connections (41 here, links to other clusters included) and suggested questions to ask the wiki. "← All clusters" goes back.

The faster YAML parsing helps more than the graph: every feature that reads pages gets it, search included.

Before starting we set a bar for making semantic search the default: paraphrase Recall@5 up by at least 15 points, lexical and identifier down by no more than 1, latency within our existing targets. The shipped build clears it on the full wiki: +19.7 for paraphrases, 0 for lexical, −1.0 for identifiers, 62 ms P95. Identifiers sat exactly on the line, which is why we waited for data the design had never seen. The independent check settled it: better on both untuned English corpora, and — once CJK text is left to BM25 - never worse on Chinese.

So new wikis are now created with vector = "auto":

Existing wikis keep the vector = false already in their config; we never rewrite a user's settings. Changing it to "auto" or true turns semantic search on.

The benchmark lives in the repo under benchmarks/retrieval/:

git clone https://github.com/axoviq-ai/synthadoc.git && cd synthadoc pip install -e ".[vectors]" python -m benchmarks.retrieval fetch # once: download + build the wikis (~30 min) python -m benchmarks.retrieval evaluate --configs bm25,hybrid python -m benchmarks.retrieval evaluate --corpora scifact,nfcorpus,duretrieval --configs bm25,hybrid python -m benchmarks.retrieval charts # redraw the charts in this post Enter fullscreen mode Exit fullscreen mode No API key needed: scoring runs only the search code, and gives the same numbers every time.

If you run a search or RAG feature, our one piece of advice: benchmark retrieval on your own kind of content, split by question type. An average would have told us vector re-ranking was a small loss. The per-type split told us why — and what to build instead.

A special thank you to [Chaire Zhang] ( @clairezhang-228 ) for her invaluable contributions to this post. Her insights on rigorous work on the benchmark validations made this piece possible.

Expand MakeYourAgent MakeYourAgent MakeYourAgent Follow Joined Jul 6, 2026 • Oct 28 Dropdown menu Copy link Hide The 1.05x clear-winner rule is a neat fix, and I like that you showed why simply weighting BM25 higher collapsed fusion back into keyword search before landing on it.

One query shape I'd want in the benchmark before trusting the rule on real wikis is the mixed one, where someone uses an exact name but asks a paraphrased question about it, like "what did we end up deciding about Postgres for the billing thing". In a notes wiki that name shows up on lots of pages, so BM25 can have a clear winner that is just the page mentioning Postgres most often, the rule pins it into the top 3, and the page that actually holds the decision never gets its vote from the embedding side. When I logged real questions against a notes wiki, that shape was more common than pure paraphrases or pure lookups.

The other thing I noticed is that generated paraphrases tend to be cleaner than what people type. Real questions are shorter, half remembered, and often borrow a word from a neighbouring page rather than using no page words at all, which is harder for BM25 than "no words from the title". Mixing a small set of questions collected from actual use into the held out set moved my rankings more than tuning thresholds did.

Did you check how often the clear-winner rule fires on paraphrase questions that happen to contain a name, and whether it helps or hurts there?