Rethinking the LLM Knowledge Base: Maybe Just Start with an LLM Wiki?

Posted on Oct 21 • Originally published at shoyataguchi.com

TL;DR: GraphRAG and LLM Wiki both "distill" raw data with an LLM, and they hit similar problems: wrong summaries, stale knowledge, lost sources, and hard evaluation. I could not find a clear answer on how thick the distilled layer should be. If you start small, I would go with an LLM Wiki.

I needed to figure out how to build a knowledge base that keeps growing, so I looked into it again and tried to sort things out. I searched widely for existing products, the main methods, and recent papers, and then deepened my understanding by talking with an AI.

When I think of LLM knowledge bases, I think of two families: GraphRAG and LLM Wiki. I dug into both and found that they have more in common than I expected, so I am leaving these notes. I will go through them in this order: the GraphRAG family, the LLM Wiki family, the problems they share, and which one to choose.

Search itself has also changed. Since around 2025–2026, Agentic Search has spread: instead of searching once, an agent repeats search, read, think, and search again. This is not an evolution of GraphRAG. It is about making search multi-step, and the search underneath can be a plain grep, Hybrid Search, or GraphRAG.

So, does the search underneath need to be GraphRAG? There is actually a paper that evaluates this: " Do We Still Need GraphRAG? " (April 2026). It is an unreviewed arXiv paper and has few citations, but I found the content interesting and the evaluation looked reproducible, so I want to share it. It runs Agentic Search on both GraphRAG and plain RAG and compares them. The gap narrows a lot, but for complex multi-step reasoning (questions that cannot be answered in one search, where you use the first result as a key to connect a second and third related document), GraphRAG keeps an advantage. That was the conclusion.

Both families try to fix the following problems by adding metadata. I left this out above, but GraphRAG also adds metadata to its nodes and edges. For example, Microsoft's GraphRAG gives each relation (edge) a reference to the text chunks it came from (text_unit_ids). Its claims also have a confidence (status) and a period (start_date, end_date) ( official docs ). LightRAG also links entities and relations to the original chunks and files, and added citation support in March 2025.

The worst one is that mistakes persist. If the distilled layer contains wrong data, that error is used as fact every time you search afterward.

So, which one to choose? As I kept researching, I found my own criteria.

First, the scale of the data. I think an LLM Wiki gets unstable in quality and cost, because the work of keeping the whole Wiki consistent grows with the number of documents. Karpathy himself writes that it works well at around 100 sources and a few hundred pages, and that you should add proper search when it gets bigger. Next, how easy it is to iterate on improvements. A Wiki is Markdown, so people can read it and fix it. Then, complexity. GraphRAG is complicated, including building the index and the data representation.

If the goal is to start small and run an improvement loop, I think the LLM Wiki family has the advantage. When the data is large and the relations are complex, I think GraphRAG would need to be considered.

But the improvement loop needs human review. I run an LLM Wiki myself, and even after cutting what I review down to the core, I cannot keep up with reading it. Then, if I tune the prompt toward higher information density, I can no longer understand the content. I end up like a boss who only gives orders and does not answer questions.

One more interesting thing. In August 2026, RAGFlow, an OSS RAG (about 93,000 GitHub stars; Dify has about 159,000, as of October 2026), released a feature called Knowledge Compilation. It can build distilled data in various forms from raw data: Tree and Mindmap, in addition to Graph and Wiki.

It made me think that technology that lowers the cost of combining and switching the form of the distilled layer may also matter.