I ran ripgrep, Serena, claude-context and my own tool on the same repo

There are a lot of ways to let a coding agent find code now. Plain grep, a language server behind MCP, a vector database, a local index. Every README says its way is the right one, so I wanted to see them do the same job on the same code.

I build one of these tools, Semlith, so treat my bias accordingly. That's also why I'm showing the questions it got wrong.

The setup is Flask 3.1.3 on an M1 MacBook Air with 8 GB. Versions are ripgrep 15.2.0, Serena 2.1.0, claude-context 0.1.15 (with Milvus 3.0.3 and Ollama running nomic-embed-text) and semlith 0.50.0.

git clone --depth 1 --branch 3.1.3 https://github.com/pallets/flask && cd flask Enter fullscreen mode Exit fullscreen mode The four approaches in one table ripgrep Serena claude-context Semlith What it is Text search, no index Language servers behind MCP Embeddings in Milvus Local index, embeddings plus keyword plus code graph Finds code by meaning No No Yes Yes Edits code No Yes, by symbol No No, read only Setup on Flask Nothing 30.9 s to index About 21 min (CPU in Docker) 84.3 s to index Moving parts One binary Python env plus language servers Docker, Milvus, Ollama, npx One binary Reads PDF, Word, Excel No No No Yes The rest of this post is how I got those numbers.

When the agent already knows the identifier, nothing beats grep.

rg -n max_content_length src Enter fullscreen mode Exit fullscreen mode src/flask/wrappers.py:55: _max_content_length: int | None = None src/flask/wrappers.py:60: def max_content_length(self) -> int | None: src/flask/wrappers.py:80: if self._max_content_length is not None: src/flask/wrappers.py:81: return self._max_content_length src/flask/wrappers.py:84: return super().max_content_length src/flask/wrappers.py:88: @max_content_length.setter src/flask/wrappers.py:89: def max_content_length(self, value: int | None) -> None: src/flask/wrappers.py:90: self._max_content_length = value Enter fullscreen mode Exit fullscreen mode Eight lines, all of them right, in milliseconds.

Now ask the same thing the way a person would, without the name. An agent turns "limit how big an uploaded request body can be" into words and greps for them.

rg -n -i "upload|big" src Enter fullscreen mode Exit fullscreen mode src/flask/cli.py:297: a bigger role. Typically it's created automatically by the src/flask/helpers.py:542: @app.route("/uploads/<path:name>") src/flask/helpers.py:545: app.config['UPLOAD_FOLDER'], name, as_attachment=True src/flask/helpers.py:549: files or uploads. Uses:func:`~werkzeug.security.safe_join` to Enter fullscreen mode Exit fullscreen mode None of these is the answer, because wrappers.py never says "upload" or "big". This is where an agent starts guessing more words and opening more files.

Serena gives the agent symbol tools backed by real language servers, over 40 languages. Since version 2 it shows up as a REPL, so the agent writes a little Python against Serena's API.

uv venv -p 3.11 venv-serena uv pip install -p venv-serena "serena-agent @ git+https://github.com/oraios/serena@v2.1.0" serena project index. Enter fullscreen mode Exit fullscreen mode Indexing took 30.9 seconds. Given the name, it's precise.

s. lsp. find_symbol ( " max_content_length " ) # [{"name_path": "Request/max_content_length", "kind": "Method", # "relative_path": "src/flask/wrappers.py", "body_location": {"start_line": 87,...}}] Enter fullscreen mode Exit fullscreen mode References come back in about 2 seconds and they come from the language server, so they are real references, not text matches. Line numbers count from zero, which caught me out the first time.

Without the name though, Serena falls back to a regex pattern search, and that returns the same lines grep did.

s. fs. search_for_pattern ( " upload|big ", relative_path = " src " ) Enter fullscreen mode Exit fullscreen mode The thing Serena does that none of the others do is edit. Replace a function body, insert after a symbol, rename across files. If your agent spends most of its time changing code it already found, that's a real advantage.

claude-context is Zilliz's MCP server. It chunks code, embeds it and stores the vectors in Milvus, with hybrid search on top. By default it wants an OpenAI key for embeddings and a Zilliz Cloud key for the database. It also supports a fully local setup, so that's what I ran.

curl -sSL https://github.com/milvus-io/milvus/releases/download/v3.0.3/milvus-standalone-docker-compose.yaml -o docker-compose.yaml docker compose up -d docker run -d --name cc-ollama --cpus = 4 -p 127.0.0.1:11434:11434 ollama/ollama:0.40.13 docker exec cc-ollama ollama pull nomic-embed-text EMBEDDING_PROVIDER = Ollama EMBEDDING_MODEL = nomic-embed-text \ OLLAMA_HOST = http://127.0.0.1:11434 MILVUS_ADDRESS = localhost:19530 \ npx -y @zilliz/claude-context-mcp@0.1.15 Enter fullscreen mode Exit fullscreen mode The images came to 6.14 GB and the first npx start pulled 418 MB of packages. While it worked I watched its connections, and it only talked to Milvus and Ollama on the machine.

Two things bit me. On the first run Milvus exited mid-index while Ollama had every CPU, so I capped Ollama at four. The second one is worth knowing if you use it. get_indexing_status said this at 92 seconds.

✅ Codebase '…/flask' is fully indexed and ready for search. 📊 Statistics: 100 files, 100 chunks Enter fullscreen mode Exit fullscreen mode It wasn't done. The server's own log finished at 1,266.9 seconds with 86 files and 1,837 chunks. If your agent trusts that status and searches early, it searches a partial index without knowing.

That 21 minutes isn't a fair speed test either. Docker on a Mac has no GPU access, so Ollama embedded on the CPU. A native Ollama install would be quicker, I just didn't measure it.

semlith index. Enter fullscreen mode Exit fullscreen mode indexed 211 files (2778 chunks, 5 images) in 84.3s — 0 already indexed, 4 skipped (4 empty), 0 removed Enter fullscreen mode Exit fullscreen mode That count includes the.rst docs, which claude-context skips by default. Serena was quicker to set up here, since it doesn't embed anything.

Setup time on the same checkout. Serena was the quickest of the indexed tools, and claude-context's time is CPU-only embedding in Docker.

semlith search -k 5 "limit how big an uploaded request body can be" Enter fullscreen mode Exit fullscreen mode 1. 0.087 vf src/flask/wrappers.py:59-71 2. 0.083 vf src/flask/wrappers.py:91-106 3. 0.077 v docs/web-security.rst:26-38 4. 0.076 vf docs/patterns/fileuploads.rst:135-153 5. 0.072 vf src/flask/wrappers.py:105-123 5 hits in 360.027667ms Enter fullscreen mode Exit fullscreen mode The first hit is the property. The letters after the score say how it was found, v for the vector list and f for the keyword list.

I asked all of them where Flask decides which error handler runs for an exception. The answer is _find_error_handler in src/flask/sansio/app.py.

Grep did best. rg -n -i "error handler" src lands on line 826 of that file, the function's docstring, because the question uses the same words the code does.

Semlith's top five were all in docs/errorhandling.rst. Those pages explain the order Flask uses, so they answer the question, but they aren't the code. claude-context had the function at rank 8.

So "semantic search beats grep" isn't true in general. When the question already uses the code's own words, grep is very hard to beat.

Three questions on one repo prove nothing, so here's what I could measure at scale.

On SWE-bench Verified, 500 real GitHub issues, the issue text is the query and the gold is every file the fix edits. Semlith had a right file in its top 10 for 76.8% of them, against 47.0% for one ripgrep pass ranked and read in order. That's retrieval only, not whether an agent then fixes the bug.

On 20 issues from SWE-bench Lite, every tool got the same checkout and issue text. Serena 1.7.0 finished 16 of 20 (4 timed out) and had the right file in its top 5 for 25.0% of those, against 56.2% for Semlith on the same 16. That run predates Serena's REPL, so 2.1.0 might do differently. claude-context isn't in that comparison yet, so I won't give it a number.

And the one that keeps me honest. On 50 questions about a single unfamiliar repo, headless Claude Code on Opus scored 92% with grep alone and 92% with Semlith installed beside it. A strong agent with room to explore gets there with grep.

Every MCP server sends its tool definitions at session start, and the agent pays for them on every turn. I counted the tools/list result at about four bytes a token.

Serena tools: 2 4544 bytes, about 1136 tokens semlith tools: 8 2668 bytes, about 667 tokens Enter fullscreen mode Exit fullscreen mode Both are small. Serena keeps the list short with a REPL, then the agent reads 9,292 characters of instructions when it calls initial_instructions.

Honestly they do different jobs, and you can run more than one.

The full runs with every command are on semlith.com/docs/compare and the benchmark harness is in the repo at https://github.com/semlith/semlith.

What are you using for code search with your agents right now, and where does it fall over?