The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev
Agent search changes the retrieval workload: models issue long, iterative queries and inspect literal matches. BM25 remains useful, but tuning, harness design, and end-to-end evaluation matter.
Agentic search is retrieval inside an agent loop, combining a tool-capable model, a harness, and a retrieval engine. BrowseComp+ has **830 questions** over roughly **105,000 documents**, with agents repeatedly querying and reformulating rather than issuing one search.
Treat BM25 as a tunable primitive, not a fixed baseline. Long model-written queries, literal matching, filesystem workspaces, and progressive disclosure can give coding agents inspectable results they can search further with familiar tools.
Agentic search is retrieval inside an agent loop, combining a tool-capable model, a harness, and a retrieval engine. BrowseComp+ has **830 questions** over roughly **105,000 documents**, with agents repeatedly querying and reformulating rather than issuing one search. Treat BM25 as a tunable primitive, not a fixed baseline. Long model-written queries, literal matching, filesystem workspaces, and progressive disclosure can give coding agents inspectable results they can search further with familiar tools. The talk does not establish that BM25 always beats embeddings. Implementations and its **two hyperparameters** can materially change results, while single-query ranking metrics miss whether the full agent loop completes its task.
This narrows retrieval evaluation from one-shot ranking to whether an inspectable agent loop completes its task. It restores tuned BM25, literal matching, and filesystem search as serious primitives for long model-written queries, without establishing that they universally outperform embeddings or other context strategies.