BM25 Full-Text Search for Logs: Why It Beats Simple grep and LIKE Queries

Where LIKE and grep Fall Short

A LIKE '%timeout%' query or a grep timeout command works fine on a small file. At production scale — billions of log rows across weeks of retention — a few problems show up quickly. There's no ranking: every row containing the substring is treated as equally relevant, whether it's the exact error you're chasing or an unrelated debug line that happens to mention "timeout" in a comment field. There's no tokenization: searching for "connection timed out" as a substring won't match "timed out: connection reset", even though a human reading both would call them related. And performance degrades linearly with data volume, because every row has to be scanned character by character unless there's a proper index behind it.

What BM25 Actually Does

BM25 (Best Matching 25) is a ranking function used by full-text search engines to score how relevant a document is to a query, based on term frequency, inverse document frequency, and document length normalization. In practice, that means:

• Rows containing your search terms more often, or containing rarer terms, rank higher.
• Common words that appear in almost every log line contribute less to the score, so they don't drown out the meaningful ones.
• Matches are found via an inverted index, not a full scan — so query time doesn't scale linearly with total data volume the way LIKE does.

This is the same class of technology that powers modern search engines and document search products — applied here to structured and semi-structured log data instead of web pages.

Match Modes You Actually Need

Search Is Even More Powerful Combined With Pattern Clustering

Full-text search answers "show me every row matching this term." Log pattern clustering answers "which template is spiking right now." Used together, the workflow looks like: clustering flags an anomalous pattern, then a phrase or regex search pulls every individual event matching that pattern's distinguishing tokens — giving you both the aggregate signal and the granular evidence in the same investigation. We go deeper on the clustering side in How Drain3 Log Pattern Clustering Cuts Incident Investigation Time.

Why the Storage Engine Matters Here

BM25 scoring is only as fast as the index behind it. A columnar, MPP (massively parallel processing) database like Apache Doris can maintain inverted indexes across distributed nodes and parallelize query execution, which is what makes sub-second search over billions of rows realistic rather than theoretical. Row-oriented databases without native full-text indexing tend to bolt search on as an afterthought, which shows up as query latency once data volume grows past a few million rows.

Conclusion

grep and LIKE queries are fine for small, ad-hoc searches. They stop being fine the moment your log volume and retention window grow to production scale, or the moment you need relevance instead of exact substring matches. BM25-backed full-text search — with proper phrase, prefix, and regex support — is the difference between "search that technically works" and "search that's fast enough to use during an actual incident."