Write your own search engine
What you end up with: a search engine you can point at a folder of documents, that indexes them, ranks results the way Elasticsearch’s BM25 does, supports phrase queries, and answers over HTTP.
Search is the most useful thing you can build once you understand indexes. It is also a topic where the theory and the code fit together unusually well: every ranking change is measurable against a real document set.
What you’ll build
Section titled “What you’ll build”- Tokenisation and normalisation (case folding, Unicode, stop words)
- Stemming, and why “running” should match “run”
- An inverted index stored on disk, with postings lists
- BM25 scoring, with the parameters you can tune
- Phrase queries and boolean filters
- The same index served over HTTP with snapshot isolation for readers
- An evaluation harness: precision, recall, and the queries your ranking gets wrong
Articles
Section titled “Articles”- What we’re building: a search enginethe index, the ranker, the API
- An in-memory inverted indexfrom term to document ids
- Tokenising and normalising textcasing, stemming, and stop words
- Persisting the index to diska format you can load back
- What TF-IDF actually measuresand the document length bias it hides
- Ranking with BM25the two constants, and what they do
- Phrase and boolean queriespositions in the index, AND/OR/NOT
- Fuzzy matching and typo toleranceedit distance on the term dictionary
- An HTTP query APIserving results as JSON
- Incremental indexingadding documents without a full rebuild
- Benchmarking against a real corpuslatency, recall, and honest comparison
- What Lucene does that we didn’tsegments, compression, ranking models
Before you start
Section titled “Before you start”Reading order matters more here: finish Collections and strings and Closures and iterators first — most of this track is data wrangling before it is search.