Let's Build a Search Engine

Search is the highest-leverage surface in e-commerce — searchers are ~24% of visitors and ~44% of revenue — and it's also the most mystified. Vendors want you to believe it's magic you must rent; tutorials teach you one tool's configuration; academic material buries the practical core under notation.

Here's the secret this post is built on: the core of a search engine is small, learnable, and buildable in an afternoon. Everything else — and there is a lot of everything else — is layers on that core, and once you've built the core with your own hands, every layer, every vendor pitch, and every architecture decision becomes legible. So we're going to build it. Eight levels, each one a working system that exposes its own limitation, which the next level fixes.

Scope, stated honestly: we're building catalog search — the search box on a store with thousands to millions of products. Not web search: no crawlers, no PageRank, no spam wars. That's a different (harder) problem, and conflating the two is why most "build a search engine" tutorials don't help e-commerce people. Prerequisites: basic Python and SQL. Every code block runs as shown.

Level 0 The Anatomy of Search

Before code, the mental model. Every search engine — from your future afternoon project to Amazon's — is this pipeline:

• Indexing: transform products into a structure that makes search fast. Done ahead of time, updated as the catalog changes.
• Query understanding: fix typos, expand synonyms, detect intent ("red nike shoes under 200" = brand + color + category + price filter).
• Retrieval: from 500k products, find the ~1,000 plausible candidates — optimized for speed and recall.
• Ranking: order those candidates so the best 4 sit on top — optimized for precision, using signals retrieval can't afford.
• Serving: return results + facets in <100ms, thousands of times per minute.

And the reason e-commerce search is its own discipline — the four ways it differs from searching documents:

Level 1 The Naive Approach — SQL LIKE

Every e-commerce platform's first search — and there's no shame in it — is one line:

SELECT * FROM products
WHERE title ILIKE '%wireless earbuds%'
LIMIT 20;sql

It works! Ship it. Seriously — for a 200-product store, this is a rational v1. But run it for a week and the failure modes introduce themselves:

Each failure above is a chapter of this post. Word order and partials → tokenization (Level 3). Ranking → BM25 (Level 4). Typos → Level 5. Performance → the inverted index (Level 3). The naive approach isn't wrong; it's a compressed list of everything search engineering exists to solve.

Level 2 Postgres Full-Text Search

Before building anything custom, exhaust what your database already ships. Postgres full-text search fixes three of Level 1's five failures in ~20 lines and zero new infrastructure:

-- 1. A generated column of processed tokens, weighted by field
ALTER TABLE products ADD COLUMN fts tsvector
  GENERATED ALWAYS AS (
    setweight(to_tsvector('english', coalesce(title, '')), 'A') ||
    setweight(to_tsvector('english', coalesce(brand, '')), 'B') ||
    setweight(to_tsvector('english', coalesce(description, '')), 'C')
  ) STORED;

-- 2. A GIN index — this is an inverted index (Level 3 spoiler)
CREATE INDEX idx_products_fts ON products USING GIN(fts);

-- 3. Search, ranked
SELECT title, ts_rank(fts, q) AS rank
FROM products, websearch_to_tsquery('english', 'wireless earbud') q
WHERE fts @@ q
ORDER BY rank DESC LIMIT 20;postgresql

What just happened: to_tsvector tokenizes and stems ("earbuds" → "earbud", so singular matches plural), the GIN index makes lookups fast at any scale, setweight makes title matches count more than description matches, and ts_rank orders results. Word order fixed, partial matching fixed, performance fixed, basic ranking included. For a surprising number of stores, this is enough — and it lives inside the database you already run, with no sync pipeline.

Where it stops:

• No typo tolerance — "earbudss" still returns nothing (pg_trgm can patch this partially, with effort)
• Crude ranking — ts_rank is not BM25 (no term-saturation or length normalization done properly), and tuning it is painful
• No facets, autocomplete, or synonyms without significant DIY
• Relevance work fights SQL — every improvement becomes a harder query

Level 3 Building Our Own Inverted Index

Now we build the data structure that powers Lucene, Elasticsearch, Meilisearch, Typesense, and that GIN index above — because after this, none of those tools will ever be a black box again.

The idea is the index at the back of a textbook: instead of scanning every product for the query words (Level 1's sin), map each word to the list of products containing it, ahead of time.

# An inverted index in ~40 lines. This is the heart of every search engine.
import re
from collections import defaultdict

def tokenize(text: str) -> list[str]:
    """lowercase, strip punctuation, split. Real engines add stemming here."""
    return re.findall(r"[a-z0-9]+", text.lower())

class InvertedIndex:
    def __init__(self):
        # term -> {doc_id -> term frequency}
        self.postings: dict[str, dict[int, int]] = defaultdict(dict)
        self.doc_len: dict[int, int] = {}
        self.docs: dict[int, dict] = {}

    def add(self, doc_id: int, product: dict):
        # field boosting, the crude-but-effective way:
        # repeat important fields so their terms count more
        text = " ".join([product["title"]] * 3
                        + [product["brand"]] * 2
                        + [product["description"]])
        tokens = tokenize(text)
        self.docs[doc_id] = product
        self.doc_len[doc_id] = len(tokens)
        for tok in tokens:
            self.postings[tok][doc_id] = self.postings[tok].get(doc_id, 0) + 1

    def search_and(self, query: str) -> set[int]:
        """docs containing ALL query terms — intersection of posting lists"""
        terms = tokenize(query)
        if not terms: return set()
        # start from the rarest term — smallest set first = fast intersection
        terms.sort(key=lambda t: len(self.postings.get(t, {})))
        result = set(self.postings.get(terms[0], {}))
        for t in terms[1:]:
            result &= set(self.postings.get(t, {}))
        return resultpython

Three things in that code are the difference between knowing about search and understanding it:

1. The postings structure — term → {doc → frequency}. Query time no longer touches the catalog; it touches only the lists for the query's 2–3 terms. That's why search engines answer in milliseconds over millions of products: the work is proportional to matches, not to catalog size.
2. Field boosting by repetition — indexing the title 3x means a title match carries 3x the frequency. Crude but real: proper engines store per-field indexes and weight at score time (that's Postgres's setweight, Elasticsearch's fields: ["title^3"]), but the principle is identical.
3. Rarest-term-first intersection — a real query-planner trick. "the nike hoodie": intersecting from "nike" (500 docs) beats starting from "the" (every doc).

Missing: word positions (for phrase queries), stemming, and — critically — any notion of which matching product is best. Our AND-search returns a set. For "wireless earbuds" on a real catalog, that's 800 products in arbitrary order. Enter ranking.

Level 4 Real Ranking — TF-IDF to BM25

Ranking answers: given 800 matches, which 20 go on page one? The intuition stack, in order:

1. TF (term frequency): a product mentioning "earbuds" 4 times is more about earbuds than one mentioning it once.
2. IDF (inverse document frequency): matching a rare word ("anc") says more than matching a common one ("black"). Weight each term by how rare it is across the catalog.
3. TF saturation: the 20th occurrence of "earbuds" shouldn't count like the 2nd — spam shouldn't win. TF's value must flatten.
4. Length normalization: a 10-word title with 2 matches is more on-topic than a 500-word description with 2 matches. Normalize by document length.

BM25 is just those four intuitions in one formula — and it has remained the default ranking function of Lucene, Elasticsearch, and essentially all production keyword search for two decades because it nails the trade-offs with two tunable knobs:

# BM25, from scratch, on top of our Level 3 index
import math

def bm25_search(idx: InvertedIndex, query: str,
                k1: float = 1.2, b: float = 0.75, top: int = 20):
    N = len(idx.docs)
    avg_len = sum(idx.doc_len.values()) / N
    scores: dict[int, float] = {}

    for term in tokenize(query):
        posting = idx.postings.get(term)
        if not posting: continue
        df = len(posting)                          # docs containing term
        idf = math.log(1 + (N - df + 0.5) / (df + 0.5))   # rarity weight
        for doc_id, tf in posting.items():
            norm = 1 - b + b * idx.doc_len[doc_id] / avg_len  # length penalty
            score = idf * (tf * (k1 + 1)) / (tf + k1 * norm)  # saturating tf
            scores[doc_id] = scores.get(doc_id, 0) + score

    return sorted(scores.items(), key=lambda x: -x[1])[:top]python

Reading the formula like an engineer:

• idf: rare term → big weight. "anc" beats "black" per match.
• tf·(k1+1)/(tf+k1·norm): as tf grows this approaches a ceiling — that's the saturation. k1 (typ. 1.2–2.0) sets how fast: low k1 = "you matched, fine, more matches barely help" — usually right for short product titles.
• norm: docs longer than average get penalized. b (0–1) sets how hard: b=0.75 for prose; for e-commerce titles, lower b often wins because title length variance is mostly noise (brands with long names shouldn't rank worse).
• Scores sum over terms — this is "OR with ranking": matching more query terms naturally scores higher without the brittle all-terms-required AND of Level 3. Production engines typically use OR + minimum-should-match, exactly this.

Notice also what BM25 doesn't know: that "earbuds" and "headphones" are related, that the query "gift for runner" implies a category, or that one product outsells another 100:1. Those gaps are Levels 5–7. But on keyword queries — the majority of e-commerce traffic — a tuned BM25 over good product data is embarrassingly competitive with anything.

Level 5 E-Commerce Table Stakes

Levels 1–4 built a search engine. This level makes it an e-commerce search engine. Five features, in order of user-visible impact:

5.1 Typo tolerance

10–15% of e-commerce queries contain a misspelling; on mobile, more. The classic approach: for a query term with no (or few) matches, find vocabulary terms within edit distance 1–2 (insertions, deletions, substitutions) and substitute the best candidate:

def edit_distance(a: str, b: str) -> int:
    """classic dynamic programming, O(len(a)*len(b))"""
    dp = list(range(len(b) + 1))
    for i, ca in enumerate(a, 1):
        prev, dp[0] = dp[0], i
        for j, cb in enumerate(b, 1):
            prev, dp[j] = dp[j], min(dp[j] + 1,       # deletion
                                     dp[j-1] + 1,     # insertion
                                     prev + (ca != cb)) # substitution
    return dp[-1]

def correct(idx, term: str, max_dist: int = 2) -> str | None:
    if term in idx.postings or len(term) <= 3:
        return None                      # exists, or too short to trust a fix
    best, best_d = None, max_dist + 1
    for cand in idx.postings:            # brute force: fine to ~100k vocab
        if abs(len(cand) - len(term)) > max_dist: continue
        d = edit_distance(term, cand)
        # tie-break by popularity: prefer the more common correction
        if d < best_d or (d == best_d and best and
                          len(idx.postings[cand]) > len(idx.postings[best])):
            best, best_d = cand, d
    return best if best_d <= max_dist else Nonepython

Production engines use n-gram indexes or Levenshtein automata instead of brute force — same idea, faster candidate generation. The two judgment calls that matter more than the algorithm: don't correct short terms (a 3-letter "correction" is usually a different word — "mat" is not a typo of "mate"), and prefer popular corrections (the customer more likely meant the word 500 products contain). And always show "showing results for nike" — silent correction of a term that was actually a niche brand name erodes trust fast.

5.2 Synonyms & query expansion

"Sneakers" must find products titled "trainers"; "tv" must find "television"; in Gulf markets, "جوال" must find "mobile". No algorithm invents these — it's a curated dictionary applied at query time (expand the query to sneakers OR trainers), fed continuously by your zero-result and reformulation logs — exactly the mining loop from the Search Analytics Playbook. Query-time expansion beats index-time because you can fix a synonym without reindexing the catalog.

5.3 Facets & filters

Facets are counts over the result set's attributes — brand (23), color (8), price ranges — and they're half of e-commerce search UX. Conceptually trivial given structured data:

from collections import Counter

def facets(idx, doc_ids, field: str) -> list:
    return Counter(idx.docs[d][field] for d in doc_ids).most_common(10)

# search "earbuds" → facets(idx, hits, "brand")
# → [("Anker", 34), ("JBL", 28), ("Sony", 21), ...]python

The engineering catch is doing this fast at scale — counting attributes across 50k matches per query is why engines maintain columnar "doc values" structures alongside the inverted index. The product catch is facets are only as good as catalog attributes — if 40% of products lack a "color" attribute, the color filter silently hides them from filtered views. Facet quality is catalog quality (the LLM catalog enrichment pipeline exists substantially for this).

5.4 Autocomplete

Suggestions-as-you-type is a different index: prefix → completions, built from popular queries and product titles, ranked by popularity — a trie or a sorted array with binary search. It must answer in <30ms to feel instant, and it quietly prevents typos and zero-results by steering users onto known-good queries. Highest UX return per line of code in this entire post.

5.5 Business ranking — beyond text relevance

The e-commerce heresy that separates store search from library search: the most textually relevant result is not the best result. Production score is always a blend:

# the production pattern: text relevance × business multipliers
final_score = bm25_score
    * availability_factor      # in stock 1.0 / low stock 0.7 / OOS 0.05
    * popularity_factor        # smoothed CTR / sales rank (log-scaled)
    * rating_factor            # 4.5★ nudged above 3.2★
    * freshness_or_margin      # strategy knobs — used with carepattern

Multiplicative blending (not additive) keeps relevance primary — a popular but irrelevant product can't buy its way into results, but among comparable matches, the sellable one wins. This one change — mostly the availability factor — is routinely worth more conversion than any pure-relevance work, and it's why the ML playbook's ranking features are dominated by behavioral signals. Beware the feedback loop though: ranking by CTR trains the index to keep showing what it already shows (position bias — Level 7 deals with it properly).

🕹️ The Live Demo — Try What We Built

Everything from Levels 3–5, running in your browser right now: an inverted index over a 109-product sample catalog, BM25 scoring (k1=1.2, b=0.6), title/brand field boosts, edit-distance typo correction, and category facets. No server — view source, it's ~150 lines of the same logic you just read.

Things worth noticing as you play: the typo correction announces itself; "watch" ranks smart watches by BM25 with no idea you might have meant a wristwatch (that ambiguity is what query understanding and semantic search address); and the facet counts update per query — that's the Level 5.3 code running on every keystroke.

Level 6 Semantic Search — Embeddings & Hybrid Retrieval

Everything so far matches words. But customers search meanings: "warm jacket for Iceland trip", "gift for 8 year old who loves space", "shoes like Air Force but cheaper". Zero keyword overlap with your catalog; full purchase intent. This — not keyword search done better — is what embeddings are for.

The idea in one paragraph

An embedding model maps text to a vector (~384–1024 numbers) such that similar meanings land close together. Embed every product offline; embed the query at search time; retrieve the nearest product vectors. "Warm jacket for Iceland" lands near parkas and thermal layers because the model — trained on billions of sentences — knows Iceland is cold, without your catalog ever saying so.

# Semantic retrieval in ~20 lines (sentence-transformers + hnswlib)
from sentence_transformers import SentenceTransformer
import hnswlib, numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")   # small, free, runs on CPU

# --- offline: embed the catalog, build the ANN index ---
texts = [f"{p['title']}. {p['brand']}. {p['category']}. {p['description']}"
         for p in products]
vecs = model.encode(texts, normalize_embeddings=True)

ann = hnswlib.Index(space="cosine", dim=vecs.shape[1])
ann.init_index(max_elements=len(vecs), M=16, ef_construction=200)
ann.add_items(vecs, np.arange(len(vecs)))

# --- online: embed the query, fetch neighbors in ~1ms ---
q_vec = model.encode(["warm jacket for iceland trip"], normalize_embeddings=True)
ids, dists = ann.knn_query(q_vec, k=50)python

The ANN index (HNSW — hierarchical navigable small world graphs) is what makes this practical: exact nearest-neighbor over a million 384-dim vectors is too slow per query; HNSW finds ~the same neighbors in ~1ms by navigating a layered graph. It's the same role the inverted index plays for keywords — the clever structure that makes retrieval affordable — and it's inside every vector DB (pgvector, Qdrant, FAISS).

Why you still need keywords: hybrid retrieval

Run semantic search alone and you'll discover its failure mode immediately: search "iPhone 15 Pro Max 256GB" and it happily returns... phones. Similar phones. Wrong phones. Embeddings are geniuses at meaning and idiots at exactness — model numbers, brand names, sizes. Keyword search has the opposite profile. So production systems run both and fuse — and the standard fusion is reciprocal rank fusion (RRF), beautiful because it needs no score calibration between the two systems:

def rrf_fuse(rankings: list[list], k: int = 60) -> list:
    """rankings: e.g. [bm25_doc_ids, semantic_doc_ids] — position is all that matters"""
    scores = {}
    for ranking in rankings:
        for pos, doc in enumerate(ranking):
            scores[doc] = scores.get(doc, 0) + 1 / (k + pos + 1)
    return sorted(scores, key=lambda d: -scores[d])

# hybrid = rrf_fuse([bm25_top100, semantic_top100])
# exact-match queries: BM25 list dominates the fusion
# natural-language queries: the semantic list carries itpython

Honest engineering notes before you deploy this:

• Hybrid solves the tail, not the head. Most queries are short keyword queries where BM25 already wins; semantic earns its keep on the long tail and zero-result recoveries. Measure the mix before sizing the investment (query mining).
• Embedding freshness is an ops problem: new products need embedding on ingestion; model upgrades mean re-embedding everything. Budget the pipeline, not just the model.
• Multilingual matters: in bilingual markets, use a multilingual model (e.g., multilingual-MiniLM / e5) — one embedding space where "جوال" and "mobile" are neighbors is worth a thousand synonym entries.

Level 7 Learning to Rank

One layer remains between our engine and the state of the art: learning the final ordering from behavior. Levels 4–6 rank by hand-designed formulas. But you sit on millions of logged searches where customers showed you the right ranking — what they clicked, carted, and bought. Learning to rank (LTR) turns those logs into the ranking function.

The production pattern is a two-stage cascade, and now every piece of it is familiar:

# The re-ranker: LambdaMART via LightGBM — same recipe as the ML playbook
import lightgbm as lgb

# one row per (query, product) pair from your search logs:
#   features: bm25_score, semantic_sim, ctr_30d, cvr_30d, price_rank,
#             rating, review_count, in_stock, delivery_days, margin_band
#   label:    0 = shown, 1 = clicked, 2 = added to cart, 3 = purchased
#   group:    rows per query (ranking is learned within-query)

ranker = lgb.LGBMRanker(objective="lambdarank", metric="ndcg",
                        ndcg_eval_at=[4, 10], n_estimators=500)
ranker.fit(X, y, group=rows_per_query)python

The three lessons that matter more than the model call:

• Position bias will poison naive training. Items shown at rank 1 get clicked because they're at rank 1; train on raw clicks and the model learns to keep them there. Production systems correct with inverse propensity weighting or randomization slices. This single issue is most of the difficulty of real-world LTR.
• The features are the moat, not the model. LambdaMART is a commodity; 30d smoothed CTR/CVR per (query-category, product), price competitiveness, delivery speed — the feature pipeline is where search teams actually spend their time (training/serving skew, the ML playbook's production section).
• Evaluate with interleaving, then A/B on revenue — offline NDCG gains routinely evaporate online (experimentation playbook, and the golden-set regression testing from the search ops playbook before anything ships).

Buy vs Build — The Feasibility Matrix

You now know what's inside the box, which means you can finally read vendor comparisons without being sold to. Every engine below is "our levels, productionized" — the differences are which levels come built-in, and at what operational cost:

Three reference architectures

What World-Class Search Adds (And What You Shouldn't Attempt)

What do Amazon-class systems have that our 8 levels don't? Fewer things than the mystique suggests — but each is an organization-sized investment:

• Query understanding models: classifiers over every query — category intent, brand detection, attribute extraction ("red nike running shoes under 200" → structured filters), spelling models trained on their own logs rather than edit distance. (The LLM-at-the-edges version of this is now buildable small — see the LLM playbook, section 4.)
• Personalization: your query "milk" ranks your usual brand first. User towers, real-time session features, per-segment rankers. Requires identity, scale, and serious infra.
• Multi-stage cascades: not two stages but four+ — retrieval → light ranker → heavy ranker → policy layer (diversity, fairness, sponsored blending), each with its own latency budget.
• Index freshness at scale: price and stock changes reflected in seconds across millions of SKUs and dozens of replicas — a distributed-systems problem, not a search problem.
• Sponsored search: an auction system living inside organic ranking, with its own relevance floors and incrementality measurement. A separate P&L and a separate post.

The honest feasibility table to end all vendor calls:

The Complete Picture — Recap & Your First 30 Days

The whole post in one map — every level in its place in the pipeline:

And the 30-day plan for a small team that wants real search:

1. Week 1: Stand up Meilisearch (or Typesense) on your catalog. Wire typo tolerance, facets, autocomplete — they're config, not code. Instrument every search event (the KPI stack).
2. Week 2: Fix the data: attribute coverage on your top categories, title quality, the first 50 synonyms from your zero-result log. This will outperform any algorithm change.
3. Week 3: Add availability + popularity boosts from your warehouse. Watch conversion move. Build the golden query set (100 queries) and eyeball it weekly.
4. Week 4: Prototype hybrid on your zero-result tail: embeddings + RRF on queries that currently return nothing — the safest possible surface, since the baseline is an empty page.
5. Then: run the weekly ritual, accumulate click logs, and revisit Level 7 in two quarters when the logs can support it.

Sources & Further Reading:
A Search Engine in 80 Lines of Python • BM25: Complete Interactive Guide • Typesense's Engine Comparison • Meilisearch's Comparison (read both — the biases cancel) • Manning et al., Introduction to Information Retrieval (free) • sentence-transformers • hnswlib