GitHub - AyushSuri8/nexus-search-engine: Distributed search engine implementing BM25, HNSW vector search, LSM storage, Bloom filters, and W-TinyLFU caching.

Modern search engines combine multiple retrieval techniques: lexical search (BM25), semantic vector search, caching, and ranking.

I wanted to understand how these components interact, so I implemented a miniature search pipeline from scratch.

Key parts:

• Bloom filter to skip zero-result queries • LSM-tree backed inverted index • HNSW graph for semantic vector search • W-TinyLFU admission-aware caching • Reciprocal Rank Fusion to merge rankings

One interesting optimization was using skip pointers in the posting lists to reduce intersection complexity from O(n*m) to roughly O(n * sqrt(m)).

Another was using deterministic N-gram embeddings to avoid external embedding APIs.

Full writeup + code: https://github.com/AyushSuri8/nexus-search-engine