Cutting doc-search latency by 4× with a two-stage retriever.

Developers searching API documentation were getting semantically plausible but factually wrong answers, and waiting two seconds for them. Pure vector retrieval missed exact identifiers — endpoint names, error codes, parameter keys — which is precisely what developers search for.

One engineer, a fixed monthly inference budget, and a corpus that changes on every upstream release. Any solution had to reindex incrementally and degrade to something useful rather than failing closed.

A two-stage retriever: lexical BM25 and dense KNN run in parallel, fuse with reciprocal rank fusion, then a cross-encoder reranks the top 24 down to 6. The generation step only ever sees six passages, which caps token cost independent of corpus size.

Reranking was the expensive call, so it sits behind a semantic cache keyed on normalized query embeddings. The cache absorbs 61% of traffic. Reindexing is diff-based against upstream release tags rather than a nightly full rebuild.

p50 dropped from 1.4s to 340ms and inference cost fell 38% despite the extra reranking stage. Exact-identifier recall went from 0.52 to 0.91 on a hand-labelled evaluation set of 400 developer queries.