Cross-encoder reranking and streaming LLM responses for sub-100ms AI search pipelines

The article is the fourth installment of a series on building a retrieval-augmented AI system, following earlier work on a PostgreSQL 16 storage layer that merges vector and full-text results. It argues that sending all 15 retrieved chunks to a language model causes degraded recall for facts buried mid-prompt, higher token costs, and hallucination risk from imprecise similarity scores. To address this, the author proposes a second-stage precision reranker using a cross-encoder, which jointly processes each query-document pair and captures finer semantic relationships than standard bi-encoder embeddings. The piece outlines a two-stage retrieval design that narrows a large corpus to a few top chunks, deployed on AWS SageMaker with BAAI/bge-reranker-base, and streams generated tokens through Bedrock ConverseStream via server-sent events. Benchmark results from live tests are presented as evidence of improved handling of ambiguous queries.