Engineer Builds Observable RAG System Using Keyword Scoring Instead of Embeddings

A software developer has shared findings from an experiment building an LLM-powered support agent that uses retrieval-augmented generation (RAG) to fetch relevant documents before answering user queries. Rather than relying on embeddings or external APIs, the implementation uses simple keyword overlap scoring to rank articles, making the system fully testable and transparent. The agent accesses knowledge through a defined interface that returns ranked articles with numeric scores, allowing developers to replay any query and inspect exactly what information the model was shown. Four unit tests enforce core retrieval guarantees, including correct ranking, result limits, and empty returns for zero-overlap queries, so any future swap to an embedding-based scorer must honour the same contract. The author argues this approach makes retrieval failures — such as a wrong document surfacing for a query — identifiable and measurable rather than hidden inside an opaque pipeline.