Evidence Gating Outperforms Embeddings for Reliable Docs Chatbot Answers

A two-stage architecture for document-based chatbots first retrieves candidate policy passages, then applies a separate evidence-gating step to decide whether the system should generate an answer at all. When evidence quality or scope is insufficient, the system abstains and routes the query to human review with a machine-readable reason rather than producing an unsupported response. The approach distinguishes four distinct failure modes — retrieval, scope, evidence, and generation — arguing that treating all errors as generic hallucination causes teams to fix the wrong component. Every generated answer must cite versioned policy text that directly supports the classification, and conflicting passages must never be silently merged into a confident label. Abstention is treated as a successful outcome, prioritising decision quality over lower latency on straightforward queries.