Why Chatbot Testing Needs Source Assertions, Not Just Readable Answers

Most teams approve chatbots after a few informal questions, a method that fails to catch subtle retrieval errors where answers sound correct but address the wrong topic. A more rigorous approach requires a fixed test suite where each case asserts that the response contains a specific marker — such as a page path or distinctive phrase — tied to the correct source. Developers building a Persian-language assistant found that a question about enterprise training was answered with accurate but entirely unrelated project-timeline content, a failure invisible without source-level checks. A practical test list should cover multiple languages, spelling variants, gibberish inputs, and explicit negative cases confirming that services not offered are never falsely promised. Rate limiting must also be accounted for, as back-to-back requests can return empty responses that mimic a broken service when the system is actually healthy.