How One Course Team Spent Weeks Training Their AI Helper to Admit It Doesn't Know

An online course platform integrated an AI assistant to answer student questions, finding the initial build took only a weekend to set up. Problems emerged quickly when the AI confidently fabricated a nonexistent feature in response to a question outside its training material. The team spent weeks refining three core challenges: limiting the AI's knowledge scope to authoritative sources only, training it to refuse out-of-scope questions rather than guess, and routing students to human support when needed. They built a structured test suite of out-of-scope and near-miss questions to catch regressions every time the prompt or source material changed. Key lessons included explicitly defining the AI's scope in the system prompt, tagging sources by authority level, and providing a concrete refusal template rather than vague instructions to avoid hallucination.