How a dev team fixed repeated LLM errors by embedding corrections in the knowledge base

A development team running a support chatbot noticed their LLM consistently gave plausible-sounding but incorrect answers to certain specific user questions, despite broad instructions to 'answer accurately'. The model would draw on general technical knowledge — such as guessing SSH hostname formats or suggesting unnecessary key-conversion steps — producing confident but wrong responses. The team found that abstract, one-time instructions in the system prompt were too vague to prevent these specific recurring errors. Their solution was to embed targeted correction notes directly within the knowledge base, placed right beside the relevant FAQ entries where the model was most likely to go wrong. This approach of attaching context-specific guardrails at the point of temptation proved more effective than relying on generalized accuracy instructions alone.