Why a 98% RAG Evaluation Score Still Leaves AI Systems Vulnerable to Attack

Retrieval-Augmented Generation (RAG) systems can score near-perfectly on quality benchmarks yet remain critically exposed to prompt injection attacks hidden inside uploaded documents. Because retrieved text chunks are pasted directly into the LLM prompt, the model cannot reliably distinguish trusted system instructions from malicious content embedded in a user-uploaded file. This attack vector, known as indirect prompt injection, requires no suspicious user input — a seemingly ordinary PDF can silently carry instructions that hijack the chatbot's behavior. Production teams are warned that evaluation metrics measure accuracy and groundedness, while trust and security are entirely separate concerns that benchmarks do not capture. The key takeaway for AI teams is that a system must be hardened against adversarial documents and users, not just optimized for performance on golden datasets.