Why LLMs in Production AI Systems Need a Data Sanitization Layer Before RAG

A developer working on Agentic AI for production support has raised concerns about excessive sensitive data being sent to large language models. The author argues that information such as hostnames, API keys, authorization tokens, and file paths is often unnecessary for LLMs to resolve incidents like disk space errors. A key insight highlighted is that RAG alone is not a security boundary, since raw data passes through embedding models before reaching a vector database, meaning sanitization must occur earlier in the pipeline. The proposed architecture adds a clean-and-validate layer before both the embedding stage and the final LLM call, covering retrieval queries and observability logs as well. The author concludes that AI governance should be enforced through system design rather than policy documents alone.