Three key decisions for adding RAG pipelines to legacy .NET and SQL Server systems

Integrating retrieval-augmented generation (RAG) into existing Microsoft .NET estates requires careful architectural choices to avoid security and maintenance pitfalls. Rather than provisioning a separate vector database, developers should use SQL Server's native vector indexing or Azure AI Search to reuse existing access-control rules instead of duplicating them. Document chunking should follow the business structure of content — such as contract clauses or report sections — rather than arbitrary token-window sizes, as poorly split fragments degrade answer quality regardless of the underlying model. Orchestration logic should remain within the existing .NET service layer using tools like Semantic Kernel, preventing a shadow authorization system from quietly emerging in a separate microservice. Treating RAG as a self-contained parallel project may work in demos but creates security and governance gaps once deployed to production systems.