Developer Tests RAG System Robustness by Swapping Every Embedder, Generator, and Judge Model

A software developer ran a structured experiment to determine whether a retrieval-augmented generation (RAG) pipeline's performance holds up when every model component is replaced. The frozen pipeline used simple 200-word chunks, cosine similarity retrieval, and a shared prompt, while three embedders, five generators, and three AI judges were swapped as variables. The evaluation set was expanded from 10 to 100 questions spanning four distinct topics, including 20 'refusal traps' where correct answers are absent from the corpus to test whether models answer from documents or pretrained memory. Three separate AI judges — from Claude, GPT, and Gemini — scored all answers independently to detect whether any judge favored outputs from its own model family. The goal was not to identify a winning model but to verify that the system's conclusions remain valid regardless of which vendor's models are plugged in.