Developer builds open-source agent eval harness, finds integration layer is the real weak point

A developer building EvalForge, an open-source evaluation harness for tool-using AI agents, designed a scenario pack system with 28 structured YAML test cases covering retrieval, security, and failure recovery. The pack format deliberately strips expected outputs and scoring metrics before any data reaches the agent, ensuring evaluations remain uncontaminated. While the scoring and rubric design held up, the integration layer between the test harness and real third-party agents proved to be the critical failure point. Real-world agents introduced problems such as module-scope imports, hardcoded model versions, and unsafe file writes that a clean test environment could not anticipate. The experience highlighted that agent evaluation differs fundamentally from model evaluation because the execution path, not just the final answer, determines reliability.