Why MCP Servers That Pass All Tests Can Still Fail When Used by AI Agents

Developers building MCP (Model Context Protocol) servers often find their tools pass all integration tests yet remain unusable when connected to AI models like Claude. The core distinction is between testing — which verifies that individual tool calls return correct responses — and evaluating, which checks whether an AI agent can actually reach the right answer using only the server's tools and descriptions. An MCP eval presents a realistic, user-phrased task with a call budget and scores whether the model arrives at the correct outcome, without specifying which tools to use. Evaluations produce four meaningful outcomes: pass, wrong answer, too many calls, and untestable — and unlike tests, they are inherently non-deterministic. Notably, call count tends to degrade before pass rate does, making it a useful early warning signal for server usability problems.