AWS Bedrock's Built-In Eval Misses Three-Quarters of Your Agent, Developer Warns

A developer building an agent on AWS Bedrock discovered that the platform's built-in Model Evaluation job only assesses the foundation model itself, leaving action groups, the Knowledge Base, and Guardrails entirely unexamined. This gap caused the agent to silently drop tool calls in production despite passing all built-in checks before deployment. The developer found that tool-call evaluation requires three distinct checks: whether the correct tool was selected, whether arguments were semantically accurate, and whether the agent actually used the tool's returned output. Retrieval quality from the Knowledge Base must also be scored independently from the final answer to distinguish bad retrieval from bad model reasoning. The developer recommends running these evaluations continuously on every change rather than relying on a single one-shot job at the start of development.