AWS Labs agent-eval sample uses same AI model as both subject and judge

An AWS Labs open-source toolkit called Agent-EvalKit contains a QA evaluation example where the same Claude Sonnet model acts as both the AI agent being tested and the judge scoring its responses. The flaw stems from a default constructor argument in a helper class, meaning no explicit decision was ever made to use the same model in both roles. The bundled evaluation report awards the agent a faithfulness score of 78.2%, but that score was generated by the very model whose faithfulness was under assessment. The author notes there is no documentation in the repository acknowledging this self-grading setup or discussing judge independence and model bias. While using a single model for both roles can be justified on cost and simplicity grounds, the concern raised is that the tradeoff was never surfaced or disclosed as a deliberate design choice.