Benchmark: AI framework choice barely matters for accuracy, but model selection does

A controlled benchmark tested LangGraph 1.2.9 and Pydantic AI 2.13.0 across 160 agent tasks covering order processing, shipping quotes, refund eligibility, and inventory decisions. Using GPT-4o at zero temperature, both frameworks achieved identical 100% correctness, though LangGraph was roughly 1.4–1.8 seconds faster per task due to Pydantic AI's async-to-sync overhead. When the model was swapped to GPT-4o-mini with all other variables held constant, overall accuracy fell to 75%, with one task — a date-arithmetic refund window check — failing all 20 of its runs systematically. The failure was traced entirely to the model miscalculating an 18-day return window, not to any difference in framework behavior. The findings suggest that for structured agent tasks, model choice drives correctness far more than framework selection, particularly where temporal or date-based reasoning is involved.