AI Agent Framework Choice Has Negligible Impact on Security, Study Finds

A controlled study involving 7,020 payload-verified trials across 6 large language models and 6 agent execution conditions found that the choice of AI agent framework — such as LangChain or CrewAI — accounts for only 0.06% of variance in security outcomes, a statistically insignificant result. In contrast, the type of attack used explained roughly 29% of variance, while the underlying model accounted for about 4%. The research was conducted to test whether framework selection meaningfully affects security in agentic AI systems, a common assumption among developers. The study is part of a series of four connected preprints also examining the reliability of automated detectors used in AI red-teaming pipelines, which the author found to be less dependable than widely assumed. All preprints are published under a CC BY 4.0 open license, with evaluation code available as open-source.