If you've picked LangChain over CrewAI (or vice versa) partly for "security reasons," this preprint is worth five minutes.

I ran a controlled evaluation — 6 LLMs, 6 agent execution conditions, 5 attack families, 7,020 payload-verified trials — to isolate what actually explains security outcomes in agentic AI systems. The headline: framework choice explains about 0.06% of the variance (not statistically significant, p ≈ 0.70). Attack family explains ~29%. Model explains ~4%.

In plain terms: which framework you build on barely matters. What attack you're facing matters a lot. If your threat model assumes "framework X is inherently safer," this data doesn't back that up.

This is one of four connected preprints I've published on agentic AI security evaluation methodology — the other three dig into a more foundational problem: how reliable are the automated detectors (refusal classifiers, prompt-injection classifiers) that most red-teaming pipelines use to generate labels in the first place? Short answer: less reliable than commonly assumed, and I show exactly where they break.

All four preprints are open (CC BY 4.0) and the evaluation code is open-source:

Preprints: https://figshare.com/authors/Waqar_Javed/24479225
Eval framework GitHub: https://github.com/AgentSafeLabs/safelabs-eval
Eval framework Website: https://agentsafelabs.com/