Highlights
- Promptfoo excels at broad model coverage (80+ providers) and mature CI/CD pipeline integration — the most battle-tested open-source red teaming platform available
- Agent OPFOR goes deeper inside the agent — direct MCP server testing at the protocol level, all four OWASP standards (LLM, Agentic AI, MCP, API) in one tool, and trace-aware judging that sees what the agent did, not just what it said
- Promptfoo is MIT-licensed, acquired by OpenAI in March 2026; Agent OPFOR is Apache 2.0, vendor-independent
- Both are free and open source
Your agent doesn't just talk. It calls tools, holds memory, and reaches into MCP servers — so its worst-case failure isn't an awkward sentence. It's a real action: a refund it should never have issued, another user's record handed over, a shell command run through a tool call. Red teaming is how you catch those before someone else does. This is what LLM security means for autonomous systems: testing what an agent actually does, not just what it says.
If you're shopping for an open-source tool, two names keep coming up: Promptfoo and Agent OPFOR. Both test agents. The difference is shape. Promptfoo casts the widest net in AI red teaming — the most models, the deepest CI integration, the biggest community. Agent OPFOR reaches the deepest into the agent itself — its tools, its MCP servers, and what it does behind the response. Width or depth. That's the real question, and which one you need depends entirely on what you're shipping.
At a glance
Comparison reflects both projects as of June 2026. Both ship quickly — Promptfoo in particular releases often, so coverage details may have changed.
Where Promptfoo casts the widest net
Promptfoo earned its reputation, and for a lot of teams it's exactly the right call. Start with the obvious: it speaks to 80+ model providers — OpenAI, Anthropic, Google, Azure, Bedrock, local models, a long tail of gateways. If you need to run the same attacks across a fleet of models and compare, nothing here matches that reach.
It's also the best citizen in a CI pipeline you'll find. Two commands and it's running:
npx promptfoo@latest redteam init
npx promptfoo@latest redteam run
Wire in its GitHub Action and the build fails when results cross a severity or assertion threshold. It emits JUnit XML and SARIF, so failures show up natively in your pipeline and in GitHub code scanning. Your whole test lives in a version-controlled promptfooconfig.yaml — diffable, reviewable, the way engineers like it. And when you want to show someone the results, there's a polished web UI with compliance report cards that map findings to OWASP, NIST AI RMF, MITRE ATLAS, and the EU AI Act — handy when the conversation turns to audits.
Add a large, active community and a deep library of plugins and attack strategies, and you've got the most mature developer red teaming platform going. If your priority is broad model coverage wired tightly into CI, Promptfoo is hard to beat.
Who owns your red teamer?
One thing to weigh before the technical stuff, because it's easy to miss. In March 2026, OpenAI acquired Promptfoo and is folding its red teaming tech into its agent platform. Promptfoo says the project stays open source and keeps supporting a wide range of providers, and that's a credible commitment.
But think about what a red teamer is: the independent referee for your AI. Now one of the two big open-source referees belongs to a model vendor. For plenty of teams that's fine. For others — especially anyone who needs to argue their testing is impartial — a referee that doesn't answer to a model lab is worth something. Agent OPFOR is independent, Apache 2.0, built by KeyValue. Where you land on that is a real input, not a footnote.
Where Agent OPFOR reaches deeper
Now the depth. The moment your target stops being a single model and becomes an agent with tools and an MCP server behind it, the wide net starts skimming the surface. This is where Agent OPFOR goes under it.
Start with MCP, because it's the clearest gap. If you've got an MCP server in the loop, the two tools aren't doing the same thing at all. Promptfoo tests MCP the way it tests everything else — it prompts a model and grades the words that come back. Agent OPFOR connects straight to the server, enumerates its real tools and resources, and fires actual adversarial tools/call requests at it: command injection, SSRF, scope escalation, a poisoned tool description, and more — then reads the raw protocol response. It ships a full MCP security suite, so you're running those attacks, not writing them. The difference is asking a model about a locked door versus walking up and rattling the handle yourself.
It covers the whole standards surface in one place. Agent OPFOR runs OWASP LLM, Agentic AI, MCP, and API Security Top 10 from a single tool, plus an EU AI Act bias suite — one install, one report, 60-plus evaluators fanning out into 400-plus attack patterns across the whole agent surface, with a separate tree just for MCP. And it's trace-aware: connect Langfuse or Netra and the judge sees what the agent did, not just what it said. That's how it catches the failures input/output testing is blind to — the PII that slipped into a tool call but never reached the user, the scope escalation that didn't change a single word of the reply.
Both tools are template-driven, so that part isn't new. What differs is where the composability lives. Promptfoo's red-team coverage is a plugin × strategy cross-product, but the built-in plugins are compiled TypeScript, and a custom attack means writing a separate generator/grader file. In Agent OPFOR, every evaluator, shipped or hand-written, is the same declarative YAML: an id, an attack template, pass/fail criteria. Drop a new one in and it's auto-discovered with no code and no registration, then auto-enrolled into whichever OWASP suite its own tag says it belongs to, instead of living on a list someone has to remember to update.
And it doesn't assume you live in a terminal. Agent OPFOR is the only open-source AI red teaming tool with a browser extension: open a deployed chatbot, click the icon, pick a suite, get a report. Your PM, your QA lead, your security analyst can all run it without touching a config file. For engineers, it's three commands:
npm install -g @agent-opfor/cli
opfor setup
opfor runRegister it as an MCP server in Cursor or Claude Desktop and your coding agent can even red team your other agents in plain English.
And when you'd rather not pick the attacks at all, it runs the whole engagement for you. Hand opfor hunt a target, a plain-English goal — "see whether the support bot can be talked into a refund it shouldn't give" — and a dollar budget, and it handles recon, adaptive multi-turn attacks, self-judging, and the report from end to end. Promptfoo has adaptive attack strategies of its own — GOAT, Crescendo — that sharpen probes against the vulnerabilities you name. Agent OPFOR's autonomous mode hands the entire engagement to a team of agents and lets them improvise.
Promptfoo brings broad model-provider coverage and a well-established ecosystem, while Agent OPFOR is purpose-built around focused agent testing. Both fit naturally into CI pipelines–Agent OPFOR can gate builds on failed checks, while Promptfoo adds capabilities such as an official Action, SARIF output, and severity thresholds. In other words, both are strong options, with the better fit depending on whether your priority is focused agent evaluation or broader CI and provider support.
For a deeper look at how Agent OPFOR works under the hood, see AI Red Teaming for Agents: Why AgentOPFOR Gets It Right and What Is Agent OPFOR? Open-Source AI Agent Red Teaming for LLM Apps and MCP Servers.
So which one?
Reach for Promptfoo if you want the broadest model coverage, you're wiring red teaming into an existing CI pipeline with build-gating, or you want a hosted UI with audit-ready compliance cards.
Reach for Agent OPFOR if you've got an MCP server you want attacked at the protocol level, you want all four OWASP standards from one tool, you need non-developers to test a live agent through a browser extension, you want trace-aware judging that sees inside the agent, or you'd rather your red teamer stay vendor-independent.
Here's the thing: width and depth aren't rivals. Promptfoo will tell you how your model behaves across a hundred providers. Agent OPFOR will tell you whether your agent can be talked into doing something it shouldn't — and whether the MCP server behind it quietly hands over the keys. Plenty of teams will run both. The real question was never which tool. It's: how far can someone push what you've shipped — and have you actually looked?
Try Agent OPFOR
npm install -g opfor→ Star and explore Agent OPFOR on GitHub
Not a developer? Install the Agent OPFOR Chrome extension and red team a deployed chatbot in a few clicks — no code required.
FAQ
1.How is Agent OPFOR different from Promptfoo?
Both are open-source AI red teaming tools that test agents. Promptfoo casts the wider net — 80+ model providers and deep CI integration. Agent OPFOR reaches deeper into the agent: it red teams MCP servers directly at the protocol level, covers all four OWASP Top 10 standards (LLM, Agentic AI, MCP, API Security) in one tool, ships a browser extension for non-developers, supports trace-aware judging, and stays vendor-independent.
2.Does Promptfoo test MCP servers?
Yes — Promptfoo can point at an MCP server and probe it with adversarial prompts. Agent OPFOR goes deeper: it connects to the server directly, enumerates its tools and resources, and fires real adversarial tool calls, backed by a dedicated MCP security suite.
3.Which tool covers the OWASP Agentic AI Top 10?
Both. Promptfoo offers an agentic plugin collection, and Agent OPFOR ships an OWASP Agentic AI suite. Agent OPFOR also covers the OWASP LLM, MCP, and API Security Top 10 in the same tool.
4.Is Promptfoo still open source after joining OpenAI?
Yes. Promptfoo is MIT-licensed, and following OpenAI's March 2026 acquisition the team committed to keeping the project open source and multi-provider. The open question isn't the license — it's independence: your red teamer now belongs to a model vendor. If you need to argue your testing is impartial, Agent OPFOR (Apache 2.0, built by KeyValue) stays vendor-neutral.
5.Can non-technical team members run Agent OPFOR?
Yes — that's a core difference. Agent OPFOR is the only open-source AI red teaming tool with a browser extension: a PM, QA lead, or security analyst can open a deployed chatbot, click the icon, pick a suite, and get a report with no terminal, no YAML, and no config files. Promptfoo is a developer-first tool that runs from the CLI. If you need people outside engineering to red team a live agent themselves, Agent OPFOR is the one built for it.
6.Which open-source tool is the best Promptfoo alternative for agent and MCP testing?
For testing AI agents and MCP servers specifically, Agent OPFOR is the closest open-source alternative. It covers all four OWASP Top 10 standards (LLM, Agentic AI, MCP, API Security) in one tool, red teams MCP servers directly at the protocol level, adds trace-aware judging and an autonomous mode (opfor auto), and runs from a CLI, a browser extension, or as an MCP server itself. Promptfoo remains the stronger choice when you need the broadest model-provider coverage and the deepest CI/CD integration.
7.Can Agent OPFOR run in a CI/CD pipeline?
Yes. opfor execute exits non-zero the moment a check fails, so a red team scan can gate a build the same way a failing test does, and it writes a JSON report that drops straight into a pipeline artifact. Promptfoo's CI integration is more featureful — an official GitHub Action, SARIF output, and severity thresholds — so if deep pipeline tooling is your priority, Promptfoo leads there; if you just need red teaming to fail the build on a finding, Agent OPFOR does that out of the box.
8.Is Agent OPFOR free to use?
Yes. Agent OPFOR is fully open source under Apache 2.0 — free to use, modify, and self-host, with every attack prompt, request, response, and judge verdict logged for reproducibility. You only pay your own LLM provider for the model that generates and judges attacks. (Trace-aware testing can optionally connect to Netra, the team's paid observability product, but it also works with open-source Langfuse.)
Use Agent OPFOR only on systems you own or are authorized to test.
