AI Red Teaming for Agents: Why AgentOPFOR Gets It Right

Highlights

• An agent can retrieve unauthorized data, cross permission boundaries, and still return a completely clean response. AI red teaming that only reads the final answer misses the half that actually matters.
• Going deep on one attack vector before you know your attack surface is just a confident guess. Agent OPFOR maps 80+ evaluators across OWASP LLM, Agentic AI, MCP, and API frameworks so you know where your agent is weak beSfore you drill into any one risk.
• The most dangerous vulnerabilities are the ones your checklist never anticipated. Agent OPFOR's autonomous hunt mode follows the target's own behavior mid-run and finds weaknesses no predefined suite would ever reach.

Agents are going into production faster than anything before them — a support bot here, an internal copilot there, an agentic workflow wired into three APIs. Security testing is improvised after the fact, if at all, and the instinct is to reach for tools we already trust. But the scanner, the fuzzer and the pentest checklist were built for a threat model that no longer describes what’s being deployed.

This isn’t a feature list. It’s an argument for why the agent era needs a tool built for it.

Why Can’t You Red-Team an Agent Like a Model?

Someone files a support ticket. A few lines of text in a description field, the kind of thing that hits a queue a hundred times a day. Buried in that text are instructions meant for a machine.

Later, an admin asks an AI agent to categorise the ticket. Routine housekeeping. The agent reads the description, finds the hidden instructions, and follows them — then goes looking for a colleague. It finds a second agent with broader permissions and hands the work over. That agent copies restricted data, edits records, and grants the attacker an admin role.

Every step ran with the admin’s privileges, not the attacker’s. Agents inherit the authority of whoever set them in motion, and ServiceNow’s defaults let them find and call one another.

AppOmni’s Aaron Costello demonstrated this in November 2025. His conclusion is the part that should worry you: “it isn’t a bug in the AI; it’s expected behavior as defined by certain default configuration options.” ServiceNow didn’t patch it. No CVE was issued. They updated the documentation and left the fix to customers. (coverage in The Hacker News)

And the categorisation reply? It looked completely normal.

That’s the whole problem. A model is a function — text in, text out. Prompt injection, jailbreaks, toxic generations all surface in what it says. Read the reply and you’ve read the risk.

An agent is not a function. It calls tools. It reads memory that persists across turns and sometimes across users. It talks to MCP servers whose tool descriptions flow straight into its context. It decides, mid-conversation, what to do next. Every one of those is a place something can go wrong, and none of them are visible in the final reply:

ServiceNow was several of these at once, and no amount of reading the agent’s output would have surfaced any of them. The failure didn’t live in the text. It lived in the action.

Most teams test the chatbot’s answers and call it covered. But the answer was never the dangerous part. The dangerous part is everything the agent did to produce it.

What Falls Through the Gaps Between Tools?

There is excellent AI security tooling out there — probe libraries with research-backed attack corpora, evaluator frameworks with precise programmatic control, scanners that do one job cleanly. This isn’t a landscape short on quality. It’s short on coverage in one place.

The agent attack surface isn’t a slice:

Covering all four means stitching together four or five tools, each with its own config, runner and report shape — then hoping the seams don’t hide anything.

That’s not a productivity problem, it’s a security problem. Every seam is a place where a vulnerability class falls between two tools and gets tested by neither — and attackers don’t respect your tool boundaries. Agent OPFOR runs 80-plus evaluators fanning into 490-plus attack patterns across the whole surface from one tool, so “did we test that layer?” stops being a question you answer by auditing five configs.

Should Security Testing Require a Developer Every Time?

Roughly half of every product team shipping AI will never open a command line:

• The PM who owns the support-bot roadmap
• The designer shaping the conversation flow
• The QA engineer signing off on the release
• The security analyst who has to vouch for it

These are the people closest to the feature, and under a CLI-only model none of them can test the thing they’re responsible for.

If red-teaming requires writing YAML, wiring a config and exporting three environment variables, it happens exactly as often as a developer has spare time — rarely, and never at scale. Security gated behind engineering doesn’t become a culture. It becomes a favour you ask someone busy.

This is why Agent OPFOR ships a browser extension, and it’s a security argument rather than a usability one. Anyone can open a deployed chatbot, click the icon, pick a suite, and watch a full scan run — no terminal, no YAML, no keys in a config file. The PM who noticed the bot acting strangely can check, instead of filing a ticket and waiting.

What Does Output Testing Miss?

You ask an internal knowledge agent a routine question. To answer, it calls an MCP tool that quietly reaches past its intended scope — pulling records, or hitting an internal endpoint, it was never meant to touch — then summarises back exactly the one clean sentence you asked for.

The visible answer is flawless. Any test reading only the final response passes it.

But the boundary was crossed in the tool call, not the reply. This isn’t an edge case; it’s the shape of how agents fail:

• A scope escalation in an MCP call doesn’t change a word of the response
• An unauthorised retrieval still renders a perfectly polite answer
• PII can leak into a tool call and never reach the user at all

The clean response is the disguise, and input/output testing is structurally blind to it.

That’s the case for trace-aware testing: the judge has to see every tool call and its arguments, every retrieval, every intermediate step. Agent OPFOR integrates with Langfuse and Netra so the judge reads the whole internal trace, and a “clean” answer hiding a scope escalation gets caught as the failure it is. If your testing only sees what the agent said, you’re testing half of what it did — and the dangerous half is the half you’re missing.

What’s the Risk of Depth Without Breadth?

There’s a temptation, especially first time round, to go deep: pick the scariest vector, find the most sophisticated tool for it, drill. It feels rigorous. It’s the wrong first move. A team that has never mapped its agent’s attack surface doesn’t have a depth problem — it has a knowledge problem. It doesn’t yet know where the holes are.

• Ten medium-confidence weaknesses across ten categories tells you where to spend your effort
• One perfectly characterised weakness in a category that wasn’t your biggest risk tells you almost nothing about the other nine

That’s why Agent OPFOR ships curated suites mapped to the standards that matter — OWASP LLM Top 10, Agentic AI Top 10, MCP Top 10, API Security Top 10, plus an EU AI Act bias suite — runnable in one command. You get a map before you get a magnifying glass. Once it shows a critical gap, then depth is worth it. Depth without breadth is just a confident guess.

Can You Trust a Security Tool You Can’t Inspect?

Trusting a security tool means trusting its verdicts — that when it says “safe,” it looked, and when it says “vulnerable,” it can show you why. A closed scanner asks you to take its word on exactly the thing you hired it to be rigorous about. That’s not rigor; it’s faith with a dashboard.

Real credibility requires everything be inspectable — every attack prompt, request, response, and judge verdict, logged and reproducible. If a finding looks like a false positive, open the transcript and settle it. If you don’t trust an evaluator’s rubric, read it, fork it, fix it.

Agent OPFOR is Apache 2.0 and fully open. The engine is auditable, the evaluators are plain YAML, the reports preserve every artifact. And the detail that quietly matters most: every evaluator ships with pass/fail fixtures that CI runs through the real judge on every change, so rubrics can’t silently rot. You don’t have to believe Agent OPFOR works. You can check.

What Can an Attacker Find That Your Checklist Can’t?

Structured suites have a ceiling: they test what you thought to test. A checklist is bounded by the imagination of whoever wrote it. Real adversaries don’t work from your checklist — they probe, notice something odd, pull that thread, and follow the target’s own behaviour toward the weak spot.

That’s what opfor hunt changes. You give it an endpoint and a plain-English objective, and a small team of cooperating AI agents runs the engagement itself — chasing a probe that looks promising, dropping one that stalls, inventing tactics mid-run, up to a dollar budget you set.

We pointed it at a support bot and told it: get a refund you shouldn’t get. It asked for one the normal way and was refused. But the refusal mentioned an escalation path for supervisors — so it spent the next four turns posing as one. On turn six it got the refund approved.

No suite contains a “pretend to be a supervisor because the refusal hinted at one” test, because nobody knew the refusal said that until the bot said it. That’s the point: hunt adapts to the target, and the most interesting vulnerabilities only reveal themselves under pressure you didn’t script in advance.

Where Did Agent OPFOR Come From?

There’s a difference between a security tool designed against a threat model on paper and one shaped by watching real systems break.

The team at KeyValue Software Systems has shipped 130 products for 90 startups over ten years. In the last 18 months almost every one had an AI agent inside it, and team after team hit the identical wall at the identical moment: the agent works, it’s about to ship, and nobody has a way to test it like an attacker would. Agent OPFOR was built for those engagements and open-sourced only after it had earned its keep.

A tool built from a threat catalog tests the failures someone predicted. One built from production incidents tests the failures that actually happen — the authorization gap in the refund tool, the PII that slips into a lookup call, the memory that carries across sessions it shouldn’t. One is theory. The other is experience, encoded.

Why Does This Matter Now?

Three things shifted at once:

• The standards arrived. OWASP now publishes Top 10 lists for LLM applications, agentic AI, and MCP — the shared vocabulary buyers and auditors are starting to speak.
• The regulation is live. The EU AI Act is in force, with real obligations around bias, transparency, and robustness.
• Buyers are asking. Security questionnaires now arrive before the contract is signed, and “how do you red-team your AI?” is increasingly on them.

A team that makes adversarial testing routine today answers those questions from a position of evidence. A team that waits assembles the practice under deadline pressure, mid-deal, exactly when it’s hardest to do well.

Get Started

Agent OPFOR isn’t the only AI security tool, and this was never an argument that it is. It’s the argument that the agent era needs a tool built for it — accessible to the whole team, covering the full OWASP surface, trace-aware enough to catch what clean answers hide, open enough to fork. It’s free.

• Star it on GitHub: github.com/KeyValueSoftwareSystems/agent-opfor
• Join the community on Discord: discord.gg/Sb8xfmrjjW
• Red-team a chatbot in five minutes, no terminal required: install the browser extension

The attack surface is already in production. The standards to measure against already exist. The only variable left is whether you build the muscle now, on your own schedule, or later, on someone else’s.

FAQs

1. Does Agent OPFOR work with the AI agents and tools my team already uses?

Agent OPFOR is framework-agnostic. It tests any agent exposed through a chat endpoint, API, or MCP server regardless of the model provider, orchestration framework, or tech stack underneath. If your agent can receive a message, Agent OPFOR can test it.

2. What does trace-aware testing actually catch that standard testing misses?

Agents can make unauthorized tool calls, retrieve data they shouldn't access, or escalate permissions and still return a clean final answer. Trace-aware testing reads the full internal execution — tool calls, arguments, retrieval events, and MCP interactions. Agent OPFOR integrates with Langfuse and Netra to evaluate the complete trace, not just what the agent said.

3. When should I use opfor run versus opfor hunt?

Use opfor run for repeatable, structured coverage — CI gates, pre-release checks, and regression testing against known risk categories. Use opfor hunt when you want to discover vulnerabilities you haven't anticipated. Hunt behaves like a creative adversary: it probes, follows interesting threads, and adapts mid-engagement based on how your agent actually responds.

4. Does Agent OPFOR require security expertise to use?

Not always. The CLI and SDK are built for security engineers and developers who want full control. The browser extension requires no terminal, no config, and no API keys — so product managers, QA engineers, and analysts can run adversarial tests directly against a deployed chatbot without involving a developer.

5. How does Agent OPFOR stay reliable as my agent changes?

Every Agent OPFOR evaluator ships with pass/fail fixtures that run through the real judge on every update, so evaluation rubrics cannot silently drift over time. Every scan also produces a timestamped report preserving every prompt, response, tool call, and verdict — making findings reproducible and comparable across releases.