Every other page under AI Security describes a category of vulnerability (prompt injection, excessive agency, supply chain risk) and what mitigates it. Red teaming is not another category on that list. It is the practice of adversarially testing whether a specific system actually resists those categories. A control described in documentation is not yet a control that works in the deployed system.
NIST’s ARIA pilot draws a specific three-way distinction worth keeping, because the terms get used interchangeably in practice and the conflation hides what each one actually checks:
- Model testing confirms an application’s stated capabilities work as described. It answers “does this do what it claims to do.”
- Red teaming deliberately attempts to induce the application to violate its own guardrails through adversarial or stress-testing input. It answers “can this be made to do what it claims it will not do.”
- Field testing observes realistic use to see what actually happens when real users interact with the system. It answers “what happens when nobody is trying to break it on purpose.”
A system that passes model testing and looks fine in field testing can still fail red teaming, and that failure is the one that matters for an adversary who is, by definition, not a realistic or well-behaved user.
The four areas the OWASP guide organises around
The OWASP GenAI Red Teaming Guide, published by the same project that maintains the LLM Top 10 covered throughout this section, structures testing into four areas instead of treating red teaming as a single undifferentiated activity:
- Model evaluation: vulnerabilities intrinsic to the model itself (bias, toxicity, susceptibility to adversarial prompting), independent of how it is deployed.
- Implementation testing: how the model is wired into an actual application: prompt injection routes, output handling, the specific attack surfaces this section’s other pages describe.
- Infrastructure assessment: the deployment platform, API surface, and access controls around the model, which fail in ways that have nothing to do with the model’s own behaviour.
- Runtime behaviour analysis: what the deployed system does under sustained adversarial use over time, not just in a single test session.
A red-team exercise that only covers one of these checks a narrow slice of the actual attack surface. Commonly the one covered is prompt-level jailbreak attempts against implementation testing. OWASP’s own vendor-evaluation criteria name this directly: distinguishing genuine adversarial testing from “jailbreak-only” offerings that test the easiest and most visible category while leaving infrastructure and runtime behaviour unchecked.
What a credible red-team engagement actually evaluates
OWASP’s vendor evaluation criteria, written to help organisations tell a substantive red-teaming engagement from a superficial one, give a concrete way to check rigour, so a red-team report’s thoroughness need not be taken on claim:
- Whether the threat model is specific to the system under test. A chatbot, a RAG application, a tool-calling agent, and an MCP-based multi-agent workflow have different attack surfaces, and a generic test suite run unchanged across all of them is a sign of superficial coverage.
- Whether findings are reproducible, not just reported - a claimed vulnerability with no reproduction steps is not evidence a fix addressed it.
- Whether the engagement covers agentic and orchestration-level risks when the system has them, not only single-turn prompt-level attacks.
- Whether governance and reporting are structured well enough to feed back into the system’s own risk register, rather than existing as a standalone report nobody acts on.
Where this fits in an AI RMF-based programme
NIST’s Generative AI Profile treats red teaming as one of the suggested actions under the RMF’s Measure function, the stage where an organisation checks whether a system’s actual behaviour matches what it is supposed to do, as distinct from Govern (deciding what should be true) and Map (identifying what could go wrong). Positioned this way, red teaming is not a one-off pre-launch activity; it is a recurring measurement, repeated as the system, its prompts, its tool integrations, and the broader threat landscape change.
Limitations
- A red-team exercise proves the specific attacks attempted did or did not succeed. It does not prove the absence of attacks nobody thought to try. Passing a red-team exercise is evidence of resistance to known technique classes, not a certification of general safety.
- Jailbreak-only testing is the most common shortcut and the easiest to mistake for thorough coverage, precisely because prompt-level attacks are the most visible and the cheapest to run.
- Red teaming a model in isolation does not test the application built around it: implementation and infrastructure-level failures require testing the deployed system, not just the underlying model.