What Is Red Teaming and Why AI Systems Need It

Red teaming is a structured adversarial exercise, and in modern AI systems it has already shifted from model-only prompt attacks to testing the full environment around the model. NIST's large-scale red-teaming work showed more than 400 participants making over 250,000 attack attempts against frontier AI models, and at least one successful attack landed on every target model tested. That's why the useful question isn't whether a model can survive a checklist, it's whether an attacker can still shape outcomes when the model sits inside tools, workflows, and business processes.

What many teams call red teaming is often too narrow. If you only test jailbreak prompts, you're checking one layer of a much larger attack surface. The stronger practice is closer to a controlled adversarial engagement, one that tries to force an operational failure and then traces exactly where the control stack broke down.

Defining Red Teaming

Most AI red teaming still stops at the wrong layer. Teams probe prompt outputs and call it done, but red teaming began as a structured adversarial exercise aimed at exposing weaknesses by thinking like the attacker. NIST's generative AI report traces the idea back hundreds of years to warfare and religious contexts, and the formal “red team” label was adopted by the U.S. military in the 1960s to model Soviet behavior against a blue team representing the United States. That history matters because the goal was never to grade a system, it was to break assumptions.

From wargaming to AI governance

NIST defines AI red teaming as a structured testing effort used to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with developers. The U.S. Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence made that definition explicit in federal policy in 2023, which helped standardize the term for government and industry use. Those two steps, formal definition and policy adoption, turned a security instinct into a governance control.

That distinction gets blurred in vendor marketing. A penetration test looks for exploitable weaknesses. A benchmark measures performance against a dataset. Red teaming tries to achieve an objective without being detected, then explains the impact if the attacker succeeds.

Practical rule: if the activity does not simulate an adversary with a goal, it is evaluation with a more dramatic name.

The useful mental model is broader than models alone. Red teaming asks what happens when someone tries to make the system fail on purpose. In AI, that means the model, the agent layer, tools, retrieval, memory, and the people and processes around them all belong in scope. Tool misuse, memory poisoning, and similar agent-specific attack paths make older checklist-style reviews incomplete.

A model that resists prompt injection can still fail through an over-permissive tool, a stale memory write, or a workflow that turns a small mistake into a business incident. That is the shift security teams need to keep in view.

How Red Teaming Works in Practice

Red teaming earns its value when it produces cause-and-effect evidence, not a pile of loose findings. The practical pattern is straightforward: define a realistic threat, map its tactics and techniques into an execution plan, then measure whether defenders can detect, block, or contain each stage. If the team gets initial access, privilege escalation, lateral movement, or exfiltration, defenders can trace that outcome back to a specific gap instead of guessing where the control failed.

Adversary emulation beats generic testing

That cause-and-effect tracing is what turns a report from archive filler into a control change. A vulnerability scanner can say a surface is exposed. A red team can show that the exposure leads to business impact under realistic attacker pressure. The primary output is operational impact analysis, not just a list of weaknesses.

In practice, strong engagements start with a defined objective and a realistic attacker profile. The red team may simulate phishing-led entry, an exposed service, or a compromised identity, then work post-compromise to see how far they can move before detection. The blue team's response matters just as much as the breach path. A system that detects late but contains fast has a different risk profile from one that never notices at all.

That same mindset carries into AI systems, where the useful question is no longer only whether the model resists prompt injection. It is whether the agent layer, tools, retrieval, and memory can be turned against the system through misuse or poisoning. Older checklist-style reviews often miss those failure paths, especially when the attack chain crosses model behavior and operational workflow.

For teams comparing tooling and process, the internal catalog entry on red team LLM tools compared helps separate single-purpose scanners from workflow-oriented adversarial testing.

A useful report doesn't say “the system was vulnerable.” It says which technique worked, which control failed, and which compensating control would have changed the outcome.

What good deliverables look like

Strong red team output usually includes:

• Attack path narrative that explains how the adversary moved.

• Chronology of actions, so defenders can line up logs and alerts.

• Root cause analysis tied to the exact control gap.

• Remediation priorities based on impact, not just severity labels.

That is why red teaming is more than offensive testing. It rehearses detection and response under pressure, then uses the evidence to improve the environment.

Red Teaming Versus Standard Security Evaluations

A lot of organizations already run scanning, pen tests, and benchmark evaluations, so the question is where red teaming adds something they don't. The answer is simple. Standard evaluations are necessary, but they're usually bounded by known patterns. Red teaming is designed to be adaptive, adversary-like, and uncomfortable enough to reveal what the checklist misses.

The gap shows up most clearly in AI-adjacent testing. NIST's AI-agent red-teaming summary reported that task-hijacking success rates rose from 11% to 81% when red teamers built novel attacks specific to agent behavior rather than relying on known baseline methods. That's a huge warning sign for teams who assume a standard prompt-injection checklist is enough. Older methods can still be useful, but they don't reliably capture how attackers behave when the system has tools, memory, and action privileges.

The same logic applies to broader security programs. A scanner can confirm that a server is patched. A benchmark can compare two models on the same dataset. Red teaming asks whether a determined adversary can still turn your environment into a business problem. If the answer is yes, the remediation priority changes immediately.

For organizations mapping red teaming into governance, the NIST AI RMF tool page at NIST AI RMF is a useful anchor for deciding where adversarial testing fits beside controls, risks, and monitoring.

The key trade-off is time and realism. Red teaming takes more planning than scanning and gives less comfort than a neat benchmark score, but it produces the kind of evidence a CISO can use to set gates, funding priorities, and control ownership.

Why AI Systems Require a Different Approach

AI systems change the risk profile, not just the vocabulary. NIST defines AI red-teaming as a structured testing exercise that is adversarial, goal-oriented, and bounded by a test objective rather than a generic benchmark score. That framing matters because failures usually emerge from the interaction between the model, the surrounding controls, and the environment it operates in.

The attack surface now includes agents and tools

Classic LLM attacks still matter, especially prompt injection and jailbreaking. Modern deployments also add tool misuse, memory poisoning, retrieval layers, external APIs, and automated actions that can move from text generation into real-world impact. Once an agent can call tools or act on behalf of a user, the attacker does not only need the model to say the wrong thing. They need it to do the wrong thing.

That is why model-only checklists miss so much. Attack families tuned for agent behavior can expose failures that older threat taxonomies do not cover well, especially when tools, permissions, and state carry the blast radius. If you only test known jailbreak patterns, you can miss the failures that matter most in deployed agentic systems.

For practitioners who want a focused view of this problem, testing LLM security in agentic systems is a useful external read because it treats tool use and execution privileges as first-class risks, not edge cases.

Practical rule: if the system can retrieve, remember, or act, then red teaming has to test those actions, not just the language model's answers.

The OWASP LLM Top 10 is a good companion for mapping these risks into categories that security and engineering teams can discuss without flattening them into generic application issues.

The other major change is opacity. Traditional security testing often relies on straightforward logic, an input triggers a condition, a condition triggers a failure. AI systems can behave differently under small prompt changes, hidden context, or interaction history. That makes static checklists incomplete. Red teaming still works, but only if the team treats the AI system as a live sociotechnical process, not a static application endpoint.

Expanding Scope from Models to Full Systems

Model-centric red teaming is no longer enough for agentic deployments. A July 2025 arXiv paper argues that current AI red teaming is too narrowly focused on individual model flaws and should be split into macro-level system red teaming and micro-level model red teaming. That distinction is useful because it reflects how real failures happen. The harm usually comes from interactions among the model, the tools around it, the user, and the deployment environment.

Decide what the test is actually for

A practical scope question helps here, and many teams get it wrong by starting too small. If the goal is to judge a model's raw behavior, then a micro-level test is appropriate. If the goal is to understand whether an AI assistant can be used safely in production, then the test has to cover workflows, integrations, permissions, logging, and incident response. Those are different exercises, and mixing them creates false confidence.

The right scope for a system-level engagement usually includes:

• Tool access, because agents can misuse APIs or call actions outside intent.

• Persistent memory, because poisoned state can change later behavior.

• Retrieval and document flows, because injected content can alter decisions.

• Deployment context, because controls in staging may not match production.

• Human response paths, because slow escalation turns technical incidents into operational ones.

The July 2025 paper on macro and micro red teaming is especially relevant for procurement teams and security leaders who need to set boundaries before a vendor demo turns into a false assurance exercise. It also helps explain why some older review models miss the point. A jailbreak test can show model fragility, while a workflow test can show whether the assistant will expose data, misuse a tool, or carry bad instructions across sessions.

A practical illustration of system-level thinking appears in running AI compute with Goblins, which is useful not because of the branding, but because it highlights how orchestration and execution context shape real AI risk.

A simple scoping test

If an attacker can change the outcome by touching anything outside the prompt box, the scope has to expand. That means red teaming should cover the full sociotechnical stack, not just the model endpoint. For agentic systems, that's the only way to get a meaningful answer about real-world harm.

Watch: Red teaming AI systems in practice (video)

Real-World Red Teaming Scenarios and Failures

A model can look safe in a demo and still fail in a live workflow. Prompt injection through retrieved content is the cleanest example. A user uploads a document, or a web page contains hidden instructions, the assistant reads it, and the injected text overrides the intended behavior. The model did not go rogue. The system let untrusted content sit too close to the control plane, so the wrong instruction won.

What attackers actually try

The more serious failures usually show up after the first successful trick. In agentic systems, a red team will test whether a model can:

• Use a tool it shouldn't because the request sounds plausible.

• Carry state across turns after being exposed to manipulated memory.

• Leak data through retrieval, summarization, or logging.

• Escalate from text into action, for example by sending a message or modifying a record.

Those tests are different from ordinary jailbreaks. A jailbreak tries to break the content policy. A system-level attack tries to make the agent do something operationally harmful while still looking like normal work. A harmless-looking prompt can be more useful than an obviously malicious one, because it exposes how the workflow behaves under pressure.

That is why the scope has shifted from model-centric checklists to sociotechnical testing. A model is only one control point. The more tool access you grant, the more you need to test authorization, separation of duties, and recovery paths. If memory is persistent, test whether poisoned state survives long enough to affect later decisions. If retrieval is enabled, test whether untrusted documents can shape outputs in ways the user never intended.

The failure usually is not one dramatic jailbreak. It is a chain of small permissions that add up to real impact.

The same logic shows up in running AI compute with Goblins, where orchestration and execution context shape the risk more than the model alone.

Real red team reports should read like incident analysis, not a prompt gallery. They should identify which component failed, which control should have stopped the chain, and whether the issue lives in the model, the orchestration layer, or the human process around it. That is what makes the findings actionable.

Building a Continuous Red Teaming Program

Red teaming should be a control, not a stunt. A one-time adversarial review does not fit AI systems that change with prompts, data, integrations, and agent behavior. The practical answer is continuous red teaming, because regulated environments need release decisions based on repeatable testing, not a single test date.

Make red teaming part of governance

The operating model is simple. Run adversarial tests on a schedule, rerun them after major model or workflow changes, and treat failures as release-gating events when the affected path touches sensitive data, external actions, or regulated decisions. That aligns with SOC 2, ISO 27001, the EU AI Act, and NIST-style risk management, because those frameworks care about repeatable controls and evidence, not intent alone.

Measurement is where most programs fail. A useful program tracks whether findings are getting closer to production reality, whether the same attack patterns keep reappearing, and whether remediation closes the gap. If the red team finds the same issue every cycle, the program is not learning. It is producing paperwork.

A practical cadence often includes:

• Pre-release testing for new models, tools, or agent workflows.

• Post-change testing after prompt, policy, or retrieval updates.

• Periodic reassessment for deployed systems with external users or data access.

• Vendor re-review when a third-party model or integration changes behavior.

The best programs also define what happens when a test fails. That can mean a release gate, a compensating control, a monitoring update, or a vendor escalation. Without that decision tree, findings do not change risk.

The shift is already visible in industry and government work. NIST's competition report, the NIST-linked task-hijacking results, and 2025 discussions of continuous red teaming all point in the same direction. AI risk is moving too fast for static review cycles, especially where agents can act on behalf of users. Tool misuse, memory poisoning, and retrieval abuse are not model-only problems. They are system problems, which means checklists aimed at a single prompt or a single jailbreak are incomplete.

If you are building that kind of program, use the controls and tool listings at AI Compliance Index to compare red teaming products, frameworks, and governance references in one place.