What is red teaming?
Red teaming is when a group deliberately attacks a system the way an adversary would, in order to find its weaknesses before anyone else does.
The method comes from defence and security work. A red team plays the attacker and a blue team defends, and both learn from the outcome.
For AI it means testing how a model or an AI system responds to attempts to trick it. The testers try to make the model break its rules, reveal information or carry out actions it should not.
The EU AI Act requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing. In March 2026 NIST reported on a large public competition in which participants tried to hijack AI agents.
How it works
A test starts by asking what would be serious. It might be the assistant repeating information from the wrong department, or an agent sending something outside the organisation. The testers then try to achieve exactly that, with anything from simple questions to hidden instructions in documents.
The findings are documented and addressed. The system is then tested again, because a change can open new routes.
Why it matters
What works in a demonstration can behave differently once someone actively tries to trick it. Red teaming shows where the limits really are and gives a basis for putting the right safeguards around an AI agent.
What to keep in mind
Test the whole chain, including permissions, tools and the documents the model reads. Many weaknesses sit in the connections around the model.
Make sure the tests cover prompt injection, one of the risks that is hardest to close completely.
What it means for you
Test your AI tools with the data and documents you actually use. That gives a more realistic picture than generic tests.
Common questions
Is red teaming the same as a penetration test?
They are similar. A penetration test often looks for technical vulnerabilities, while red teaming mimics an adversary and can use whichever routes an attacker would choose.
Do we need to do red teaming ourselves?
That depends on how you use AI. For an agent that can take actions in your systems, it is worth testing with your own data. Read more about AI agents.
Sources: NIST on red teamNIST on the AI agent competition (2026)Regulation (EU) 2024/1689 on EUR-Lex
Related terms
Read more
All terms Updated