A disciplined adversarial process testing AI systems to find vulnerabilities, misuse risks, and failures before deployment.
Also known as: Red Team, Adversarial Testing, Adversarial Evaluation
In plain English
Red teaming is like hiring people to try and break a product before customers use it, like a staged break-in or crash test. For AI, teams try to make the system fail, reveal sensitive data, or be misused so developers can fix problems before release.
In practice
Organisations run red teaming to simulate attackers, regulators or malicious users against AI tools; findings drive mitigations, vendor selection and deployment rules. It matters because it exposes real-world risks—prompt injection, data leakage, harmful outputs, not visible in standard QA or accuracy tests.
Under the hood
Red teaming combines threat modelling, adversarial input generation (including prompt-injection and jailbreak techniques), and both black-box and white-box testing to surface failure modes. It uses scenario-based evaluations, metrics-driven Evals, and iterative mitigations (guardrails, filtering, fine-tuning, RLHF) to reduce operational risk.
Example
"We should schedule a red teaming exercise on the LLM integration to check for prompt injection and data-exfiltration risks before launch."