AI Red Teaming: What it Means

Testing AI systems is becoming increasingly crucial as the technology becomes more widely adopted. With AI's increasing role in everything from healthcare to customer service, ensuring these systems are reliable, unbiased, and secure is more important than ever. Recently, the US government introduced a new term that testers should familiarize themselves with: AI red teaming. Let's explore why testing AI is essential and how AI red teaming fits into this landscape.

Why it's important to test AI

AI systems are revolutionizing everything from healthcare and finance to customer service and entertainment. Relying on an AI system to make crucial decisions, like diagnosing medical conditions or approving loans, comes with great responsibility. If these systems fail or produce biased or inaccurate results, the consequences can be severe.

A recent example of AIs causing a company trouble is Air Canada’s chatbot that promised a discount to a passenger, an offer the airline later retracted. This led to a lawsuit which the airline lost, not to mention a lot of bad press because they tried to push the responsibility to the chatbot, which they claimed was "a separate legal entity that is responsible for its own actions".

It’s likely that other businesses who attempt to ‘blame the bot’ will face similar resistance.

Yet, data from Stanford University’s 2024 Artificial Intelligence Index Report showed that people are increasingly looking to push responsibility to those developing the foundation models rather than organizations using them. They would also like to see globally agreed generative AI governance.

According to the report:

• 88% of respondents either agreed or strongly agreed that companies that develop the foundation models will be responsible for the mitigation of all associated risks, rather than organizations using these models/systems.
• 86% of respondents either agreed or strongly agreed that generative Al presents enough of a threat that globally agreed generative Al governance is required.

But let’s be realistic. Angry customers will be angry customers. And nothing feeds the news better these days than stories about AI gone wrong. Businesses who aren’t deploying AI in a responsible way - regardless of whether they created the foundational model or not - will be seeing the side-effects. AIs need to be tested.

“The world of enterprise software is going to get completely rewired. Companies with untrustworthy AI will not do well in the market.” - Abhay Parasnis - Founder and CEO of Typeface

Here are a few reasons why testing AI is so important:

• Ensuring reliability and performance: AI systems need to perform consistently and accurately, especially in critical applications where human lives are at stake (think medical diagnosis or autonomous driving). Testing helps ensure these systems can be trusted to deliver reliable results.
• Identifying and mitigating bias: AI models learn from data, which can include biases. If left unchecked, these biases can lead to unfair or discriminatory outcomes. For instance, an AI system used in hiring might favor certain demographics over others if trained on biased data. Testing helps identify and correct these biases.
• Securing against threats: AI systems can be vulnerable to various attacks, such as adversarial inputs designed to trick the system into making mistakes. Ensuring robust security through rigorous testing is vital to protect against these threats.

What is AI red teaming?

Red teaming is a concept gaining traction in the AI testing community. The term "red teaming" comes from military exercises where opposing forces are designated by colors. Traditionally, the "red team" represents the adversaries or enemy forces, while the "blue team" represents the defending or friendly forces. This concept has been adopted in cybersecurity and other fields to simulate attacks (red team) and evaluate the effectiveness of defenses (blue team), hence the name "red teaming."

In the context of AI testing, red teaming means thoroughly challenging AI systems to uncover vulnerabilities and flaws that might not be apparent through traditional testing methods.

How red teaming applies to AI testing

Though still in its early days, red teaming is a concept we're starting to see put into practice, often by specialized consultants hired specifically for this purpose. These experts bring unique skills and are typically employed in contexts where the stakes are exceptionally high.;

While not every organization using generative AI will employ designated “red teams”, it's a critical approach for those handling sensitive or high-stakes AI applications. That said, every QA team should consider how they are securing their systems.

Here’s how red teaming can be applied:

• Simulating real-world attacks: For example, consider an AI-powered chatbot handling customer inquiries. A red team would simulate various attack scenarios, such as injecting malicious inputs, to test the chatbot's defenses. This ensures the system is resilient against people trying to exploit the system.
• Uncovering hidden biases: AI models can inadvertently learn biases from their training data. Red teams can systematically test the AI to identify any biased behaviors, such as favoring certain demographics, and suggest ways to mitigate these biases.
• Stress testing: Red teams push AI systems to their limits by subjecting them to extreme conditions. For instance, they might flood a system with a high volume of requests to see how it performs under pressure, ensuring it remains stable and reliable.

Implementing red teaming in AI testing

Here are some practical steps to start incorporating red teaming into your AI testing strategy, inspired by the latest recommendations from industry experts and government guidelines:

1. Assemble a diverse red team: Your red team should include AI experts, cybersecurity professionals, domain specialists, and ethical hackers. This diverse mix ensures all potential attack vectors are covered.
2. Set clear objectives: Define what you aim to achieve with red teaming. Are you looking to uncover security vulnerabilities, test for bias, or evaluate performance under stress? Clear objectives help guide the process.
3. Use adversarial testing: Intentionally feed misleading or malicious inputs to the AI to see how it reacts. This helps expose weaknesses and improve the system's robustness. Make sure to read our blog on this topic: Adversarial Testing: Definition, Examples and Resources.
4. Continuous learning and improvement: Red teaming should be an ongoing effort. Regularly update your strategies and defenses based on new threats and vulnerabilities, ensuring continuous improvement. Integrating adversarial testing into your continuous testing setup can be a way to ensure consistent monitoring.

Using AI to test AI

Ways to approach testing of Generative AI systems have begun to emerge. One of those approaches is Leapwork's AI validation capabilities.

A part of Leapwork’s test automation platform, the AI validation capability allows you to compare prompts for your AI with expected outcomes. Whether you're guarding your systems against malicious attacks or simply looking to ensure that your employees get an accurate response from their internal resource bot, this capability can be used to validate that the input given generates a desired response.

The benefit of having this AI-augmented validation step as a part of your automated testing setup, rather than for example hiring a team of red team experts to execute these tests, is that it can run continuously. You need confidence that your AI responds in a certain way today, tomorrow, and the day after that. With an automated instance, you can run your test every 5 minutes if you wanted to and thereby always have confidence in the system.

Download our report, AI and Software Quality: Trends and Executive Insights, to gain a comprehensive understanding of how AI is reshaping software quality. This report offers key insights and actionable solutions to help your business adapt, scale, and consistently deliver exceptional user and customer experiences in today’s AI-driven landscape.

About the authors

Maria Homann has 5 years of experience in creating helpful content for people in the software development and quality assurance space. She brings together insights from Leapwork’s in-house experts and conducts thorough research to provide comprehensive and informative articles. These AI articles are written in collaboration with Claus Topholt, a seasoned software development professional with over 30 years of experience, and a key developer of Leapwork's AI solutions.