AI systems are becoming increasingly core to the products and services we engage with day-to-day, optimising everything from customer service to cybersecurity tooling and automated testing. While AI and LLMs present exciting possibilities for both the present and future of how we interact with businesses, they also introduce unique attack surfaces via APIs and integrations.
LLM red teaming is crucial for discovering vulnerabilities before adversaries do, empowering your business to provide AI-augmented services while minimising the possibility of LLM vulnerabilities falling into the wrong hands.
This blog provides a step-by-step framework for red teaming LLMs, with tools and scenarios of likely attack techniques to support you in getting started.
Why red teaming matters for AI security
Red teaming matters for AI security because it differs significantly from traditional testing. While both simulate adversarial attacks to assess your business’s cybersecurity posture, LLM red teaming has a different methodology.
This is because AI has a different composition from traditional networks. It is significantly more dynamic and probabilistic, which means that a lot more can be done with it: for better or worse. Prompt injections, data leakage, and unsafe outputs all pose major risks to AI models, with data poisoning or model tampering also posing further risk to the model’s integrity and reliability.
Unlike traditional penetration testing that focuses on static systems, red teaming AI systems involves simulating adversarial attacks that exploit the model’s unique vulnerabilities, such as prompt injections that manipulate system prompts or jailbreaks that bypass safety filters. LLM red teaming blends established cybersecurity disciplines like network engineering, reverse engineering, and social engineering to identify vulnerabilities and evaluate LLM outputs under real-world conditions.
How it fits into the AI assurance pipeline
Red teaming LLM also fits into an overall AI assurance pipeline by providing incident detection and response mechanisms. This is critical in signalling to both clients and business partners that your AI use is monitored and audited, assuring that their data is safe and not at risk of exploitation via LLM-based attacks.
Core attack surfaces in LLMs
As with any component of your cybersecurity, LLMs are susceptible to unique attack techniques and exploitations. While there is a broad range of more general attack methods that could be used against your LLM by malicious actors, these are the core ones to educate yourself on to approach LLM red teaming effectively.
Prompt injection and jailbreaks
Prompt injection and jailbreaks are critical attack vectors in LLM red teaming. Direct prompt manipulation involves crafting inputs that override or alter the system prompt, potentially causing the model to bypass safety filters and produce unsafe or disallowed content.
For example, an attacker might use a cleverly worded prompt to trick the LLM into ignoring its ethical guidelines and generating harmful outputs.
Data exfiltration and leakage
Data exfiltration involves unauthorised extraction of sensitive information from systems. AI models can accidentally reveal private information they learned during training.
In many instances, an attacker will use carefully crafted prompts to trick an AI model into reproducing memorised sensitive information from its training data, such as personal details, which they can then exploit for personal or financial gain.
Model misuse and output manipulation
Model misuse includes generating malicious code or disallowed content, posing serious risks. For example, an attacker might craft prompts that trick the LLM into producing harmful instructions or offensive material. Such misuse can lead to security breaches, reputational damage, and compliance violations for your business, harming customer trust and brand loyalty.
Over-reliance and business logic risks
LLMs can make mistakes or “hallucinate” false information. Using their output directly for money decisions, medical care, or security without human checks is dangerous, as they have the potential to provide contradictory or even completely false information. Regular LLM security testing minimises the risk of this occurring by evaluating the model’s ability to retrieve accurate information.
Step-by-step guide to red teaming an LLM
It’s important to ensure your security team’s red teaming efforts are both structured and implemented efficiently to achieve useful results. To ensure your LLM red teaming process flows well and supports you to identify risks, it’s best to use the methodology below as a rule-of-thumb.
Define threat scenarios
Align with business risks (misinformation, data leaks, harmful outputs) by identifying specific vulnerabilities relevant to your industry and use cases. This ensures red teaming focuses on realistic and impactful threats, helping to prioritise mitigation efforts effectively.
Threat intelligence tools can help your business detect initial vulnerabilities and identify priority areas for your red teaming efforts. OnSecurity’s threat intelligence tool, Radar, offers a competitive edge by spotting threats before attackers do. With its flexible configuration, you can choose which threat intelligence and web scanning features to activate for each domain or asset, including options to filter out irrelevant or noisy data.
Set up tools and environment
Setting up the tools and environment for LLM red teaming involves utilising specialised adversarial testing prompt libraries and custom scripts tailored for AI systems. It is important to combine automated testing frameworks with manual testing approaches to harness the benefits of both efficiency and human expertise. The environment should support safe execution of tests, including comprehensive logging and monitoring to capture model behaviour and identify vulnerabilities.
Craft and execute attacks
Once you’ve set up the tools and optimised your environment, it’s time to start designing and actually executing the attacks against your LLM systems. Otherwise known as ‘prompt engineering’, this is the process of testing vulnerabilities within your AI through specific prompt injections and crafted inputs. By simulating attacks such as prompt injections and code manipulations, your team aims to uncover security vulnerabilities that could cause the LLM to expose sensitive information.
You should also test if your system is vulnerable to SQL injection, a common and dangerous attack method where malicious SQL code is inserted into input fields to manipulate or extract sensitive data from your database, potentially compromising the entire system’s security.
Analyse outputs and risks
AI systems can fail in three key ways: safety (causing harm through incorrect responses), security (being exploited by attackers), and compliance (violating regulations or policies).
Automated scoring frameworks help organisations monitor these risks by rating outputs against predefined criteria. For example, scoring content toxicity (safety), detecting prompt injections (security), or flagging non-compliant advice (compliance) – enabling systematic risk management.
It’s essential to assess the security of your LLM during this process by analysing outputs and risks to inform future security decisions and evaluate whether further AI guardrails are necessary.
Remediate and retest
Once testing is complete and vulnerabilities in your LLM have been identified, it’s crucial to address and fix these issues effectively. Solutions such as implementing guardrails on your AI can help filter inputs and block harmful requests, minimising the risk of personally identifiable information being extracted by hackers.
Fine-tuning your LLM can also offer significant improvements by adapting foundational large language models through specialised security-focused training. This process transforms foundational models into an expert assistant capable of analysing threats, explaining attack methods, generating security policies, and supporting incident response, going beyond providing generic responses.
Alongside remediations, it’s imperative that your security team continuously retest your LLM systems. Retesting ensures that the edits you have made have actually been effective in fixing issues flagged by your red teaming. Investing in an external pentest vendor to retest your LLM is also an excellent alternative method to receiving insightful feedback on the efficacy of your remediations, and may even flag further security issues your team missed.
Best practices for continuous AI red teaming
To ensure best practice in red teaming LLMs, we recommend the following:
Integrate red teaming into CI/CD pipelines:
- Automate regular testing to catch vulnerabilities early
- Streamline security within your development workflow
- For more information, why not take a look at: Penetration Testing in CI/CD
Combine automated and manual red teaming:
- Use automated tools for scalable generation of adversarial prompts and attacks
- Employ an external human red-team vendor to uncover nuanced and complex vulnerabilities
- Achieve comprehensive coverage by leveraging both approaches
Monitor emerging threats:
- Track jailbreak marketplaces and AI security forums to stay in the loop
- Stay updated on new attack methods targeting LLMs and generative AI
- Adapt defences proactively to evolving risks, considering business logic threats relevant to your industry
Document findings thoroughly:
- Record failures and vulnerabilities for compliance and audits
- Maintain detailed reports to support responsible AI practices
- Provide transparency and accountability for security teams and stakeholders
By following these best practices, you can effectively improve model resilience and build trust with users and regulators alike.
LLM red teaming, done right.
Don’t let your LLMs leave your organisation at risk of being hacked.
For consultative, real-world validation and proof of secure deployment readiness, look no further than OnSecurity’s LLM red teaming service. We identify weaknesses within your AI tooling by using highly complex simulated attacks and prompt engineering, providing you with real-time insights into your security posture through direct communication with our pentest team.
