{"id":96882,"date":"2024-07-31T13:04:45","date_gmt":"2024-07-31T13:04:45","guid":{"rendered":"https:\/\/staging.checkmarx.com\/?post_type=learn&p=96882"},"modified":"2026-04-15T22:35:08","modified_gmt":"2026-04-15T20:35:08","slug":"the-complete-guide-to-ai-application-security-testing","status":"publish","type":"learn","link":"https:\/\/checkmarx.com\/learn\/appsec\/the-complete-guide-to-ai-application-security-testing\/","title":{"rendered":"AI Security Testing: Safeguarding DevSecOps in the Age of GenAI and LLMs"},"content":{"rendered":"
\n

Summary: <\/b>The use of LLMs and other AI advances have changed the way developers generate code and deploy applications, but they don\u2019t come without risk. This article is a deep-dive into the <\/span>application of AI in cyber security<\/span> to support DevSecOps teams in leveraging GenAI and LLMs, while reducing the growing risk landscape associated with AI.\u00a0<\/span><\/p>\n<\/blockquote>\n

The <\/span>application of AI in cyber security<\/span> is a huge topic, and one that\u2019s changing all the time. In this guide – we will look specifically at artificially intelligent (<\/span>AI) application security <\/span>solutions, and click down into areas such as <\/span>AI-driven software composition analysis<\/span>, the <\/span>application of AI in SAST (Static Application Security Testing)<\/span>, and the <\/span>application of AI in security<\/span> where it relates to generating and deploying more secure code at every stage of the Software Development Lifecycle (SLDC).\u00a0<\/span><\/p>\n

Going deeper into practical strategies and products, this guide will also discuss a variety of tools that leverage AI, including SAST tools, AI tools for IaC (Infrastructure as Code) and Checkmarx features that provide SCA (Software Composition Analysis) and describe how they can reduce risk while supporting DevOps and AppSec teams in keeping up with the pace of development.\u00a0<\/span><\/p>\n

\"Chart

“2024 Appsec Executive Survey:
What is your level of concern about security threats stemming from developers using AI code generation tools to write code?”<\/p><\/div>\n

If you\u2019ve been asking yourself, \u2018What are the risks of the growing use of AI in application security?\u2019, \u2018Where does GenAI come into the picture?\u2019, and \u2018How can teams leverage AI without adding risk factors to their organizations?\u2019 you\u2019re in the right place!\u00a0\u00a0<\/span><\/p>\n

\nThe Rise in the <\/span>Application of AI in Cybersecurity<\/span>\n<\/h2>\n

The truth is, the <\/span>application of AI in security<\/span> is nothing new. The global market for AI in cyber security is <\/span>predicted to hit 133.8 billion by 2030.<\/span><\/a> AI tools enable cyber security professionals to free up time from being weighed down by manual tasks, and work in a smarter and more efficient way to face the growing number of threats organizations are facing. AI can already take on roles across varied areas of cyber security such as threat detection, data analysis, automating manual tasks, alerts, reporting and more.\u00a0<\/span><\/p>\n

More recently, GenAI, a subset of AI that involves creating new content such as images, text or code using complex algorithms and models, has stolen many a headline. This branch of AI that focuses on creating original and creative outputs – rather than simply analyzing or classifying existing data, is behind the Large Language Models (LLMs) that are increasingly prevalent, including ChatGPT, Bard, Copilot, and more. LLMs for developers focus on generating code, making it quicker and easier for Dev teams to do their work.\u00a0<\/span><\/p>\n

However, as development teams increase their use of AI to speed up their work, as well as create their own LLMs to enhance their products, security concerns need to be front and center. Organizations should take some time to consider how they can support application security teams with making security a core part of both application development and deployment.\u00a0<\/span><\/p>\n

Already, <\/span>82.5% of developers<\/span><\/a> are using GenAI to write code, and 42% say they trust the output of LLMs, with just 31% holding back and saying they are unsure. 77% of respondents are happy to say that they like the experience of using AI tools, and up to 80% of developers recognize that their development workflow will look extremely different 12 months from now – as a direct result of AI tools. There\u2019s no doubt that AI-based code generation is the future, from writing code itself, to getting feedback and troubleshooting during development.\u00a0<\/span><\/p>\n

But before rushing into a new way of working, it\u2019s critical for businesses to understand the risks. How does AI work, how can LLMs do so much so quickly, and what threats should security teams be aware of before giving developers the freedom to use AI in their day-to-day work?\u00a0<\/span><\/p>\n

Understanding the Risks of GenAI for Development Teams\u00a0<\/span><\/h2>\n

To help fully grasp the risks involved in using LLMs, let\u2019s look to the <\/span>OWASP Top 10 for LLMs<\/span><\/a>, a list of the most critical vulnerabilities found in applications that are utilizing LLMs. This list is a practical support to guide developers and security professionals in understanding the threat landscape, whether they are leveraging AI and LLMs in their work, or whether they are building LLMs as part of their product or service. It categorizes the challenges businesses will face to help them to implement the right security tools to close gaps in visibility and control, and to enable AppSec velocity while securing the new ways of working.\u00a0<\/span><\/p>\n

1. Prompt injection<\/span><\/h3>\n

Threat actors can manipulate an LLM using unique prompts that make the LLM execute malicious intentions. This can be done directly, usually called jailbreaking – where the underlying system prompt is overwritten or exposed, which means the attacker can access data or backend systems.<\/span><\/p>\n

\"Stat<\/p>\n

It can also happen indirectly, when an LLM accepts input from other sources, which could be controlled by an attacker, and may not even be recognizable to the human engaging with the LLM. Prompt injection can lead to data leakage, social engineering attacks, and more.\u00a0<\/span><\/p>\n

2. Insecure output handling<\/span><\/h3>\n

LLM-generated content is controlled by prompts, which means there can be challenges in validating information, sanitizing data, and even with threat actors injecting malicious content before it\u2019s passed downstream. This can result in remote code execution on backend systems, privilege escalation, or CSS and CSRF in web browsers. In some cases, third party plugins may not validate LLM input at all, or the LLM has privileges which it does not need, making this threat even more of a risk.\u00a0<\/span><\/p>\n

3. Training data poisoning<\/span><\/h3>\n

All LLMs use raw data, or training data, to learn what they need to know to generate the right outputs, and also use fine tuning data to narrow down an LLMs remit. Using neural networks, LLMs learn to find patterns and create outputs based on their training data. However, if attackers manipulate the original data, the model can be compromised or contain vulnerabilities. Users may automatically trust an LLM, when actually it is surfacing malicious or incorrect information. Training data poisoning is a kind of integrity attack, which is more prevalent when a model is training from external data sources where the LLM owners do not have control over the data.\u00a0<\/span><\/p>\n

4. Model denial of service<\/span><\/h3>\n

Similar to a traditional Denial of Service attack, where a target is overwhelmed by an exceptionally high amount of traffic, a model denial of service attack is when an attacker consumes enough resources from an LLM to reduce service quality or result in a high level of expense for the owner. One approach is to manipulate the context window – which is the maximum length of the text that an LLM can work with, and tightly linked to the complexity an LLM can manage. Attackers can attempt a model denial of service attack by sending unusual queries that are resource-intensive, continuous input overflow, repetitive long inputs, and forcing recurring resource usage through tasks in a queue.\u00a0<\/span><\/p>\n

5. Supply chain vulnerabilities<\/span><\/h3>\n

There are many players involved in the world of GenAI and LLMs, including third parties who provide pre-trained models or training data, as well as LLM plugin extensions. Unlike with traditional supply chain vulnerabilities, machine learning vulnerabilities do not need to rely on software components, as developers often rely on third party downloads and packages, without considering the wider risk. Threats in this category include poisoned crowd-sourced data, vulnerable pre-trained models, a and the use of deprecated or outdated models.\u00a0<\/span><\/p>\n

6. Sensitive information disclosure<\/span><\/h3>\n

In today\u2019s compliance-heavy landscape, personally identifiable information needs to be protected at all costs. However, LLMs can reveal sensitive information, including customer data, algorithms, or trade secrets and intellectual property. If a user unintentionally inputs information, it can resurface at another time. However, many users leverage LLMs without data sanitization, which stops user data becoming further training data. <\/span><\/p>\n

\"biggest<\/p>\n

Security teams should look out for incomplete filtering of sensitive information in responses, unintended disclosure of confidential information, and memorization of sensitive data in the training process.\u00a0<\/span><\/p>\n

7. Insecure plugin design<\/span><\/h3>\n

For added agility, LLM plugins can be automatically called by a model, usually without the risk reduction benefit of application control. According to OWASP, they often have free-text inputs, without validation or type checking, which means an attacker could even send configuration strings instead of parameters. A threat actor can use this vulnerability to create a malicious request, forcing their agenda through the model directly, including privilege escalation, data exfiltration, and even remote code execution. Tracking authorization across plugins is critical, as you cannot simply assume that the inputs were sent by the end user.\u00a0<\/span><\/p>\n

8. Excessive agency<\/span><\/h3>\n

LLMs are not predictable, we all know that by now. When LLMs present an unexpected or ambiguous output, the controls may not be in place to restrict what happens next. This is usually across three categories, the functionality of the LLM, the permissions it has, or the autonomy it holds. Either way, the developer has provided it with too much agency. Examples of each include:<\/span><\/p>\n