These may be improper model functioning, suspicious behavior patterns or malicious inputs. Attackers may also make attempts to abuse inputs through frequency, making controls such as rate-limiting APIs. Attackers may also look to impact the integrity of model behavior leading to unwanted model outputs, such as failing fraud detection or making decisions that can have safety and security implications. Recommended controls here include items such as detecting odd or adversarial input and choosing an evasion-robust model design.
Development-time threats
In the context of AI systems, OWASP’s AI Exchange discusses development-time threats in relation to the development environment used for data and model engineering outside of the regular applications development scope. This includes activities such as collecting, storing, and preparing data and models and protecting against attacks such as data leaks, poisoning and supply chain attacks.
Specific controls cited include development data protection and using methods such as encrypting data-at-rest, implementing access control to data, including least privileged access, and implementing operational controls to protect the security and integrity of stored data.
Additional controls include development security for the systems involved, including the people, processes, and technologies involved. This includes implementing controls such as personnel security for developers and protecting source code and configurations of development environments, as well as their endpoints through mechanisms such as virus scanning and vulnerability management, as in traditional application security practices. Compromises of development endpoints could lead to impacts to development environments and associated training data.
The AI Exchange also makes mention of AI and ML bills of material (BOMs) to assist with mitigating supply chain threats. It recommends utilizing MITRE ATLAS’s ML Supply Chain Compromise as a resource to mitigate against provenance and pedigree concerns and also conducting activities such as verifying signatures and utilizing dependency verification tools.
Runtime AppSec threats
The AI Exchange points out that AI systems are ultimately IT systems and can have similar weaknesses and vulnerabilities that aren’t AI-specific but impact the IT systems of which AI is part. These controls are of course addressed by longstanding application security standards and best practices, such as OWASP’s Application Security Verification Standard (ASVS).
That said, AI systems have some unique attack vectors which are addressed as well, such as runtime model poisoning and theft, insecure output handling and direct prompt injection, the latter of which was also cited in the OWASP LLM Top 10, claiming the top spot among the threats/risks listed. This is due to the popularity of GenAI and LLM platforms in the last 12-24 months.
To address some of these AI-specific runtime AppSec threats, the AI Exchange recommends controls such as runtime model and input/output integrity to address model poisoning. For runtime model theft, controls such as runtime model confidentiality (e.g. access control, encryption) and model obfuscation — making it difficult for attackers to understand the model in a deployed environment and extract insights to fuel their attacks.
To address insecure output handling, recommended controls include encoding model output to avoid traditional injection attacks.
Prompt injection attacks can be particularly nefarious for LLM systems, aiming to craft inputs to cause the LLM to unknowingly execute attackers’ objectives either via direct or indirect prompt injections. These methods can be used to get the LLM to disclose sensitive data such as personal data and intellectual property. To deal with direct prompt injection, again the OWASP LLM Top 10 is cited, and key recommendations to prevent its occurrence include enforcing privileged control for LLM access to backend systems, segregating external content from user prompts and establishing trust boundaries between the LLM and external sources.
Lastly, the AI Exchange discusses the risk of leaking sensitive input data at runtime. Think GenAI prompts being disclosed to a party they shouldn’t be, such as through an attacker-in-the-middle scenario. The GenAI prompts may contain sensitive data, such as company secrets or personal information that attackers may want to capture. Controls here include protecting the transport and storage of model parameters through techniques such as access control, encryption and minimizing the retention of ingested prompts.
Community collaboration on AI is key to ensuring security
As the industry continues the journey toward the adoption and exploration of AI capabilities, it is critical that the security community continue to learn how to secure AI systems and their use. This includes internally developed applications and systems with AI capabilities as well as organizational interaction with external AI platforms and vendors as well.
The OWASP AI Exchange is an excellent open resource for practitioners to dig into to better understand both the risks and potential attack vectors as well as recommended controls and mitigations to address AI-specific risks. As OWASP AI Exchange pioneer and AI security leader Rob van der Veer stated recently, a big part of AI security is the work of data scientists and AI security standards and guidelines such as the AI Exchange can help.
Security professionals should primarily focus on the blue and green controls listed in the OWASP AI Exchange navigator, which includes often incorporating longstanding AppSec and cybersecurity controls and techniques into systems utilizing AI.
