Description
The product invokes a generative AI/ML component whose behaviors and outputs cannot be directly controlled, but the product does not validate or insufficiently validates the outputs to ensure that they align with the intended security, content, or privacy policy.
Potential Impact
Integrity
Execute Unauthorized Code or Commands, Varies by Context
Mitigations & Prevention
Since the output from a generative AI component (such as an LLM) cannot be trusted, ensure that it operates in an untrusted or non-privileged space.
Use "semantic comparators," which are mechanisms that provide semantic comparison to identify objects that might appear different but are semantically similar.
Use components that operate externally to the system to monitor the output and act as a moderator. These components are called different terms, such as supervisors or guardrails.
During model training, use an appropriate variety of good and bad examples to guide preferred outputs.
Detection Methods
- Dynamic Analysis with Manual Results Interpretation — Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.
- Dynamic Analysis with Automated Results Interpretation — Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.
- Architecture or Design Review — Review of the product design can be effective, but it works best in conjunction with dynamic analysis.
Real-World CVE Examples
| CVE ID | Description |
|---|---|
| CVE-2024-3402 | chain: GUI for ChatGPT API performs input validation but does not properly "sanitize" or validate model output data (CWE-1426), leading to XSS (CWE-79). |
Related Weaknesses
Frequently Asked Questions
What is CWE-1426?
CWE-1426 (Improper Validation of Generative AI Output) is a software weakness identified by MITRE's Common Weakness Enumeration. It is classified as a Base-level weakness. The product invokes a generative AI/ML component whose behaviors and outputs cannot be directly controlled, but the product does not validate or insufficiently validates the outputs to ensure...
How can CWE-1426 be exploited?
Attackers can exploit CWE-1426 (Improper Validation of Generative AI Output) to execute unauthorized code or commands, varies by context. This weakness is typically introduced during the Architecture and Design, Implementation phase of software development.
How do I prevent CWE-1426?
Key mitigations include: Since the output from a generative AI component (such as an LLM) cannot be trusted, ensure that it operates in an untrusted or non-privileged space.
What is the severity of CWE-1426?
CWE-1426 is classified as a Base-level weakness (Medium abstraction). It has been observed in 1 real-world CVEs.