Package: Invicti AppSec Core (on-demand)
LLM-based app vulnerability testing
Identify security vulnerabilities unique to AI-enabled systems using the LLM security scan profile in Invicti AppSec Core. This profile focuses on weaknesses in chatbots, AI assistants, and other applications that integrate with language models.
This document provides technical details about:
- What prompts and payloads are used
- What specific tests are executed
- How to verify that LLM security tests were successfully injected
- Detection and confirmation methods
Why this matters
LLM-powered applications introduce a class of vulnerabilities that traditional web application scanners aren't designed to catch. Using the LLM security scan profile, you can identify AI-specific weaknesses - such as prompt injection and insecure output handling - before attackers exploit them, and get confirmation of exploitability rather than just theoretical risk.
Overview of LLM security testing
LLM security testing in Invicti AppSec focuses on web applications that incorporate generative AI components.
Target applications
LLM security testing is designed for web applications that integrate with:
- AI chatbots embedded in web interfaces
- Virtual assistants for customer support
- Content generation tools powered by language models
- AI-powered search and recommendation systems
- Code generation interfaces that use LLM capabilities
- Document processing tools with AI summarization
Configure an LLM scan
To scan LLM-powered applications for AI-specific vulnerabilities:
- Select DAST Scans > New scan from the left-side menu.
- Select LLM security as the scan profile.
- Configure your target URL - the web application that includes LLM functionality.
- Configure additional scan settings as needed.
The LLM security scan profile automatically detects and tests AI-powered components within your target application, including chatbots, virtual assistants, and other LLM integrations.
For detailed information about creating and configuring scans, see the New scan document.
Testing approach
The LLM security scan profile uses Invicti AppSec's DeepScan engine to perform sophisticated analysis of LLM-powered endpoints. Unlike traditional web application testing, LLM security testing requires:
- Contextual understanding of conversational interfaces
- Prompt generation using predefined payloads and patterns
- Behavioral analysis to detect AI model manipulation
LLM response detection and analysis
Before testing for vulnerabilities, Invicti AppSec first establishes that it's communicating with an actual LLM by analyzing response patterns and formats. The scanner identifies and parses various response types:
- Server-Sent Events (SSE): Streaming responses in text/event-stream format
- JSON responses: Structured API responses containing LLM output
- Plain text responses: Direct text-based LLM outputs
- Streaming responses: Various streaming formats used by LLM applications
This detection phase ensures that the scanner accurately identifies LLM interfaces and understands how to extract and analyze the actual AI-generated content from various response formats.
LLM security vulnerabilities tested
Invicti AppSec tests for the following LLM-specific security issues:
| Vulnerability type | What it tests | Detection method |
|---|---|---|
| Prompt injection | Injecting malicious instructions to manipulate LLM behavior | Response analysis for deviation from intended behavior |
| System prompt leakage | Extracting hidden system prompts or internal instructions | Pattern matching to identify exposed configuration content |
| LLM command injection | Executing system commands through LLM interfaces | Out-of-band (OOB) callback verification |
| LLM-enabled SSRF | Using the LLM to access internal or restricted resources | OOB callback verification and content analysis |
| Insecure output handling | Vulnerabilities in how LLM output is processed and rendered | Analysis of how AI-generated content is sanitized before display |
| Tool usage exposure | Enumerating and misusing tools the LLM has access to | Response analysis for exposed or manipulable tool capabilities |
| LLM fingerprinting | Identifying the specific model and configuration in use | Response pattern analysis for model-specific identifiers |
1. Prompt injection
What it tests: Attempts to manipulate the LLM's behavior by injecting malicious instructions into user inputs.
Test methodology:
- Direct prompt injection: Injecting commands directly into user input fields
- Indirect prompt injection: Using data sources that the LLM might reference
- Role manipulation: Attempts to change the AI's assumed role or permissions
- Context manipulation: Exploiting conversation history to alter behavior
Example attack patterns:
Invicti AppSec tests prompt injection by sending specially crafted prompts that attempt to override the LLM's original instructions. The scanner uses verification techniques to confirm whether the injection was successful.
Typical patterns include:
- Instructions to ignore previous directives
- Commands requesting specific factual information to verify compliance
- Role manipulation attempts
- Context boundary violations
Detection method: The scanner analyzes responses to determine if the LLM followed the injected instructions rather than its original system directives or safety guidelines. Successful prompt injection is confirmed when the model demonstrates it has deviated from its intended behavior.
2. System prompt leakage
What it tests: Attempts to extract the system prompt or internal instructions that guide the LLM's behavior.
Test methodology:
- Direct extraction attempts: Asking the model to reveal its instructions
- Conversational manipulation: Using social engineering to extract system information
- Role reversal techniques: Attempting to make the AI explain its own configuration
Example attack patterns:
Invicti AppSec attempts to extract system prompts through various techniques:
- Direct requests for the model to reveal its initialization instructions
- Social engineering approaches to manipulate the model into sharing configuration details
- Techniques that exploit conversational context to expose hidden directives
- Methods that attempt to bypass confidentiality restrictions
Detection method: The scanner uses pattern matching and content analysis to identify when system prompts or internal instructions have been successfully extracted from the LLM's responses.
3. LLM command injection
What it tests: Attempts to execute system commands or access unauthorized functionality through LLM interfaces.
Test methodology:
- System command injection: Attempting to execute shell commands
- Function call manipulation: Exploiting tool/function calling capabilities
- API access attempts: Trying to access backend APIs through the LLM
- Code execution: Attempting to execute Python or other code
Example attack patterns:
Invicti AppSec tests for command injection through various approaches:
- Attempts to execute system-level shell commands
- Requests to run encoded or obfuscated commands
- Instructions to execute code in various programming languages (Python, Bash, etc.)
- Combined prompt injection with command execution requests
Detection method: The scanner uses out-of-band detection techniques, including Invicti OOB integration, to verify whether commands were actually executed on the backend system. This provides definitive proof of exploitability rather than relying solely on response analysis.
4. LLM-enabled Server-Side Request Forgery (SSRF)
What it tests: Exploits the LLM's ability to make web requests to access internal resources.
Test methodology:
- Internal network scanning: Attempting to access localhost and internal IPs
- Cloud metadata access: Trying to access cloud service metadata endpoints
- Port scanning: Using the LLM to probe internal network ports
- Service discovery: Attempting to identify internal services
Example attack patterns:
Invicti AppSec tests for SSRF vulnerabilities by attempting to make the LLM access restricted resources:
- Requests to fetch content from internal network addresses
- Attempts to access cloud provider metadata endpoints
- Instructions to probe internal services and ports
- Requests to retrieve data from private network resources
Detection method: The scanner uses both out-of-band callback verification (via Invicti OOB) and content analysis to confirm successful SSRF attacks. This dual approach validates that the LLM actually made the request and accessed the target resource.
5. Insecure output handling
What it tests: Identifies vulnerabilities in how LLM outputs are processed and displayed.
Test methodology:
- XSS through LLM output: Injecting scripts that get rendered in the browser
- Template injection: Exploiting server-side template engines
- Code injection: Attempting to inject executable code in LLM responses
- HTML injection: Manipulating page structure through AI-generated content
Example attack patterns:
Invicti AppSec tests how the application handles potentially dangerous content in LLM responses:
- Requests for the LLM to generate HTML with embedded scripts
- Instructions to create responses containing template injection payloads
- Attempts to inject executable code into LLM-generated output
- Requests for malicious markup that could affect page structure
Detection method: The scanner analyzes how LLM-generated content is rendered in the application, checking whether dangerous output is properly sanitized, escaped, or filtered before being displayed to users.
6. Tool usage exposure
What it tests: Attempts to enumerate and misuse tools and functions that the LLM has access to.
Test methodology:
- Tool enumeration: Discovering what tools and functions are available to the LLM
- Parameter discovery: Identifying tool parameters and their expected values
- Unauthorized tool access: Trying to use tools beyond intended scope
- Tool parameter manipulation: Exploiting tool parameters for malicious purposes
- Privilege escalation: Attempting to access higher-privilege tools
Example attack patterns:
Invicti AppSec attempts to discover and exploit LLM tool capabilities:
- Requests for the LLM to list available tools and functions
- Instructions to enumerate tool parameters and capabilities
- Attempts to invoke tools with manipulated or unauthorized parameters
Detection method: The scanner analyzes LLM responses to identify when tools are exposed or can be manipulated, and validates whether tool usage restrictions are properly enforced.
7. LLM fingerprinting
What it tests: Identifies the specific LLM model and configuration being used.
Test methodology:
- Model identification: Determining the specific AI model in use (for example, OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama, Mistral)
- Version detection: Identifying model version and capabilities
- Configuration probing: Discovering model parameters and settings
- Capability enumeration: Mapping available functions and tools
- Response pattern analysis: Analyzing response characteristics to identify the underlying model
Example attack patterns:
Invicti AppSec queries the LLM to reveal its identity and capabilities:
- Direct questions about the model's identity
- Requests for version information
- Capability probing questions
Detection method: The scanner analyzes responses for model-specific identifiers, response patterns, and behavioral characteristics. This information helps understand the attack surface and potential vulnerabilities specific to the identified model.
How tests are executed
The scanner follows a structured four-phase approach to identify and validate LLM-specific vulnerabilities.
1. Application discovery phase
The scanner first identifies LLM-powered components by:
- Analyzing JavaScript: Looking for chatbot frameworks and AI integration code
- Detecting API endpoints: Identifying endpoints that accept conversational input
- Form analysis: Finding text areas and input fields connected to AI processing
- Response pattern matching: Detecting AI-generated content patterns
2. Conversation initiation
Once LLM interfaces are identified, the scanner:
- Establishes sessions: Creates proper conversation contexts
- Tests basic functionality: Verifies the LLM is responsive
- Identifies input validation: Testing what types of input are accepted
3. Vulnerability injection
For each identified LLM endpoint, the scanner:
- Sends crafted prompts: Uses the vulnerability-specific payloads
- Monitors responses: Analyzes AI-generated responses for signs of success
- Tests multiple variations: Uses different phrasings and techniques
4. Response analysis
The DeepScan engine analyzes responses using:
- Pattern matching: Looking for specific indicators of successful injection
- Behavioral analysis: Detecting unusual AI behavior patterns
- Content inspection: Analyzing response content for security issues
- Context validation: Ensuring responses indicate actual vulnerabilities
Troubleshooting
The scanner doesn't detect the LLM interface in my application
The scanner identifies LLM interfaces by analyzing JavaScript, API endpoints, and input fields. If it doesn't detect one, check that the LLM functionality is accessible from the target URL configured in the scan. If the chatbot or AI interface loads dynamically after login or user interaction, ensure the scan is configured with appropriate authentication and that the LLM component is within the scan scope.
The LLM security scan completes but shows no findings
A scan with no findings can mean the application is secure, or that the LLM interface wasn't reached. To verify the scan tested the LLM component, check the scan activity log for evidence of conversational requests to AI endpoints. If no such requests appear, the interface may not have been discovered - see the troubleshooting step above.
I get findings for prompt injection but I'm not sure they're exploitable
Invicti AppSec uses verification techniques to confirm successful injections - findings are reported when the scanner observes the model deviating from its intended behavior, not just when a payload was sent. For command injection and SSRF findings, the scanner also uses out-of-band (OOB) callback verification to provide definitive confirmation of exploitability.
Need help?
Invicti Support team is ready to provide you with technical help. Go to Help Center