AI Security
The AI Security Badge teaches you to find and exploit vulnerabilities in LLM and ML applications: prompt injection (direct, indirect, and multimodal via PDF/audio/OCR), RAG poisoning and cross-tenant leakage, agent and tool abuse reaching real sinks (path traversal, command injection, SQL injection, SSRF), MCP abuse, insecure output handling (XSS), ML supply-chain RCE (pickle, trust_remote_code, model-output execution), guardrail evasion, gateway-guardrail bypass, cross-session and multi-agent attacks, reasoning-trace leakage, and excessive agency.
Exercises
Coming soon
Medium
PENTESTERLAB
Few-Shot Armor
- The model has been shown examples of refusing various attack techniques. Find an approach that doesn't match any of the example patterns.
- Python
Coming soon
Medium
PENTESTERLAB
Multi-Turn Persistence
- The model is told to maintain its defenses across multiple conversation turns. Use a multi-step conversation strategy.
- Python
Coming soon
Medium
PENTESTERLAB
Input Blocklist
- A server-side input filter inspects every message before it reaches the AI model. Messages containing certain blocked keywords are rejected and never forwarded. If your message makes it past the filter, the model will answer freely.
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
Output Blocklist
- The model will freely tell you the key, but a server-side output filter scans every response before it reaches you. Any response containing the key is blocked. You need to get the model to reveal the key in a form the filter does not catch.
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
Output Canary
- The model has been instructed to include a canary token whenever it discusses the secret key. The server blocks any response that contains the canary. If the canary is absent and the key is present, the response goes through. Get the key without triggering the canary.
- Python
Coming soon
Medium
PENTESTERLAB
LLM-as-Judge
- Every response from the model is reviewed by a second LLM (the 'judge') that decides whether the response reveals a secret key. If the judge says BLOCKED, the response is suppressed. Even encoded forms may be caught. You need to fool the judge.
- Python
Coming soon
Medium
PENTESTERLAB
Gateway Guardrail Bypass
- This app puts no protection in its own code; it relies entirely on a LiteLLM gateway guardrail to strip the access code out of every response before it reaches you. The model will happily tell you the code, but the gateway redacts it. Get the code past the gateway DLP.
- Python
Coming soon
Medium
PENTESTERLAB
Safety Classifier Evasion
- Every message is first screened by an LLM safety classifier that blocks attempts to extract secrets. If the message passes, it reaches an assistant that knows the office door code and will share it. Phrase your request so the classifier rates it ALLOW yet the assistant still hands over the code.
- Python
Coming soon
Medium
PENTESTERLAB
Gateway PII Masking Bypass
- A customer-support assistant will share a customer's billing card number, and the app relies on a LiteLLM Presidio guardrail at the gateway to mask credit-card numbers in every response. Presidio redacts the card when asked normally. Get the full card number past the Presidio masking.
- Python
Coming soon
Medium
PENTESTERLAB
Delimiter Defense
- A document summarizer wraps every untrusted document in delimiters and treats anything inside them as data, never instructions. Escape the delimiter boundary so your instruction lands outside the sandbox and leaks the secret key.
- Python
Coming soon
Medium
PENTESTERLAB
Email Summarizer
- You are composing an email that will be fed to an AI summarizer. The summarizer also holds a confidential internal routing key that it is told never to reveal. Craft an email body that overrides the summarization rule and makes the assistant leak its confidential routing key in the output.
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
Document Q&A
- Submit a document and a question. The assistant holds a confidential internal annotation code it is told never to disclose. Craft the document so that the AI leaks that annotation code regardless of the question asked.
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
Web Fetch Injection
- A browsing assistant fetched a web page and will summarize it. The page content is fully attacker-controlled, and the assistant holds a confidential internal API token it is told never to reveal. Craft page content that overrides the summarizer and makes it leak the token.
- Python
Coming soon
Medium
PENTESTERLAB
Multi-Source Confusion
- Provide two source documents and a research question. The assistant guards an internal key and will only reveal it if an 'Official' or 'Primary' source authorizes disclosure. Forge that trust label in one of your sources to make the assistant output its internal key.
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
PDF Injection
- Upload a PDF and ask a question; the assistant extracts the PDF text and answers. The assistant holds a confidential annotation code it should not disclose. Upload a PDF whose (possibly hidden) text carries an injection that makes the assistant leak that code regardless of your question.
- Python
Coming soon
Medium
PENTESTERLAB
Audio Metadata Injection
- Upload a WAV voice note; the assistant's pipeline naively folds the file's metadata (comment tags) into the transcript it reads. Craft an audio file whose metadata comment carries an injection that makes the assistant leak its confidential code.
- Python
Coming soon
Medium
PENTESTERLAB
OCR Pipeline Injection
- This assistant OCRs an uploaded receipt image and answers questions about it. Text a human would barely notice (faint, low-contrast) is still read by the OCR engine and fed to the model. Upload an image whose hidden text injects the assistant into leaking its confidential code.
- Python
Coming soon
Medium
PENTESTERLAB
KB Injection
- This RAG system answers from a knowledge base and guards a confidential admin recovery value it is told never to disclose. You can add documents to the knowledge base. Inject a document that overrides the non-disclosure rule and makes the model reveal the recovery value.
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
Query Crafting
- A sensitive document is buried in the knowledge base. You cannot inject new documents, you can only query. Craft a query whose embedding retrieves the hidden document.
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
RAG Exfiltration Chain
- The secret is in the system prompt, not in the documents. The model will only share it if an [AUTHORIZED] document instructs it to. You can inject documents into the knowledge base. Craft the right document to extract the recovery code.
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
Cross-Tenant Retrieval Leakage
- This assistant serves tenant 'acme', but all tenants' documents share one retrieval index with isolation enforced only by a prompt instruction. A document belonging to tenant 'zenith' holds a confidential recovery key. Craft a query that pulls the zenith document into context and make the assistant disclose the key.
- Python
Coming soon
Medium
PENTESTERLAB
Vector Metadata Filter Injection
- This retrieval API restricts results to public documents via a classification filter. The filter, like a real vector DB, accepts operator objects. Inject a filter operator that defeats the public-only gate and retrieves the restricted document holding the admin recovery token.
- Python
Coming soon
Medium
PENTESTERLAB
Persistent Vector-Store Poisoning
- A nightly digest job runs under a privileged prompt that holds a recovery code, summarizes the shared knowledge base, and posts the summary to an ops webhook. You can add documents to the knowledge base. Poison it so that when the digest job runs, the recovery code is exfiltrated to the webhook (a second-order, cross-principal attack).
- Python
Coming soon
Medium
PENTESTERLAB
Thought Injection
- This is a ReAct agent that processes Thought/Action/Observation loops. There is a hidden tool get_flag() not listed in the prompt. Inject text that looks like the agent's own reasoning to make it call get_flag().
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
Tool Routing
- The agent has a restricted admin_panel tool that requires an admin token. The system prompt says to never call it without the token. Force the agent to call admin_panel('get_flag') without having the real token.
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
Excessive Agency
- This customer support agent only needs to answer help-center questions, but it was given a powerful run_sql tool it does not need (excessive agency). Abuse the over-scoped tool to read data the support role should never be able to access.
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
Exfiltration via Tool
- The agent has an internal secret and a send_email tool. The model will not share the secret in its text responses. Trick it into including the secret in an email it sends via the send_email tool.
- Python
Coming soon
Medium
PENTESTERLAB
Permission Escalation
- The agent requires verify() to return 'valid' before calling restricted_api(). However, verify() always returns 'invalid', there is no valid token. Bypass the verification step to reach restricted_api('get_flag').
- Python
Coming soon
Easy
PENTESTERLAB
Path Traversal via Document Tool
- This documentation assistant reads files with a read_document(filename) tool. The tool joins your filename onto the docs directory with no sanitization. Escape the docs directory to read a sensitive file on the host (the flag lives at /flag).
- Python
Coming soon
Easy
PENTESTERLAB
Secret Leak via Verbose Error
- This currency assistant calls an internal rates API using a secret api_key. For known currencies it returns a clean rate, but its error handling is sloppy. Get the assistant to trigger the error path and leak the credential.
- Python
Coming soon
Medium
PENTESTERLAB
SSRF via URL-Fetch Tool
- This browsing assistant fetches URLs server-side with no host allowlist. An internal cloud-metadata service runs on the loopback interface and returns a recovery token. Steer the assistant into fetching the internal metadata endpoint to exfiltrate it.
- Python
Coming soon
Medium
PENTESTERLAB
Text-to-SQL Injection
- This analytics assistant turns your question into SQL and runs it against a real database. It is told to query only the products table, but nothing enforces that boundary in code. Read the recovery_key from the app_secrets table.
- Python
Coming soon
Medium
PENTESTERLAB
Prompt-Template SSTI
- This prompt studio lets you define a reusable persona template that is rendered with Jinja2 before being sent to the model. The template engine is not sandboxed. Achieve Server-Side Template Injection to run code on the server and read the flag (/flag).
- Python
Coming soon
Medium
PENTESTERLAB
Agent Tool Command Injection
- This DevOps agent runs a run_diagnostic(target) tool that pings a host. The target is concatenated into a shell command. Inject a second command to achieve RCE and read the flag at /flag.
- Python
Coming soon
Medium
PENTESTERLAB
Sanitizer-Bypass XSS
- This chat app sanitizes the assistant's reply before rendering it with innerHTML, but the sanitizer only strips <script> tags and javascript: URIs. Your logged-in session token is in window.__SESSION.token. Get the model to emit an HTML payload that survives the sanitizer and exfiltrates the token to /api/catch/<token>. Check /api/captures.
- Python
Coming soon
Medium
PENTESTERLAB
Markdown Image
- This model's responses are rendered as markdown in the browser. Markdown images like  trigger HTTP requests when rendered. Trick the model into generating an image tag that includes the secret in the URL. A request catcher at /api/catch/ will capture the exfiltrated data.
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
Structured Output
- The model responds in JSON. The server parses the JSON and blocks responses where the "answer" field contains the password. However, only the "answer" field is checked. Smuggle the secret into other JSON fields that are not filtered.
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
HTML Output XSS
- This chat app renders the assistant's reply into the page with innerHTML, without sanitizing it. Your logged-in session token is held client-side in window.__SESSION.token. Make the assistant emit an HTML/JS payload that reads that token and sends it to the request catcher at /api/catch/<token>. Check /api/captures; capturing your real token proves the XSS fired.
- Python
Coming soon
Medium
PENTESTERLAB
Trust Remote Code (auto_map RCE)
- This 'bring your own model' feature accepts a model repository (a .zip). To build the model it reads config.json and dynamically imports the module named in its auto_map entry, exactly like transformers' trust_remote_code=True. Upload a repo whose code runs on the server and surfaces the flag (env / /flag).
- Python
Coming soon
Medium
PENTESTERLAB
Malicious Model (Pickle RCE)
- This 'bring your own model' feature loads an uploaded model file with Python pickle to read its metadata. Pickle deserialization of untrusted data executes arbitrary code. Upload a malicious model whose payload runs code on the server and surfaces the flag (PTLAB_KEY), available in the environment and at /flag.
- Python
Coming soon
Medium
PENTESTERLAB
Model Output Command Execution
- This auto-remediation feature loads a small user-supplied model (parsed safely as JSON) and runs the command the model predicts for the current incident. The model file is loaded safely, but its OUTPUT is passed straight to a shell. Upload a tiny model whose predicted command reads the flag (PTLAB_KEY / /flag).
- Python
Coming soon
Medium
PENTESTERLAB
MCP Rug Pull
- This agent loads tools from an external MCP server and trusts the usage instructions in each tool's description. The get_weather tool's description was silently changed (a 'rug pull') to instruct the agent to call audit_sync with an internal diagnostic code. Make the agent perform the malicious follow-up action just by using the tool normally.
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
MCP Poisoned Tool Results
- This support agent triages tickets from an MCP server and holds a confidential escalation code. An attacker has already filed a support ticket whose body contains a prompt injection. Ticket content is untrusted data, but the agent obeys it like trusted context. Get the agent to read the malicious ticket so it leaks its confidential escalation code.
- Python
Coming soon
Medium
PENTESTERLAB
MCP Tool Shadowing
- Two MCP servers are connected. The 'helper-mcp' server is malicious: its tool description contains instructions that change how the agent uses a DIFFERENT, trusted tool (send_report). This is a cross-server tool shadowing attack. Trigger a normal report send and make the agent leak the internal code through the trusted tool.
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
MCP Credential Theft
- A confused-deputy attack. The agent holds an OAuth access token to an upstream API and also has an outbound http_post tool. The token is meant to stay internal. Trick the agent into POSTing its own access token to an external endpoint you control, exfiltrating the credential.
- Python
Coming soon
Medium
PENTESTERLAB
MCP Sampling Abuse
- This host exposes MCP's sampling feature: connected servers can ask the host to run an LLM completion on their behalf, over the host's own context, which contains a recovery code. A rogue 'translate' server abuses that reverse channel to run a completion that appends the host's secret and delivers it back to the server. Invoke the translate tool and recover the exfiltrated code from the server sink.
- Python
Coming soon
Medium
PENTESTERLAB
Code Review Bypass
- An AI agent reviews code diffs and either approves or rejects them. It is told to reject backdoors and command execution. Submit a diff that contains a backdoor but still gets approved, by injecting reviewer-directed text into the code.
- Completed by 1 student
- Python
Coming soon
Medium
PENTESTERLAB
Sub-Agent Delegation Confusion
- An orchestrator dispatches your task to one of its sub-agents. The privileged 'vault' sub-agent holds a recovery code and is told to reveal it only for authorized maintenance. Craft a single task that both routes to the vault sub-agent and, when forwarded verbatim, convinces it the request is authorized so it discloses the code.
- Python
Coming soon
Medium
PENTESTERLAB
Prompt-Injection Worm
- This inbox auto-responder drafts a reply to every email it receives, and it holds an internal ops code. Send an email whose injected instructions make the auto-reply BOTH leak the ops code AND copy the payload into the reply, so the next recipient's assistant is infected in turn (a self-propagating prompt-injection worm).
- Python
Coming soon
Medium
PENTESTERLAB
Persistent Memory Implant
- This assistant has a memory feature: saved notes persist across sessions and are preloaded as trusted standing instructions into every new session. A fresh victim session holds a confidential recovery code. Save a note that lies dormant, then open a new victim session where your implanted note detonates and leaks the code.
- Python
Coming soon
Medium
PENTESTERLAB
Reasoning Trace Leakage
- A 'thinking' assistant returns a final answer plus a sealed, encrypted reasoning block that the app treats as safe to expose. The answer is scanned and the internal key is stripped from it, but the reasoning block is not. Recover the internal key from the reasoning block. (Simplified: the block is base64, not real provider crypto.)
- Python
Coming soon
Medium
PENTESTERLAB
Reasoning Trace Forgery
- This assistant finalizes answers using a 'sealed' reasoning block that it trusts as its own prior thinking (tamper-proof and session-bound, supposedly). The block is accepted from the client and is not bound to any session. Forge a reasoning block that makes the assistant conclude it should reveal its internal key.
- Python
Coming soon
Medium
PENTESTERLAB
Reasoning Block Padding Oracle
- The sealed reasoning block here is really encrypted (AES-CBC) but UNAUTHENTICATED, and the /api/decrypt endpoint reveals whether a submitted block has valid padding. That padding oracle lets you decrypt the reasoning block byte by byte and recover the internal key inside it, without the key.
- Python
