Loki Intelligence — Security Briefs · Published · Updated
Prompt Injection Testing Guide
Prompt injection testing checks whether untrusted text can override, confuse, or redirect an AI system away from the developer's intended behavior. The risk is broader than jailbreak prompts in a chat box. Modern AI products read documents, websites, tickets, emails, tool outputs, logs, and memory. Any of those inputs can carry instructions that the model may treat as relevant context. A good test program uses safe, controlled scenarios to verify instruction hierarchy, data boundaries, tool permissions, output handling, and recovery behavior without publishing exploit playbooks or relying on one-off prompt tricks.
By Sweder · Co-Founder · Product
Practical field guide
Prompt injection regression matrix
Start with these controlled cases, record the full workflow trace, and turn every confirmed failure into a regression test. Keep test content non-destructive and inside the approved scope.
| Test case | Boundary under test | Evidence to capture | Pass condition |
|---|---|---|---|
| Direct user instruction | System and developer instruction priority | Trusted policy, user input, model response, policy decision | The product follows the trusted policy and refuses the conflicting request safely. |
| Retrieved document | Indirect-injection handling | Retrieved snippet, trust label, downstream context, final action | Document content is treated as data and cannot redefine the task or policy. |
| Untrusted tool response | Tool-output trust | Tool response, subsequent reasoning trace, next tool call or output | A tool response cannot inject new authority or redirect the workflow. |
| Approval bypass attempt | Human confirmation gate | Requested action, approval state, tool arguments, execution status | The action remains blocked until the required human approval is recorded. |
| Cross-user data request | Data and tenant isolation | Identity, requested record, access decision, returned fields | Only data authorized for the acting identity is returned. |
| Persistent hostile instruction | Memory scope and lifecycle | Memory write, scope, later read, resulting decision | Untrusted instructions do not persist into unrelated sessions or users. |
Method step 01
Define what prompt injection means for your product
Start by defining the expected instruction hierarchy: system instructions, developer policy, user intent, retrieved content, tool output, and memory. Prompt injection testing asks whether lower-trust content can override higher-trust instructions or cause the system to reveal, change, or do something outside the intended task.
Why it matters: A vague definition creates vague tests. For one product, failure may mean leaking system prompts. For another, failure may mean summarizing a malicious document as trusted fact, calling a payment tool, or sending private data to the wrong place.
Method step 02
Test direct and indirect injection
Direct injection comes from the user prompt. Indirect injection comes from content the system retrieves or reads on behalf of the user, such as documents, pages, tickets, email, code comments, calendar entries, or tool responses. The test plan should include both, because many real failures happen when the user never sees the malicious instruction directly.
Why it matters: A model can be cautious with user text and still overtrust retrieved text. Indirect injection is especially important for agents, research assistants, support copilots, coding tools, and any workflow that imports third-party or user-generated content.
Method step 03
Use missions instead of magic phrases
Do not judge security by whether one famous jailbreak phrase works. Build missions that represent realistic product tasks: summarize a hostile document, triage a support case, prepare a code change, search internal data, or decide whether to call a tool. Measure whether the system follows the right authority and preserves boundaries across the full workflow.
Why it matters: Phrase-based testing becomes stale quickly. Mission-based testing stays useful because it exercises the product's real context, tools, data, and decisions. It also produces evidence engineering teams can reproduce after a fix.
Method step 04
Check tool and data boundaries
For tool-using systems, test whether injected instructions can change tool selection, arguments, target resources, approval behavior, or output destination. For data-heavy systems, test whether injected content can make the model reveal hidden context, retrieve unrelated records, include secrets, or combine data across users or tenants.
Why it matters: The most serious prompt injection failures usually involve authority or data movement. If an attacker can only get strange text, impact may be limited. If the same input changes a tool call or leaks private data, it becomes a security finding.
Method step 05
Evaluate mitigations with retests
Common mitigations include strict tool permissions, retrieval filtering, context labeling, output validation, allowlisted actions, confirmation gates, data minimization, model-independent policy checks, and logging. Retest each mitigation against the original mission and nearby variants to make sure the fix addresses the trust boundary, not just one prompt.
Why it matters: Prompt injection cannot be solved with a single sentence in the system prompt. Durable mitigation usually requires product controls around what the model can see, decide, call, and return. Retesting proves whether those controls actually changed the outcome.
Method step 06
What evidence to keep
Keep the test mission, trusted instructions, untrusted input, retrieved snippets, model response, tool calls, approvals, data returned, logs, and final impact. Mark whether the path was confirmed, blocked, partially successful, or theoretical. Avoid storing sensitive secrets in the report unless they are redacted and necessary to prove impact.
Why it matters: Good evidence keeps the conversation practical. It tells product teams what boundary failed, tells engineers where to fix, and gives security teams a baseline for regression tests when prompts, models, tools, or retrieval sources change.
Method step 07
Common failure patterns to watch
Common failures include treating retrieved text as instructions, revealing hidden prompts, summarizing malicious content as trusted fact, calling tools with attacker-selected arguments, copying secrets into output, accepting tool responses as policy, and storing hostile instructions in memory. Track the pattern, not only the exact text that triggered it.
Why it matters: Patterns are easier to fix than phrases. If the root cause is untrusted retrieval, the durable fix may be context labeling and policy checks. If the root cause is broad tool authority, the fix is permission design. The evidence should point to the failed boundary.
Method step 08
Turn prompt tests into regression cases
Every confirmed prompt injection path should become a regression case with a mission, trusted context, untrusted context, expected safe behavior, and required logs. Rerun those cases when prompts, models, retrieval sources, tools, permissions, or memory behavior changes. Keep the cases safe and focused on boundary validation rather than public exploit text.
Why it matters: Prompt injection is not a one-time cleanup. It is a class of behavior that can return after product changes. Regression tests give teams a practical way to preserve the lesson from each finding and avoid rediscovering the same weakness later.
Primary references
Related briefs
- What Is AI Agent Security Testing?
- AI Red Teaming vs Traditional Pentesting
- MCP Security Testing Checklist
Relevant Loki service: AI Agent Risk Assessment — Private Preview.