Test the actual instruction paths
Direct prompt injection
User-supplied instructions try to override the approved task or policy.
Indirect prompt injection
Documents, pages or tool responses carry instructions into context.
Retrieval poisoning
Manipulated retrieved material steers decisions or presents misleading authority.
System prompt extraction
Requests try to disclose hidden instructions or configuration; assess the sensitivity of what is actually exposed.
Tool manipulation
Content attempts to influence tool choice, arguments or data destinations.
Instruction hierarchy conflicts
Conflicting instructions test whether the intended authority boundary holds.
Data boundary attacks
Cases examine whether information can reach an unauthorized audience or context.
Jailbreak attempts
Adversarial instructions seek behaviour the system should refuse or escalate.
Define success before observing the output
State the boundary
Name the prohibited action or disclosure and legitimate task.
Place the hostile input
Use the actual channel and a controlled test fixture.
Observe downstream behaviour
Inspect relevant tool calls and effects as well as the final answer.
Judge the criterion
Record results, missing evidence and uncertainty.
Retest the change
Make the meaningful case part of regression coverage for the affected version.
Focused injection testing and broader red teaming
| Question | Prompt injection testing | Broader red teaming |
|---|---|---|
| Primary focus | Instruction influence and related data or tool boundaries. | End-to-end misuse, delegation, permissions and operational controls. |
| Typical evidence | Input placement, context, response and attempted action. | Connected observations across multiple steps and systems. |
| Limit | Cannot establish security of the entire application. | Still bounded by scope, access and selected scenarios. |
Avoid overfitting to a memorable attack string
A regression case preserves the boundary being tested, not just one phrase. Include variations appropriate to actual channels and populations while keeping expected behaviour explicit. Record which case version and agent configuration produced each result.
Defences outside the model matter. Authorization, data access checks and human approvals can limit consequences even when generated text follows hostile instructions. Test those controls in the relevant workflow and escalate broader questions into agent red teaming.
Choose criteria for the boundary being tested
Start with the differences between prompt injection and jailbreaks to separate delivery paths, behavioral restrictions and observed consequences. The controlled jailbreak testing methodology then explains how to define cases, scoring criteria and retests without treating every refusal as success.
If a case concerns private instructions, use the prompt extraction verification and defense guide to check what was actually disclosed and which independent controls still protect the system.
Questions about prompt injection and jailbreak testing
Can an instruction-only defence eliminate prompt injection?
No universal prevention claim is made. Assess relevant input channels and enforce important permissions outside the model as part of the system boundary.