Prompt Injection

Maliciously crafted input that manipulates a model to ignore instructions or execute unintended behaviour.

Also known as: input injection, prompt-based attack

In plain English

Prompt injection is when someone tricks an AI by feeding it sneaky text that makes it do things it shouldn't. It's like slipping a misleading note into instructions so the assistant follows the wrong steps.

In practice

It appears when untrusted documents, user uploads or web content are included in prompts and contain instructions that override system goals. In practice this can cause data leakage, incorrect outputs or bypassing of compliance controls, so workflows must validate and isolate external inputs.

Under the hood

Attackers embed adversarial payloads inside user-supplied content that gets concatenated into the model context, exploiting the model's instruction-following behaviour and token-conditioned generation. Key concepts include token ordering and the instruction hierarchy within the context window, along with retrieval-augmented inputs and tool calls that can be hijacked. Mitigations include strict input sanitisation, schema validation, isolation of untrusted text, and use of model-level guards or function-calling interfaces.

Example

Let's run red-team tests for prompt injection on the retrieval-augmented generation pipeline before going live.

Related terms

Jailbreak · Prompt Engineering · Retrieval-Augmented Generation · Hallucination

•