Red Teaming Enterprise LLM Applications: A Playbook

Most enterprise security teams we work with have spent a decade hardening their web stack, their APIs, and their cloud accounts. Then someone in product shipped an LLM feature, the CISO asked "have we tested this?", and the answer was a vendor questionnaire and a copy of the OWASP LLM Top 10 taped to a wall. That is not a red team. This piece is what we actually do when an enterprise asks us to red team a production LLM or agent system, and what we tell them to do after we leave.

Why LLM red teaming is not pen testing

Traditional penetration testing assumes a deterministic system. You probe inputs, you compare outputs to a spec, you find the cases where the spec breaks. LLM applications do not have that spec. The model is a probability distribution wrapped in a system prompt, surrounded by retrieval, surrounded by tools, surrounded by an application. Every layer can be subverted, and the subversions compound.

That changes the work in three concrete ways:

• The attack surface is the data, not just the code. A poisoned document in a vector store, a malicious payload in a Confluence page the agent reads, a tampered tool description — these all become exploitable surfaces a SAST tool will never see.
• One-shot tests prove nothing. A model that refuses a jailbreak nine times out of ten is still a one-in-ten liability at scale. We test with statistical rigour, not single-prompt screenshots.
• The blast radius grows with autonomy. A chatbot that hallucinates a refund policy is embarrassing. An agent with database write access and the same hallucination is a Sev-1 incident waiting to happen.

The attack taxonomy that actually matters in production

We have run red teams against retrieval systems, copilots, customer-facing assistants, and multi-agent platforms. The same handful of attack classes drive almost every real finding.

Direct and indirect prompt injection

Direct prompt injection — a user typing "ignore previous instructions" into a chat box — is the version everyone tests for. It is also the least dangerous, because the attacker is the user and the blast radius is their own session. The harder version is indirect prompt injection: an attacker plants instructions in a document, an email, a webpage, or a calendar invite that the LLM later ingests on behalf of a different user. Now the attacker is anyone who can write content the agent reads, and the victim is the user whose session the agent is acting in.

We have found indirect injection paths in resumé screening tools, customer support copilots that read inbound emails, sales agents that summarised competitor pricing pages, and procurement bots that parsed supplier PDFs. In every case the team had tested direct injection and called it done.

Tool-use exploitation

Once the model can call tools, the question is no longer "what will it say?" but "what will it do?" The attacks we see most often:

• Argument smuggling. The user asks for something benign, but a prompt-injected document convinces the model to call a tool with attacker-controlled arguments — a SQL query, a webhook URL, a file path traversal.
• Tool chaining for exfiltration. The agent has both "read internal docs" and "send email" tools, and an injected instruction stitches them together to leak data to an external address.
• Cost and rate abuse. An attacker triggers expensive tool loops — recursive retrievals, search-API spam, image generation in a tight loop — to burn budget or starve other tenants.

Output handling failures

The model returns a string. The application renders it, executes it, or passes it to another system. Every time we have looked, there is at least one path where model output reaches a sink — a markdown renderer, a SQL parameter, an HTML attribute, a shell command — without the validation that would exist for any other untrusted input. XSS through model-generated links, SSRF through model-fetched URLs, and SQL injection through model-emitted query fragments are all live findings from the last twelve months.

Privilege and identity confusion in agentic systems

Multi-agent systems make this worse. When agent A delegates to agent B, whose permissions apply? Whose audit trail? Whose rate limits? We routinely find systems where a low-privilege user can route a request through an internal planning agent and end up exercising a high-privilege execution agent's credentials. The fix is not a clever prompt — it is identity propagation and per-call authorisation at the tool boundary, the same controls we would demand from any service mesh.

Our six-phase red team process

Every engagement runs through the same backbone. We tune the depth, not the structure.

• Phase 1 — Threat model. Two days with the engineering and product teams to map data flows, trust boundaries, tools, agents, identities, and the worst-credible-outcome for each. No testing yet. If the threat model is sloppy, the rest is theatre.
• Phase 2 — Static review. System prompts, tool schemas, retrieval configuration, output handlers, observability. We are looking for design-level weaknesses before we touch a payload.
• Phase 3 — Automated adversarial suites. We run curated and generated attack corpora — jailbreak families, injection variants, harmful-content probes, data exfiltration patterns — at scale, with statistical reporting on success rates rather than anecdote.
• Phase 4 — Manual deep dives. Senior engineers chase the interesting failures, chain them, and turn them into end-to-end exploit narratives a board can understand.
• Phase 5 — Production telemetry analysis. We look at real traffic — sanitised — for behaviours the team has been missing. Real users find attack surfaces no red team budget can cover.
• Phase 6 — Remediation pairing. We sit with the engineering team and fix the top findings together, then re-test. A red team report that does not produce shipped fixes is a procurement artefact, not security work.

What separates production-grade red teaming from theatre

Three signals tell us whether a programme is real or performative.

• Findings are reproducible and rated. Every issue ships with a payload, a repro rate, a CVSS-equivalent severity, and a fix recommendation tied to the codebase. "The model said something bad once" is not a finding.
• Coverage is measured. We track which attack classes were tested against which surfaces, with what intensity. Gaps are visible and prioritised, not hidden behind a green tick.
• Tests run continuously. The high-value suites live in CI and run on every prompt, tool, or model change. A point-in-time red team that is not converted into regression tests has a shelf life of one deployment.

Where to invest first

If you have not started, the order matters. We tell clients to spend in this sequence:

• Identity and tool authorisation. Before you harden prompts, make sure tools enforce the calling user's permissions and that every tool call is logged with the originating identity. This single control kills more real exploits than any prompt-level defence.
• Output sinks. Treat every model output as untrusted input to whatever consumes it. Sanitise, parameterise, escape. This is boring, ancient appsec work and it pays.
• Indirect injection defences on retrieval. Mark untrusted content, strip instructions in untrusted regions, use structured retrieval, and never let untrusted text drive tool selection.
• Continuous adversarial testing. Stand up an internal red team capability or contract one. One-off tests cannot keep pace with model swaps and prompt changes.
• Observability with an attacker's lens. Log prompts, retrievals, tool calls, and outputs in a form that lets you go back and ask "did anyone try this?" — because someone will.

The companies that get LLM security right will not be the ones with the most policies. They will be the ones whose engineers can show you the regression test for the last attack that worked.

If you are standing up an AI red team capability, or you need a senior team to stress-test a system before it goes wide, this is the work we do. We bring engineers who have shipped production LLM systems and broken them on purpose, not consultants reading from a framework deck.