Red Teaming LLMs: Methodology and Tooling

LLM red teaming is structured adversarial testing. The goal is not to collect scary prompts; it is to discover where the product violates its safety, security, privacy, or reliability contract under realistic attacker behavior.

The running example is a customer-support agent with email, CRM, and refund tools. The red team asks what a malicious user, malicious email, compromised document, or confused employee could cause the agent to do.

Automated probes are good for breadth: jailbreak strings, injection templates, policy edge cases, and malformed tool arguments. Human scenario testing is better for chained attacks where one harmless step sets up the next. Both need severity scoring and reproducible evidence.

For the support agent, one scenario might place an instruction in an email, ask the agent to summarize it, then attempt a refund. The important result is whether the harness blocks the risky action, not whether the model says something odd.

Trigger: the model refuses a direct jailbreak in chat, so the team marks the feature safe. Symptom: a malicious email later causes the agent to send private account notes to an external address. Root mechanism: testing focused on model refusal, while tool authorization accepted model intent too easily.

Recovery is to disable the risky tool path, inspect audit logs, and add approval gates. Prevention is to test model, retrieval, tools, memory, and UI confirmation as one system.

Write ten cases across four categories: direct jailbreak, indirect prompt injection, data exfiltration, and unsafe tool action. For each case, define expected behavior and severity if it fails.

Pass when the suite can run before each release and reports the exact failed boundary. Break it by allowing one external-send tool call without confirmation; recover by adding a policy gate and a regression test.