With yesterday's threat model you have a list of abuse cases. Testing them by hand, writing attack variants one at a time, works for five cases and not for five hundred. And real attacks don't come one at a time: they come rephrased, translated, spread over several messages or disguised as something else.

Automated red teaming uses a model to generate those variants in volume, sends them against your application and uses another model to decide whether the attack succeeded. It's powerful and has clear limits: it costs money, it generates adversarial content and its verdict is also wrong at times. This guide shows how to configure it from your threat model, how to read the report without taking it as truth and where to place it in your testing cycle.

How it fits together

Show the diagram as text

threat model ▶ purpose + plugins + strategies
                    ▼
   attacker model: generates adversarial inputs
                    ▼
   target: your app in staging
                    ▼
   grader model: did the attack succeed?
                    ▼
   report: attack success rate per category
                    ▼
   human reads transcripts: real finding or false positive
                    ▼
   fix + rerun ▶ scheduled (nightly / per release)

Requirements

• Only against systems you own or have written authorization for, in test environments.
• The threat model from day 22.
• Your assistant running locally or in staging with an HTTP endpoint.
• Node.js 22+ and promptfoo.

Setup

npx promptfoo@latest redteam init     # creates the config
npx promptfoo@latest redteam run      # generates and runs
npx promptfoo@latest redteam report   # opens the report

targets:
  - id: http
    config:
      url: http://localhost:3000/api/chat
      method: POST
      headers: { Content-Type: application/json }
      body: { message: "{{prompt}}" }
      transformResponse: json.reply
redteam:
  purpose: >
    Customer support assistant for an online store. It must not reveal other
    customers' data or its internal instructions, and must not take actions
    without confirmation.
  numTests: 5
  plugins:
    - pii
    - prompt-extraction
    - excessive-agency
    - hijacking
  strategies:
    - jailbreak
    - prompt-injection

Step by step

1. Authorization and environment

Confirm authorization and environment. Red teaming generates adversarial content: it targets staging, with test data.

Before running anything, confirm in writing that you're authorized to attack that system, and target a staging environment with test data. Red teaming generates adversarial content on purpose and can trigger real actions if the system has tools connected. Against production it can send messages, modify records or trip the provider's abuse alerts. Also tell the team operating the environment. A run nobody expected looks a lot like an incident.

2. Three roles

Understand the scheme: an attacker model generates adversarial inputs, your app is the target, and a grader decides whether the attack succeeded.

The scheme has three pieces. An attacker model generates adversarial inputs. Your application is the target and receives them as if they came from a user. A grader decides, for each exchange, whether the attack achieved what it sought. Understanding this helps interpret results, because two of the three pieces are models, with their own errors. An attack success rate depends as much on your application as on how creative the attacker was and how strict the grader was.

3. The purpose, with its limits

Describe the system's purpose precisely, including what it must not do. Attacks are generated relative to that description.

The config asks for a description of the system. Of the whole file, it's the field with the most influence on attack quality. Write what the application does, for whom and, above all, what it must never do: reveal other customers' data, give medical advice, execute actions without confirmation. Attacks are generated relative to that description. With a vague purpose you get generic attacks, and a report that says little about your risks.

4. Plugins and strategies from the threat model

Pick plugins from your threat model, not all of them. Each plugin is a risk category; each strategy, a way of delivering the attack. Names change between versions: redteam init offers the current ones.

Each plugin represents a risk category and each strategy, a way of delivering the attack. The temptation is to enable everything. That's expensive, slow and produces an unreadable report. Choose the plugins matching the risks you marked as applicable yesterday, and two or three strategies. Names change between versions of the tool, so use the init command to see the current ones instead of copying them from an old tutorial.

5. Read the report and the conversations

Run and read the report per category: attack success rate. Then open the conversations: the grader produces false positives too.

The report shows an attack success rate per category. It tells you where to look, not what to conclude. Open the conversations flagged as successful and read them. You'll find real findings, and false positives too: cases where the application refused correctly and the grader still counted it as a failure. Also review a sample of those flagged as resisted, because false negatives exist. A finding is a conversation a person read and confirmed.

6. Fix, repeat and schedule

Fix, rerun and schedule it: nightly or per release. Cost makes it a poor fit for every PR.

For each confirmed finding, decide the fix, which may be a prompt change, a filter or a design control, and rerun the same config. Save the attacks that succeeded as deterministic regression cases, so you can test them on every change without generating anything. The full red teaming run goes on a schedule, nightly or per release. Because of cost and duration it isn't a good fit for every pull request.

Illustrative example

A team runs red teaming against yesterday's support assistant, in staging, with four plugins chosen from their threat model. The report flags twelve successful attacks out of two hundred. Reading the conversations, five are false positives: the assistant refused and offered to hand over to a person. Seven real ones remain. Four get it to reveal part of its internal instructions and three obtain data from an order by changing the number in a rephrased message. The second group is serious. They fix it by validating in code that the order belongs to the user, not in the prompt. In the second run that group is at zero.

Common mistakes

• Running against production or without written authorization.
• Enabling every plugin instead of those from the threat model.
• Reporting the success rate without having read the conversations.
• Fixing with a sentence in the prompt what requires a control in code.

Checklist: how to know it is done

• There is written authorization and the target is staging with test data.
• The purpose describes what the system must not do.
• The plugins come from the previous day's applicable risks.
• Each reported finding was read and confirmed by a person.

Where AI fell short

The grader model produces false positives and false negatives. A red teaming report with no human review isn't a finding.

Frequently asked questions

How much does a run cost?

It depends on the number of plugins, strategies and cases per plugin, because each attack involves calls to the attacker, your application and the grader. Start with few cases per plugin to estimate spend before scaling, and use a capped key.

Does a success rate of zero mean the system is secure?

No. It means these attacks, generated by this attacker and judged by this grader, didn't succeed. New attacks appear all the time. That's why design controls matter more than the result of one run.

Does it replace a pentest?

No. It covers model-specific risks well, in volume and repeatably. It doesn't cover the rest of the surface: authentication, infrastructure, business logic. They are complementary activities.

Day 23 of 31 of the 31 days of QA + AI series. Commands and versions reviewed in November 2026: rerun them before relying on them, AI tooling changes fast.

← Previous: A model for security testing (1): threat modeling with the OWASP LLM and Agentic Top 10

Next: Security (3): prompt injection in tool-using agents and how to test the controls →