An eval that runs only when someone remembers ends up not running. The way to sustain the quality of an AI feature over time is the same as for the rest of software: a gate in the pipeline that blocks the change when something gets worse. What changes are the constraints. Evals cost money per run, take longer than a unit test and their result varies.

Putting every eval on every pull request makes the pipeline slow and expensive, and the team ends up skipping it. Putting in none lets regressions through. This guide organizes evals into three tiers, per PR, nightly and per release, each with a different scope, cost and consequence, and shows how to keep spending under control.

How it fits together

Show the diagram as text

pull request touching prompts / evals / AI code
   ▼
PR tier: dev subset x3 ▶ exit code 0?
                           ├─ no  ▶ merge blocked + artifact
                           └─ yes ▶ merge
                                      ▼
            nightly: full test set x5 + judge + red team ▶ trend + alert
                                      ▼
            release: regression + performance ▶ human sign-off ▶ production

Requirements

• A repo with evals (day 8) and a golden dataset (day 17).
• CI: GitHub Actions or Jenkins.
• An API key stored as a CI secret, with a spending cap.

Setup

name: evals
on:
  pull_request:
    paths: ["prompts/**", "evals/**", "src/ai/**"]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm ci
      - name: Run evals
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PROMPTFOO_CACHE_PATH: .promptfoo-cache
        run: npx promptfoo eval -c evals/bug-triage.yaml --repeat 3 -o results.json
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: eval-results, path: results.json }

stage('Evals') {
  when { anyOf { changeset 'prompts/**'; changeset 'evals/**'; changeset 'src/ai/**' } }
  steps {
    withCredentials([string(credentialsId: 'llm-api-key', variable: 'OPENAI_API_KEY')]) {
      sh 'npm ci && npx promptfoo eval -c evals/bug-triage.yaml --repeat 3 -o results.json'
    }
  }
  post { always { archiveArtifacts artifacts: 'results.json' } }
}

The three tiers

• PR: runs when prompts, evals or AI code change. Reduced dev set, 3 runs, format and content checks. Blocks the merge.
• Nightly: every night. Full test set x5, judge and red teaming. Alerts the team and shows the trend.
• Release: before shipping. Full regression plus performance. A person signs off.

Step by step

1. The PR tier

Start with the PR tier: a small, fast subset that runs only when prompts, evals or AI code change.

It's a small, fast subset designed to answer within a few minutes. It runs only when the change touches prompts, eval config or code that calls the model. For a frontend style change, spending tokens makes no sense. Include the deterministic format and content checks, and a reduced set of cases with few repetitions. Its job isn't measuring quality precisely, but catching obvious breakage before the merge.

2. The exit code as the gate

The eval command returns a non-zero exit code when cases fail. That's enough to block the merge.

The mechanism is the same as for any test. The eval command ends with a non-zero code when cases fail, and the pipeline marks the step red. That's enough to block the merge through branch protection rules. Nothing special needs building. You do need to verify it works: force a failure on purpose on a test branch and confirm the pull request ends up blocked.

3. Artifacts always, on failure too

Always save the result as an artifact, including on failure. Without the detail nobody can diagnose.

Configure the step that saves results to run even when the eval fails. It's a config detail often forgotten, and then the pipeline goes red with no information about which case failed or what the model answered. The artifact must include inputs, outputs and verdicts per case. With that file, whoever opened the PR can diagnose in minutes without rerunning everything on their machine.

4. Controlling cost

Control cost: path filters, response caching, a subset on PRs and a CI API key with a spending cap.

There are four levers. Path filters, so evals don't run on changes that don't affect them. Response caching, so you don't pay twice for the same input with the same prompt. A subset at the PR tier instead of the full set. And a CI-only API key with a spending cap, so a config error or a loop doesn't consume the month's budget. Publish each run's spend next to the result.

5. The nightly tier

Add the nightly tier with the full set, five runs, the judge and red teaming. That's where you watch the trend, not a single PR.

Once a day, with the full test set, five repetitions, the calibrated judge and red teaming. This is where you measure precisely. The result doesn't block a particular PR: it alerts the team and feeds a trend chart. It's where you'll detect problems no PR introduced, such as a silent provider update or gradual degradation. Since it runs at night, time and cost are more tolerable.

6. The release tier

At release run regression and performance, and leave the final call to a person with the report in front of them.

Before shipping, full regression and performance tests run. The difference from the other tiers is who decides. Here the result is a report a person reads and signs. Automatic gates cover what can be expressed as a threshold. The decision to ship with a metric at the limit or with a known risk is a business judgment, and someone has to take responsibility for it with the information in front of them.

Illustrative example

A team sets up the three tiers. The PR one runs fifteen cases with three repetitions, only on changes to the prompts and evals folders, and takes four minutes. In the first week it blocks a pull request where a wording change broke the JSON format in two cases. The author opens the artifact, sees the two outputs and fixes it in ten minutes. The nightly tier runs one hundred and twenty cases with five repetitions. One Tuesday the pass rate drops four points with no PR merged. The run card shows the model that answered wasn't the expected one, because of an unpinned alias nobody had noticed.

Common mistakes

• Running the full set on every pull request.
• Not saving artifacts when the eval fails.
• Using an API key with no spending cap in CI.
• Treating a single green as proof that quality holds.

Checklist: how to know it is done

• The PR tier runs only on relevant changes and within minutes.
• A forced failure blocks the merge.
• Results are saved as an artifact on failure too.
• A nightly tier with a trend and a capped key exists.

Where AI fell short

The same code can pass a gate and fail hours later. With models, a single green doesn't replace the trend.

Frequently asked questions

What do I do with an eval that sometimes passes and sometimes fails on PRs?

First check whether the case is unstable by design, in which case it goes to the nightly tier with repetitions, or whether the PR tier's threshold is too demanding for so few runs. A gate that fails at random loses the team's trust very fast.

Does it work with any CI system?

Yes. All you need is to run a command, read its exit code, filter by paths and save files. The guide carries examples for two systems, and the same structure carries over to the rest.

Who watches the nightly tier?

It must have a named owner. An alert landing in a channel where nobody answers is the same as having no alert. Define who reviews it every morning and what they do when the trend drops.

Day 28 of 31 of the 31 days of QA + AI series. Commands and versions reviewed in November 2026: rerun them before relying on them, AI tooling changes fast.

← Previous: Performance (3): AI generating and analyzing load tests

Next: QA infra for AI: ephemeral runners, local models, secrets and cost →