When a feature includes a model, the classic test stops working as is. An assert comparing against exact text fails even when the answer is right, because the model words it differently every time. The usual reaction is to mark the test as flaky, add retries and move on. That hides the problem: the test stopped measuring.

The way out isn't abandoning automation but changing the question. Instead of "is it equal to this?", we ask "does it meet these criteria, most of the time?". That is an eval. This guide shows how to build one in three layers, from the cheapest and most deterministic to the most expensive, and how to choose a threshold.

How it fits together

Show the diagram as text

model output
   ▼
Layer 1: format     (deterministic: JSON, fields, length)
   ▼
Layer 2: content    (contains / regex on key facts)
   ▼
Layer 3: judgment   (rubric graded by a judge model)
   ▼
run N times ──▶ pass rate >= threshold?
                  ├─ yes ▶ pass
                  └─ no  ▶ fail + inspect the failed runs

Requirements

• Node.js 22 or later.
• One of your tests that compares exact text.
• Optional: Ollama with a local model.

Setup

node -v
npx promptfoo@latest --version

Step by step

1. Assert versus eval

Understand the difference: an assert compares against an exact value; an eval measures whether the output meets criteria, several times, and yields a percentage.

An assert has one input, one expected output and a binary result. An eval has a set of inputs, criteria instead of a single output, several runs per input and a percentage result compared against a threshold. How you read a failure changes too: in an assert, red means bug; in an eval, one bad run may be noise and five in a row are a regression. That is why you need to record the pass rate over time and not just the latest result.

2. Layer one: format

Layer 1, format: validate structure with deterministic checks (valid JSON, required fields, length).

It's the cheapest layer and the one that catches the most. Check that the output is valid JSON, that required fields exist with the right type, that maximum lengths hold and that no text sits outside the structure. All of that is solved with a schema and no model call. If this layer fails, evaluating the next ones is pointless: an answer your application can't parse is already a defect, whatever it says.

3. Layer two: content

Layer 2, content: verify key facts with contains or regex, without demanding the exact sentence.

Here you verify facts, not wording. If the answer must mention an amount, a date and a status, look for those three with contains or a regular expression. Checking the opposite also helps: that information which shouldn't be there doesn't appear, such as another customer's data or a nonexistent step. This layer is still deterministic and cheap. The practical rule is that every hard fact gets validated here, never by another model's opinion.

4. Layer three: judgment

Layer 3, judgment: use a rubric graded by another model for what rules can't express (tone, completeness).

What remains is what rules can't express: whether the tone fits, whether the explanation is complete, whether the answer really addresses the question. For that you use a rubric graded by another model, with observable criteria and a pass or fail result. It's the most expensive and least reliable layer, so keep it small. Day sixteen shows how to calibrate that judge against your own judgment before trusting it.

5. Runs and threshold

Run N times and set a threshold: for example, pass if 9 of 10 runs satisfy all three layers.

Run each case several times: three for daily work, five or ten when you're about to make a decision. The threshold depends on risk. For the format layer, demand one hundred percent, because broken JSON breaks the application. For judgment, ninety percent can be reasonable. What matters is setting the number before looking at results and writing down why, so you don't adjust it afterwards to whatever came out.

6. Its place in the pyramid

Place it in your pyramid: unit and contract stay deterministic; evals are an extra layer for anything that touches a model.

Evals replace nothing you already have. Unit and contract tests stay deterministic and keep running on every commit. Evals are an extra layer covering only what touches a model, and they usually run at a different frequency because they cost more. A common mistake is trying to cover logic that doesn't depend on the model with evals: that is tested better, faster and cheaper with a classic test.

Illustrative example

A team has a test comparing an order summary against fixed text, failing one run in three. They rewrite it like this. Layer one: the output is JSON with summary and status fields, and the summary stays within 280 characters. Layer two: the summary contains the order number and the total amount. Layer three: a rubric checks that it adds no steps the order doesn't have. They run it ten times. Layer one passes ten out of ten, so does layer two, layer three nine out of ten. With a ninety percent threshold it passes, and the failed run is saved for review.

Common mistakes

• Covering an exact assert with retries instead of changing what is measured.
• Using a judge model to validate hard facts such as amounts or dates.
• Setting the threshold after seeing the results.
• Drawing conclusions from a single run.

Checklist: how to know it is done

• The original test is rewritten into three separate layers.
• Format and content are validated without calling a model.
• There is a written threshold with its justification.
• You saved and reviewed at least one failed run.

Where AI fell short

The judge model approved an answer with a wrong fact. Facts get validated with rules, not with another model.

Frequently asked questions

Doesn't setting temperature to zero solve the variation?

It reduces it, it doesn't remove it. Differences remain across model versions, across providers and sometimes across runs. Besides, in production you won't always use temperature zero. Your way of measuring should tolerate variation instead of depending on its absence.

How many cases do I need to start?

Ten well-chosen cases are enough to see the failure pattern. The dataset grows afterwards, with real cases that show up in use. Later in the series we see how to build one with fifty or more using selection criteria.

Do evals have to run on every commit?

Not necessarily. The deterministic layers are cheap and can always run. The judge layer costs tokens and time, so it usually runs only when prompts, model or data change, plus on a schedule once a day. Start by running everything by hand for a week: you'll learn how long it takes and what it costs before deciding where it belongs in the pipeline.

Day 3 of 31 of the 31 days of QA + AI series. Commands and versions reviewed in November 2026: rerun them before relying on them, AI tooling changes fast.

← Previous: How the SDET role is being reshaped by AI

Next: Your AI QA lab: full setup in 30 minutes →