Some qualities of an answer are captured by no rule: whether the explanation is clear, whether the tone fits, whether it answers what was asked. Reviewing them by hand doesn't scale beyond a few dozen cases. The usual solution is asking another model to grade them, known as LLM as a judge.

The problem is that this judge is another non-deterministic system with its own errors, and its verdicts come with very convincing justifications. Using it uncalibrated is like measuring with an instrument you never compared against a standard. This guide shows how to write a rubric that can be verified, how to measure agreement between the judge and you, and when that agreement is enough to trust it with a gate.

How it fits together

Show the diagram as text

30 real outputs
   ├─▶ human labels (pass / fail + reason) ──┐
   └─▶ judge with rubric ────────────────────┴─▶ agreement = matches / total
                                                     ▼
                                              above minimum?
                                   ├─ no  ▶ refine rubric ▶ run the judge again
                                   └─ yes ▶ use in CI ▶ recalibrate on judge change

Requirements

• An eval harness with a judge configured (days 8 and 15).
• 30 real outputs from the model under test.
• A sheet with columns: id, human verdict, judge verdict, reason.

Setup

assert:
  - type: llm-rubric
    value: |
      PASS only if ALL are true:
      1. States the severity explicitly.
      2. Mentions the affected component named in the report.
      3. Adds no steps or facts that are absent from the report.
      Otherwise FAIL and name the criterion that failed.

Step by step

1. Observable criteria

Write observable criteria. "Is the answer good?" doesn't work. "Mentions the component from the report" does.

A rubric works when two people reading it reach the same verdict on the same answer. "Is the answer good?" doesn't qualify. "Mentions the component named in the report" does, because it can be checked by looking at the text. Write few criteria, each verifiable, and define the result as pass or fail with all criteria met. Avoid one-to-five scales: nobody knows what separates a three from a four, and neither does the judge.

2. Thirty labels of your own, first

Hand-label 30 outputs yourself, pass or fail, with the reason. Do it before looking at what the judge says.

Take thirty real outputs from the system and label them by hand, pass or fail, noting the reason in one line. Do it before running the judge and without looking at what it would say. If you see its verdict first, your label gets contaminated and the agreement you measure afterwards will come out inflated. Thirty cases are a reasonable floor to start. Aim for passes and fails in similar proportions: with almost everything passing you won't know whether the judge detects the bad ones.

3. Run the judge on the same cases

Run the judge on the same 30.

Run the judge with the rubric on those same thirty outputs, with the stable configuration you defined yesterday. Save the verdict and justification for each case. If you can, run two or three times to see whether the judge is consistent with itself. A judge that changes its verdict on the same input between runs has a problem that comes before agreement with you: the rubric leaves too much room or the config isn't stable.

4. Agreement and direction

Compute agreement: matching cases divided by total. And check the direction of disagreement: is the judge more lenient or stricter than you?

The calculation is simple: matching cases divided by the total. But the number alone isn't enough. Look at the direction of disagreement. If the judge passes cases you failed, it's lenient and your gate will let defects through. If it fails cases you passed, it's strict and will block valid work. Those are different risks and they're fixed differently. Build the two-by-two table, you against the judge, and read the two disagreement cells separately.

5. Fix the rubric

Fix the rubric where you disagree: a sharper criterion or one pass and one fail example.

Read each disagreement and ask who is right. Sometimes it's the judge, and you correct your label. Most often the criterion was ambiguous. Two fixes work: rewriting the criterion to make it sharper, or adding to the rubric one example that passes and one that fails. Don't add ten examples, because the judge starts copying the examples instead of applying the criterion. And don't use as an example any of the thirty cases you'll measure afterwards.

6. Measure again and set a minimum

Measure again. Set a minimum agreement to use the judge in CI and recalibrate whenever you change the judge model.

Run again and recompute. Decide in advance which minimum agreement you need to use the judge in CI and write it down. If you don't reach it, the judge is useful for ordering what to review by hand, but not for blocking a merge. Calibration expires: every time you change the judge model, its version or the rubric, measure again against your labels. Keep those thirty labels as an asset, because they are your reference standard.

Illustrative example

A team evaluates summaries of support tickets. They label thirty by hand: eighteen pass and twelve fail. The judge matches on twenty-three, an agreement of seventy-seven percent. Of the seven disagreements, six are cases the judge passed and they didn't: it's lenient. Reading them, they find the pattern. The criterion said "includes the customer's problem" and the judge accepted vague mentions. They rewrite it as "names the product and the concrete symptom" and add one example of each verdict. In the second measurement, twenty-eight out of thirty match. They set that level as the minimum for using it in the pipeline.

Common mistakes

• Labeling after seeing the judge's verdict.
• Using numeric scales instead of pass or fail.
• Looking only at the agreement percentage and not its direction.
• Changing the judge model without recalibrating.

Checklist: how to know it is done

• The rubric has criteria that can be checked by looking at the text.
• There are thirty cases labeled by hand before running the judge.
• You computed agreement and know whether the judge is lenient or strict.
• There is a written minimum agreement for using it in CI.

Where AI fell short

The judge explains wrong verdicts very well. Don't trust the justification: measure agreement.

Frequently asked questions

What agreement is enough?

It depends on which decision the judge makes. As a practical reference, first measure how much two people on your team agree on the same cases: it makes no sense to demand more agreement from the judge than humans reach with the same rubric.

What if I don't have time to label thirty cases?

Then you have no way of knowing whether the judge works. Thirty pass-or-fail cases take under an hour. It's the smallest investment that turns an automatic opinion into a measurement.

Can I use several judges and average them?

You can, and it sometimes reduces errors, but it multiplies the cost and doesn't replace calibration. Three uncalibrated judges agreeing with each other can be wrong in the same way.

Day 16 of 31 of the 31 days of QA + AI series. Commands and versions reviewed in November 2026: rerun them before relying on them, AI tooling changes fast.

← Previous: How to set up models for quality testing

Next: Golden datasets: how to label, balance and version →