The word "harness" shows up in almost every conversation about AI and testing, and almost always with a different meaning. For some it's the usual automation framework. For others, the tool that evaluates a prompt. For others, the program that turns a model into an agent able to edit code. All three exist and all three share the name, and that confusion makes teams buy or build what they don't need.

This guide separates the three meanings with a single common definition and shows how they relate. There is no code today: the goal is for you to look at your own stack and say which harness you have, which one you lack and, above all, which one verifies which.

How it fits together

Show the diagram as text

Agent harness (model + tools + rules + permissions)
        │ writes tests and code
        ▼
      repo
   ├─▶ Test harness (runner + fixtures + reports) ── pass / fail ──┐
   └─▶ Eval harness (dataset + scorers + report)  ── pass rate ────┤
                                                                   ▼
                                                             quality gate
                                                                   │ feedback
                                                                   └──▶ agent

Requirements

• Know your current automation stack.
• The list below, to fill in with your tools.

The three harnesses

• Test harness: wraps your app. Runner, fixtures, mocks and reporters. Examples: Playwright Test, JUnit, pytest.
• Eval harness: wraps a model or prompt. Dataset, runner, scorers and report. Examples: promptfoo, DeepEval, Inspect AI.
• Agent harness: wraps a model that acts. Loop, tools, context, permissions and memory. Examples: Claude Code, Copilot agent, Cursor.

Step by step

1. A single definition

Keep the definition: a harness is everything around the thing under test that lets you run, control and measure it.

A harness is everything around the thing under test that lets you run, control and measure it. Running it means starting it in a repeatable way. Controlling it, fixing its inputs and its environment. Measuring it, getting a comparable result. The definition works for all three types; the only thing that changes is what sits in the middle: your application, a model or an agent. When someone uses the word, ask what's in the middle and the conversation sorts itself out.

2. Test harness: the one you already have

Test harness: you already have it. Runner, fixtures, mocks and reports around your app.

If you run automated tests, you already have one. The runner executes, fixtures prepare state, mocks isolate dependencies and reports show the result. It's deterministic: the same input gives the same output and a failure means something. It's worth recognizing as a harness because it's the base for the other two. A team with a slow, brittle test harness that nobody looks at will carry those same problems into everything it builds with AI on top.

3. Eval harness: the same around a model

Eval harness: the same idea around a model or prompt. Asserts become scorers and cases become a dataset.

Here the thing under test is a model or a prompt. The pieces have direct equivalents: test cases become a dataset, asserts become scorers that grade an output, and the report shows pass rates instead of green or red. Two things are added that the test harness didn't need: repeating each case several times and recording cost and latency. You'll need it the day a feature of your product depends on a model.

4. Agent harness: the one that works

Agent harness: the loop that turns a model into an agent, with tools, instructions, permissions and memory. Here the agent does QA work.

It's the loop that turns a model into an agent: it receives a task, decides which tool to use, reads the result and decides again, until done. Around it are repository instructions, available tools, permissions and memory. The key difference from the other two is the role: this one doesn't measure, it produces. It writes tests, modifies code, investigates a failure. That's why configuring it feels more like onboarding a new teammate than setting up a tool.

5. Who verifies whom

Key relationship: the agent harness produces tests and changes; the test harness and eval harness verify what it produced. Never the other way round.

The relationship runs in one direction only. The agent harness produces tests and changes; the test harness and the eval harness verify what was produced. If you reverse the arrow and let the agent judge its own work, you lose independent verification, which is the whole point of QA. In practice this becomes a concrete rule: what counts as done is the result of a command running outside the agent, with its exit code, and not the summary the agent writes.

6. Your stack on the list

Fill the list with your stack and flag the harness you lack. That's your next step.

Fill in the list of three harnesses with real names. Which runner, fixtures and reports you use. If you evaluate a model, with which dataset and scorers. If you use an agent, which rules, tools and permissions it has. Empty cells are information. If you lack the eval harness and your product already uses a model, that's the next step, and it's tomorrow's topic. If you lack the agent one, it can wait: it's productivity, not verification.

Illustrative example

A product team fills in the list. Test harness: Playwright, login fixtures, HTML report in CI; complete. Eval harness: empty, even though the product has had an assistant answering customer queries for three months. Agent harness: two people use a coding agent with no shared rules. The diagnosis is clear. The riskiest feature, the assistant, has no measurement at all, while the team was debating which agent to adopt. They decide to build a twenty-case eval for the assistant first and then write a common rules file for the agents.

Common mistakes

• Using "harness" without saying what sits in the middle.
• Letting the agent declare its own work done.
• Investing in an agent harness while the test harness is brittle.
• Evaluating a model with exact asserts instead of an eval harness.

Checklist: how to know it is done

• You can explain the three harnesses with the same definition.
• Your list has concrete names in each cell or an explicit gap.
• You identified which one you lack and why it matters.
• You have the rule written down that the gate runs outside the agent.

Where AI fell short

Without an external gate, the agent reports success without having run anything. Verify with the exit code, not the summary.

Frequently asked questions

Is an automation framework a harness?

It's a part of one. The framework provides the runner and utilities; the full harness also includes your project's fixtures, data, mocks, environment and reports. That's why two teams with the same framework can have harnesses of very different quality.

Do I need all three?

The test harness, always. The eval harness, only if something in your product or process depends on a model's output. The agent harness is optional: it improves productivity but adds no verification by itself.

Can I use the same model in the agent and in the eval?

You can, but verification shouldn't depend on it alone. The deterministic layers of the eval and the classic test harness are what provide independence. A model grading its own output tends to approve it.

Day 7 of 31 of the 31 days of QA + AI series. Commands and versions reviewed in November 2026: rerun them before relying on them, AI tooling changes fast.

← Previous: Synthetic test data with LLMs: realistic, valid and PII-free

Next: Your first eval harness with promptfoo →