The evals, agents and security tests from the last few days run on something: runners, containers, models, keys and data. That infrastructure is rarely designed. It gets assembled from whatever is at hand, and one day the pipeline costs three times as much, a result can't be reproduced or a key shows up in a log.

Infrastructure decisions for QA with AI are few and have long consequences. Where the model runs defines cost and privacy. What runners are like defines reproducibility. Where an agent works defines how much damage a mistake can cause. This guide walks through those decisions with a practical rule for each, and with two metrics almost nobody publishes that are worth having from day one: tokens and spend per run.

How it fits together

Show the diagram as text

ephemeral runner
   ├─ agent sandbox: no prod credentials, limited egress
   ├─ local model service: smoke + format evals
   └─ test and eval runners ◀── secrets manager (per-environment key + spend cap)
            │               ◀── synthetic / anonymized datasets
            ├─▶ cloud model: judged evals
            └─▶ artifacts: results + run card + tokens and cost
                     ▼
            trend: pass rate and cost per run

Requirements

• The pipeline from day 28.
• Docker on the runner.

Setup

jobs:
  smoke-evals:
    runs-on: ubuntu-latest
    services:
      ollama:
        image: ollama/ollama
        ports: ["11434:11434"]
    steps:
      - uses: actions/checkout@v4
      - run: |
          curl -s http://localhost:11434/api/pull -d '{"model":"llama3.2"}'
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm ci && npx promptfoo eval -c evals/smoke.yaml

The five decisions

• Where the model runs: cloud API (simple, pay per token, data leaves) or a local model on the runner (free, private, slower and less capable). My rule: local for smoke and format, cloud for judged evals.
• Runners: persistent or ephemeral (a clean container per run). My rule: always ephemeral.
• Where an agent works: on the runner with credentials or in a sandbox with no production credentials and limited network egress. My rule: sandbox.
• Secrets: one shared key or one per environment, with least privilege and a spending cap. My rule: per environment.
• Data: a production copy or synthetic and anonymized. My rule: never production.

Step by step

1. Where the model runs

Decide where the model runs in CI. A local model as a service covers smoke and format checks at no cost; judged evals need a more capable model.

There are two options, and the usual thing is to combine them. A local model, started as a service inside the runner, has no per-token cost and takes no data out of your infrastructure, but it's slower and less capable. A cloud API offers more capable models, charges per use and receives your data. The rule that works best is assigning each job to the cheapest model that solves it: smoke and format checks with the local model, judged evals with a more capable one. Measure before assuming the small one is enough.

2. Ephemeral runners

Use ephemeral runners: every run starts clean. Leftover state from a previous run is a frequent cause of results you can't reproduce.

An ephemeral runner starts clean on every run: a new container, with no files, caches or processes from the previous run. Persistent runners accumulate state, such as a dependency installed by hand, a temporary file or a model someone updated. That state makes the same commit give different results depending on which machine it landed on. With models, which already bring their own variation, adding the environment's makes it impossible to know what changed. Always ephemeral, with versions pinned in the image.

3. Sandbox for agents

If an agent executes commands in CI, have it do so in a sandbox: a container with no production credentials and restricted network egress.

If an agent executes commands inside the pipeline, for example to repair a test or investigate a failure, it must do so in an isolated environment: a container with no production credentials, with the file system limited to the repository and with network egress restricted to the necessary destinations. The agent reads content that may carry malicious instructions, as we saw on day twenty-four. The sandbox guarantees that, if one of those instructions takes effect, nothing valuable is within its reach.

4. Secrets

Secrets: one API key per environment, with least privilege and a spending cap, and verify it doesn't show up in logs or artifacts.

Use one API key per environment: one for development, one for CI and one for production. Each with least privilege and a spending cap. That way a leaked key or a runaway loop in CI affects neither production nor the overall budget. Store them in the CI system's secrets manager, never in the repository. And actively verify they don't show up in logs or artifacts, because some tools print the full configuration when they fail.

5. Cost as a metric

Cost as a metric: report tokens and spend per run next to the pass rate. What isn't measured grows.

Publish tokens consumed and spend on every run, next to the pass rate. What isn't measured grows. A dataset that doubles, a judge that gets added or one more repetition are all reasonable changes nobody adds up. With the number in view, each change shows its cost and the team can decide whether it's worth it. It also lets you detect anomalies: a run costing ten times the usual tends to indicate an error, not more work.

6. Reproducibility

Reproducibility: pinned tool versions (lockfile) and model versions, and results archived with their run card.

Two things make a run reproducible. The first is pinning versions: tools through the lockfile and the container image, and models through their exact identifier. The second is archiving results with their run card, the one from day fifteen, for a defined period. When someone asks why a result changed between two dates, you'll be able to compare which version of each thing was used on each, instead of trying to remember.

Illustrative example

A team reviews its eval pipeline after a month with higher spending than planned. On publishing tokens per run, they discover the format check, which runs on every pull request, uses the most expensive model. They move it to a local model started as a service on the runner. Format is validated just the same, and that step stops costing anything. Judged evals stay in the cloud. They also find that two of their three runners were persistent and had different versions of a tool, which explained different results for the same commit. They replace them with ephemeral containers with a pinned image.

Common mistakes

• Using the most expensive model for checks a local one handles.
• Keeping persistent runners with accumulated state.
• Letting an agent run in CI with production credentials.
• Sharing a single API key across all environments.

Checklist: how to know it is done

• Each type of eval has an assigned model, with its justification.
• Runners are ephemeral and the image pins versions.
• Agents run in a sandbox with no production credentials.
• Each run publishes tokens and spend next to the result.

Where AI fell short

A small local model is enough for smoke and format, not for judgment. Choosing a model on cost without measuring quality gets expensive.

Frequently asked questions

Doesn't a local model in CI make the pipeline slower?

Downloading the model is what takes longest. It's solved by including it in the runner image or caching it. After that, for format checks with short answers, the time is acceptable even without a GPU.

Which data can I use in CI?

Synthetic or anonymized. Production data shouldn't reach a pipeline or be sent to an external provider from there. Day six covers how to generate it and day seventeen, how to anonymize real cases.

Is it worth hosting large models on your own infrastructure?

Only if volume or privacy requirements justify it. Operating GPUs has a fixed cost and demands expertise. For most QA teams, a small local model plus an API with a spending cap is the simplest combination.

Day 29 of 31 of the 31 days of QA + AI series. Commands and versions reviewed in November 2026: rerun them before relying on them, AI tooling changes fast.

← Previous: Evals in CI: quality gates for AI features

Next: LLM observability in production: traces, online evals and alerts →