Blogs
How an Evaluation Harness Makes AI Results Reproducible, Costed, and Traceable
Every enterprise faces recurring questions that require correct and consistent answers. This was simpler when all enterprise code performed deterministic tasks. With the rise of AI and agentic workflows, tasks and responses have become more probabilistic, and the same metrics no longer work for agents. This is because your agents follow orders well. In fact, they are so good that they’ll invent facts (“hallucinate”) to produce a viable answer rather than say they don’t know. These hallucinations and errors have causes, but you can’t see them in the output because a fabricated answer can appear just as confident as a correct one.
To illustrate these concepts better, we'll discuss a mock requirement and deliverable from a bank. Assume the client provides us with a loan file that includes the code and its governing policy. Engineers need to determine which symbol the file calls and what rule the loan states at that bank. An error in either leads to a failed review. The answer must also remain valid over time, even as the code, policy, and model evolve.
To address this, we conduct A/B/C testing across three proposed solutions. The first is a baseline: an agent with tools but no context, which searches the repository independently. The second uses a knowledge graph built from the code, mapping each method call as a node and edge, allowing the agent to reference structure directly. The third combines the graph with business rules, incorporating policy text to inform the agent of permitted actions.
Each solution answers the same eight questions: four about which symbol a loan file calls and four about the rule it states at a given bank. We took all of our candidate agents and our questions and created a mock exam for the agents. The baseline answered four policy questions correctly at a cost of $0.0081797. The graph answered the four code questions at $0.0003042. The graph with rules answered all eight at $0.0003024.
| Configuration | Context provided | Score | Context provided |
|---|---|---|---|
| bare | None | 4 / 8 (policy only) | $0.0081797 |
| map | Mapped chunk | 4 / 8 (code only) | $0.0003042 |
| map_rules | Mapped chunk + rule text | 8 / 8 | $0.0003024 |
Table 1. Results by context configuration (8 tasks each).
Two solutions achieved the same score, but their costs differed by a factor of 27. Each failed in different areas: One understood the code but not the rules, while the other understood the rules but not the code. In the baseline, four failed retrieval calls accounted for about half the total cost, highlighting the expense of unsuccessful retrievals.
If we were to present this to a team, we would expect responses like:
“Which one should we ship?”
“The table says these two are the same.”
“Then the table is wrong.”
The table accurately reported counts but omitted critical context. It did not specify the number of tool calls, input and output tokens used, the pricing used, or the extent of noise in the 4-out-of-8 result. Any change in these factors could alter the result without changing the solution. The table provided a result, but the team needed evidence.
In this context, evidence means a result that others can reproduce using the same cases, repository state, judge configuration, prices, and a sufficient sample size to avoid misinterpreting minor differences. Evidence must be reproducible, costed, and traceable. A demo demonstrates a single occurrence; evidence demonstrates frequency and reliability.
This distinction matters because incorrect answers in this domain can look as confident as correct ones. The count alone cannot resolve the issue, making the missing metadata more significant than the result itself.
This is not a debate about which context strategy is superior. Instead, it addresses what a release team requires to trust any solution. While the graph-plus-rules approach performed best, we could not explain why it would remain efficient over time.
The Problem
Each component changes independently: model, prompt, context, judge, and repository all evolve separately. Releases evaluate multiple configurations.
An ad hoc script captures one configuration. Changing the model, judge, or repository alters results without notice, making outcomes unreliable and irreproducible across releases.
Cost must be considered alongside performance. Outperforming by one task at significantly higher cost isn't necessarily preferable. Changing the judge can create an illusion of progress through scoring changes rather than improvement.
A release decision requires:
- Consistent cases for evaluation
- A fixed repository state
- Recorded and versioned judge configuration
- Transparent pricing details
- Sufficient sample size to ensure minor differences are not misinterpreted as significant
These are standard requirements, yet they were absent from the script that generated the results in Table 1.
The core issue is that teams face a demo problem disguised as evaluation. A result that cannot be reproduced isn't.
The script provided only inconsistent temporary answers.
The Answer: An Eval Harness
An Eval Harness is the instrument answering the release question: how often, at what cost, under which code, and whether it happens again. It starts with one YAML spec naming cases, repositories, candidates, metrics, prices, and gates. That file is the source of truth. A run creates a snapshot.
To create this YAML spec, enumerate test cases and their sources. As we did in our mock run, we define candidates including adapter, model, prompt, and context. List metrics and pricing for model calls, retrieval, and judges. Specify gates or thresholds for acceptance. Version the spec with repository changes to keep results traceable and reproducible. Your YAML spec serves as the overall blueprint.
The planner expands the spec into cells. A cell is one candidate run against one test case, repeated per replica, and it is the unit of execution, cost, and evidence. Cell IDs are deterministic, so the same spec produces the same matrix twice. Before cell one, the harness snapshots.
Context selection runs once per candidate and case before cells run. Each cell keeps chunk IDs and selection.
Six protocols define the system: Candidate, Judge, Metric, Source, Sink, and ContextSelector. Core imports protocols and a registry, never concrete adapters. This allows a retrieval-augmented generation (RAG) pipeline, HTTP endpoint, and provider API in one.
Why Build the Eval Harness Like Release Infrastructure?
The harness runs like a service rather than a script. The CLI and HTTP API drive the same orchestration. A run keeps one ID through cancel and resume, concurrency is bounded, retries stop at budget, and a budget guard stops the run when spend crosses.
It treats inputs as evidence, storing spec and price snapshots, pinning judge configurations with hashes, recording repository digest and commit, and keeping context chunk IDs per cell. The receipt collects these facts; the diff compares them with the previous run, so a judge swap shows drift.
It is honest about uncertainty. The plain rate carries its n and a 95% Wilson interval; replicas add case-level pass@k and passk. The decision view puts cost per correct check beside the rate, and the compare view adds cost per correct answer. When the interval includes zero, the compare page calls it inconclusive.
It is built to be trusted, not certified. Keys are BYOK, encrypted at rest, and redacted from logs, events, artifacts, and HTML. The suite holds a 100% line coverage gate and parity test replaying a recorded three-arm scoreboard. Conformance runs per adapter kind, and a smoke script drives the running service. The console is server-rendered.
What an Eval Harness Run Looks Like
Before the first cell, validate the spec. We parse it, check it, and refuse it before any spend if a field is wrong; the errors name the field, the value, and the fix. Snapshot the repository when the spec names one. Then come the six stages, in four moves.
- Plan and Select: The planner builds the matrix and fills context for each case.
- Execute: Cells run under limited concurrency and the budget guard, and each candidate returns an answer and usage.
- Score: Metrics turn cells into checks. A judge-backed metric gets a verdict whose tokens are priced and charged to the triggering cell, separate from answer cost and rolled up per provider.
- Record and Read: The CLI prints the run, its cost split, and the aggregates, and the console, JSON API, and receipt read the same rows.
This is a real smoke run with two stub candidates and no network. Only one aggregate row is displayed. The focus is on the shape of the result rather than the score.
run cdfd34e19b6547168eaad9ae54865371 status=ok cells=2 ok=2 cost=$0.000013
cost: answer $0.000013, judge $0.000000, tools $0.000000
stub-concise exact_match n=1100.0%, range: 20.7% to 100.0%
The run ID, interval, and cost analysis are consolidated in a single location. The receipt contains the spec hash, harness revision, operator, seed, judge identifiers, price fingerprint, source digest, and any differences.
Figure 1. One run, end to end.
Getting Started with an Eval Harness: Choose Twenty Meaningful Cases
The solution is more straightforward than the debate. For teams adopting this method, start by selecting twenty defensible cases that represent common, meaningful scenarios for your application. Choose cases that are clear, stable, and easy for the team to review. Set up an initial candidate and configure a gate to fail the build if performance falls below an agreed benchmark. Use the eval harness to test your candidate on these cases. Retain the receipt, including the spec, repository state, judge configuration, and prices, to ensure results are auditable and reproducible. Display the confidence interval with the score to clearly communicate uncertainty. As the process stabilizes, review with the team whether the cases and the candidate still align with your objectives, and then introduce a second candidate for comparison. The first run may not be perfect, but this is acceptable if subsequent runs are comparable and improvements are intentional.
To operationalize this process and automate evidence collection, integrate the eval harness into your CI/CD pipeline. Configure the pipeline to automatically execute the evaluation suite on each commit or pull request, using the current spec and repository state. Set up release gates so that builds are promoted only if evaluation results meet predefined benchmarks, such as minimum accuracy or maximum cost thresholds. Store receipts and run artifacts as pipeline outputs for traceability. This approach ensures every release is backed by reproducible evidence and makes quality gates enforceable as part of your delivery process.
These steps do not guarantee model trustworthiness, but they ensure auditable results that are essential for release decisions. A demo presents a claim, while the harness provides the evidence.
View the entire demo here: https://github.com/kanitmann01/eval-harness