Agent Evaluation Architecture
Measuring whether an agent actually works — layered deterministic checks, golden datasets, judge models, and trajectory scoring, wired into CI and production so quality regressions are caught before users find them.
By Jai Ganesh
What is agent evaluation architecture?
Agent evaluation architecture is the system that measures whether an AI agent actually works — before deployment and continuously after. It layers deterministic checks, golden datasets with known-correct outcomes, LLM-as-judge scoring for qualitative criteria, and trajectory evaluation that grades the agent’s path, not just its final answer. Wired into CI and production monitoring, it turns “the agent feels worse lately” into a regression you can bisect — and it is the prerequisite for widening agent autonomy safely.
The Problem
Agents fail differently than software. A unit test proves a function returns the right value; an agent can return a different, equally plausible answer every run — and the worst failures look like successes. A confident wrong answer, a tool called with subtly wrong arguments, a workflow completed by an expensive and fragile path: none of these throw exceptions, and none show up in a test suite built for deterministic code.
So most teams ship on vibes. Someone edits a prompt, tries five questions, and merges. Quality drifts one plausible-looking regression at a time, and the first reliable signal is a user complaint — or a quiet loss of trust that never files a ticket. Meanwhile every model upgrade is a gamble, because there is no baseline to compare against.
Evaluation architecture treats agent quality as a measured property of the system, not an impression. It is also the gate on everything else this library describes: autonomy thresholds, model swaps, prompt changes, and cost optimizations are all safe exactly to the degree that an evaluation harness can prove they did not make the agent worse.
The Architecture
How It Works
Golden datasets
The foundation: curated sets of real inputs with known-correct outcomes — the answer, the expected tool calls, the required escalations. Each authority domain gets its own set, and every case earns its place: representative of real traffic, discriminating enough to catch regressions, and labeled with why it exists. Fifty sharp cases beat five hundred vague ones.
Deterministic checks first
The cheapest layer runs first: schema validity, required citations present, no forbidden actions, format contracts honored, latency and token budgets respected. These are binary, instant, and free of judgment — a large share of regressions are caught here before any model-graded evaluation spends a token.
LLM-as-judge for qualitative criteria
Groundedness, tone, completeness, and helpfulness cannot be asserted with string matching. A judge model scores them against explicit rubrics — one criterion per judgment, with the rubric versioned like code. Judges are calibrated against a set of human-labeled examples, so “the judge says 4.2” has a known relationship to “a human would agree.”
Trajectory evaluation
Agents are graded on the path, not just the destination. Did it choose the right tool, or brute-force through the wrong one? Did it recover from a failed call or spiral? Did it reach the answer in four steps or forty? Trajectory scoring catches the failure outcome-only evals miss: right answers produced by processes that will not survive contact with harder inputs.
Regression gates in CI
Every prompt edit, model upgrade, and tool change runs the full harness before merge, and the gate compares score distributions against the current baseline. A drop blocks the change with a diff of exactly which cases degraded — turning “did this make the agent worse?” from a debate into a build status.
Online evaluation and drift monitoring
Offline evals prove readiness; production proves reality. A sample of live traffic is scored continuously with the same judges and rubrics, tracked as trends rather than incidents. Drift — from changing user behavior, upstream model updates, or shifting content — shows up as a slope on a chart weeks before it becomes a complaint. Production failures are triaged into new golden cases, which is how the harness compounds.
Design Decisions
Evaluate trajectories, not just outcomes
Outcome-only evaluation certifies agents that get right answers by fragile means — until the input distribution shifts and the fragility surfaces in production. Scoring tool selection, step efficiency, and recovery behavior evaluates the process you are actually shipping, not just today’s results from it.
The judge must be independent of the judged
A model grading its own family’s output inherits its family’s blind spots — errors both make look like agreement. A different model, a separate prompt, and no shared context keep the judge adversarial. Self-evaluation is a reflection step, not an evaluation.
Curate golden sets from production failures, not imagination
Cases invented in a workshop test the failures you already anticipated. Cases harvested from escalations, corrections, and complaints test the failures that actually happen — and every production incident that becomes a golden case is a regression that can never ship silently again.
Eval gates as merge criteria, not dashboards
A dashboard nobody is forced to look at is decoration. Wiring the harness into CI makes quality a blocking property of every change — the same social contract as a failing test suite, applied to behavior instead of code.
Score distributions over pass/fail
Binary gates catch collapses and miss decay. Tracking the distribution — means, tails, per-category slices — surfaces the slow degradations and the localized ones: an aggregate score that held steady while one critical category quietly slid.
Trade-offs & Limits
- ›Evaluation is permanent curation work: golden sets, rubrics, and judge calibration need an owner, or the harness slowly certifies against a world that no longer exists.
- ›LLM-as-judge inherits judge bias — position effects, verbosity preference, leniency drift. Calibration against human labels is recurring maintenance, not one-time setup.
- ›Online evaluation costs inference on top of serving. Sampling rates are a genuine cost/coverage tradeoff, tuned per use case rather than set globally.
- ›A harness can overfit: teams optimizing scores instead of quality. Periodic blind human review keeps the metric honest.
When To Use It
- ✓Any agent moving from prototype to production — the harness is the promotion criteria
- ✓Before widening autonomy: evidence, not optimism, is what justifies raising thresholds
- ✓Regulated domains where quality claims need documentation behind them
- ✓Platforms where many teams change prompts, models, and tools that ripple into each other
When Not To
- ✕Throwaway prototypes — spot-checking is honest at that stage; build the harness when something is worth protecting
- ✕Fully deterministic pipelines where conventional tests already cover the behavior
- ✕Single-shot generation reviewed by a human every time — the human is the evaluation layer
Enterprise architect and independent AI consultant — I help teams take agentic systems from deck to production, with the governance story intact.