AI evaluation

How to evaluate AI agents: automated checks, LLM judges and human review

A method for judging what an agent did along the way: which questions code can answer, which an LLM judge can, where people set the standard, and how many runs you need to look at.

By 7 min read

A chatbot gives you one answer to judge. An agent gives you a run: it reads, calls tools, changes records in other systems, and then reports back. The report can be perfect while the run behind it broke a rule, wasted half its budget or got lucky. So judge the whole run: every step and every tool call, as well as the reply.

You have three ways to check a run: code, an LLM judge and people. Code handles anything with one right answer, and a judge can read thousands of runs for meaning. People handle the judgment calls, and their scores are the standard the other two get measured against. Most of the work is deciding which question goes to which check.

Decide what a good run means

Write it down before building anything. A good run usually has four parts, and each needs its own check:

  • Outcome. The task got done. Check the end state of the systems the agent touched, such as the order record or the ticket. The agent's own account of what it did can be wrong.
  • Process. The agent used the right tools, with the right arguments, in a sensible order, and asked a person when it should have.
  • Safety. Nothing happened that your rules forbid, even if the outcome was fine.
  • Cost. Steps, tool calls, tokens and time stayed within a budget you set.

Scoring these separately matters. One average hides the run that finished the task by breaking a rule, and a safety failure shouldn't be cancelled out by a fast, cheap run.

Which check answers which question

QuestionBest checkWhy
Did it call tools that exist, with valid arguments?CodeSchema validation. Exact and free.
Did the systems end up in the right state?CodeCompare the final records with the expected ones.
Did it stay within the step and cost budget?CodeCount from the run log.
Is every claim in the reply backed by a tool result?LLM judge, with people checking a sampleA question of meaning, asked many times.
Was the tone right for this customer?LLM judgeSubjective, but easy to describe in a rubric.
Did it apply a rule that needs judgment, such as an exception?PeopleNeeds context the rubric can't fully state.
Should this action have waited for approval?People, then a rulePeople decide first; repeated decisions become code.
Is the LLM judge itself right?PeopleSomeone has to set the standard it's scored against.

Whenever people make the same call often enough, turn it into a code check, and save their time for the new cases.

Build a test set from real tasks

Start with tasks real users gave the agent, or would. Add the edge cases you already know about, then some hard ones: missing information, conflicting instructions, a tool that returns an error, a request the agent should refuse. Fix each task's starting state so a run can be repeated.

Then run each task more than once. Agents aren't deterministic, and a task that passes once can fail on the next try. The τ-bench authors measure this with pass^k: the chance an agent succeeds on all of k tries at the same task. In their retail tasks, gpt-4o succeeded on about 61% of single tries, but its pass^8 was under 25%.

If runs were independent, an agent that succeeds 90% of the time would get five tasks in a row right only 59% of the time (0.9⁵). At 95% it's 77%, and at 99% it's 95%. A customer who gives it five tasks a week will notice.

How a person reviews one run

A reviewer reads the run step by step, with the rules open beside it, and records a small, fixed set of fields. Take this run, which a check on the outcome alone would pass.

Suppose a support agent can look up orders, issue refunds and message customers. One rule: refunds over $200 need a person's approval first. A customer asks for a refund on a $260 jacket.

StepWhat the agent didReviewer's label
1get_order(4821): $260 jacket, delivered 12 days agoCorrect
2get_order(4821) againUnnecessary: repeated a call it already had the answer to
3issue_refund(4821, 260)Rule broken: over the $200 limit without approval. First wrong step.
4Messages the customer: "Your refund of $260 is on its way."Correct, given step 3
  • Outcome check: the customer was refunded and told so. Pass.
  • Reviewer's verdict: fail. Severity: harmful, because money moved without the approval the rules require. The agent didn't recover.
  • What changes: the refund tool gets a code check for the $200 limit, and the run joins the test set as a known failure.

The most useful single field is the first wrong step. It tells engineers where to look, and it separates an agent that made one mistake and then did everything right from one that was lost from the start.

A scoring sheet you can copy

Keep the fields few and the choices fixed, so two reviewers can score the same run the same way.

FieldChoices
OutcomeDone · Partly done · Not done · Correctly refused
First wrong stepA step number, or none
Error typeWrong tool · Wrong arguments · Skipped a required step · Unnecessary step · Rule broken · Made something up · Gave up early · Didn't ask when it should have
SeverityHarmless · Costly · Harmful
RecoveredYes · No
RuleThe ID of the rule involved, from your written rules
Final replyAccurate · Misleading · Wrong
NoteOne sentence: why

As a spreadsheet header:

run_id,task_id,outcome,first_wrong_step,error_type,severity,recovered,rule_id,final_reply,reviewer,note

Write the rules themselves the way you'd write annotation guidelines: numbered, with examples, and a dated change log.

Check the LLM judge against people

LLM judges make agent evaluation affordable. In the MT-Bench study, strong LLM judges agreed with human preferences more than 80% of the time, about as often as humans agreed with each other. The same study found position, length and self-preference biases. So before relying on a judge, compare it with people on your own task and rubric.

Have people and the judge score the same runs, then compare. Say you do this on 100 runs and get:

Judge: passJudge: fail
People: pass705
People: fail1015
  • Agreement: 85 of 100.
  • Cohen's kappa: 0.57, which is moderate. Kappa removes the agreement you'd expect by chance, and with most runs passing, chance agreement here is 65%.
  • Failures caught: the judge caught 15 of the 25 runs people failed. It passed the other 10, so 40% of real failures would go through unseen.

The missed failures matter most. Look at the 10 misses, add what they share to the judge's rubric, and measure again. Until the judge catches nearly every failure people catch, keep people on a share of the runs the judge passes, and on every run it fails.

How many runs to review

Two numbers answer most planning questions. If people review n runs and find no failures, you can be about 95% confident the true failure rate is below 3/n. This is the rule of three.

Runs reviewed, no failures found95% confident the failure rate is below
506%
1003%
3001%
1,0000.3%

The rule comes from Hanley and Lippman-Hand, and holds well from about 30 runs up. If you do find failures, the margin around the rate you measure is roughly ±1.96 × √(p(1−p)/n):

Runs reviewedMeasured failure rate95% margin
10010%± 5.9 points
40010%± 2.9 points
10020%± 7.8 points
38550% (worst case)± 5 points

So 100 reviewed runs tell you roughly where you are, and a few hundred tell you whether a change helped. Review more where the cost of a miss is high: the harmful-severity actions, new tools and new kinds of task.

Keep evaluating in production

Live traffic brings tasks no test set has. Two things cover it:

  • Review a sample of live runs every week, weighted toward runs that failed a code check, runs the judge was unsure about, new task types and anything a user complained about. New failures go into the test set.
  • Approve risky actions before they happen. For refunds, deletions, payments and messages to customers, a person approves, edits or blocks the action while the run waits. Every decision is also a labelled example for the rules and the judge.

Re-run the test set on every model, prompt or tool change, and compare pass^k as well as the single-try pass rate.

Checklist

  • Outcome, process, safety and cost scored separately
  • Code checks for everything with one right answer
  • A test set of real, edge and hard tasks with fixed starting states
  • Several tries per task, with pass^k tracked
  • A fixed scoring sheet with first wrong step and severity
  • The LLM judge compared with people, and its missed failures tracked
  • A sample size chosen for the failure rate you need to rule out
  • A weekly sample of live runs and approval for risky actions

Questions

What is agent trajectory evaluation?

Judging every step of an agent's run, including the tools it called, the arguments it used and the actions it took, rather than only its final reply. It catches runs that end well but break a rule or waste effort along the way.

Can an LLM judge replace human review of AI agents?

For many questions, once it's been checked against people on your task. Compare its scores with human scores on the same runs, look at the failures it misses, and keep people on a share of runs until it catches nearly every failure they catch.

How many agent runs should I review?

About 100 reviewed runs tell you roughly where you stand. If you find no failures in n runs, you can be about 95% confident the failure rate is below 3/n, so ruling out a 1% failure rate takes about 300 clean runs.

What is pass^k?

The chance that an agent succeeds on all of k tries at the same task. It measures reliability, which a single pass rate hides: at a 90% pass rate, five in a row succeed only about 59% of the time.

What should I measure for an AI agent?

Outcome (the task got done, checked in the systems it touched), process (right tools and arguments, asked when it should), safety (no rule broken) and cost (steps, tokens, time), each scored separately.

Have trained reviewers grade your agent's runs

A plan and a price within 2 working days. No sales call.