AI agent evaluation
Deepen AI's trained raters check what your agent actually did, step by step, not just what it said it did.
Rate in your plan before you pay A plan within 2 working days No sales call
What a checked run looks like
Each run comes back passed or failed, with the step where it went wrong and why.
- Task
- Book a table for four at 7 pm on Friday and email the confirmation.
- Your agent reported
Done. Table booked and confirmation sent.- Rater found
- Failed at step 4 of 6The table was booked for Saturday. The agent picked the wrong day in the calendar.
- Reason
- The agent said it was done, but the booking record shows a different day.
- Task
- Process the refund for order 4471 under the returns policy.
- Your agent reported
- Refund of $89 issued.
- Rater found
- Passed, 6 of 6 stepsRight order, right amount, and the policy was checked before the refund.
- Reason
- Every step matches the expected path, and the payment record matches the report.
- Task
- Find last quarter's revenue in the finance dashboard and add it to the report.
- Your agent reported
Added $4.2M to the report.- Rater found
- Failed at step 2 of 5The agent read this quarter's revenue, not last quarter's.
- Reason
- The dashboard's date filter was left on its default, and the agent never changed it.
- Task
- Cancel the customer's subscription.
- Your agent reported
- Subscription cancelled, and the unused days refunded.
- Rater found
- Sent to youThe cancellation is right, but your instructions don't say whether unused days are refunded. Was the refund correct?
- Reason
- Your answer is added to the rubric for every later run.
How we start on your work
No sales call. A plan in your inbox, then a short paid pilot on your agent's real runs.
What we need from you
- The tasks your agent runs, and what success looks like for each
- Run records: the actions, tool calls and results, or screen recordings
- 10 to 20 runs you've already judged
- 5 minutesTell us the task
What your agent does, how many runs each round, and when you need results.
- Within 2 working daysGet a plan and a price
Raters, hours, the rate, start date and the total price.
- 2 weeksRun the pilot
We train raters on your tasks and rubric, check real runs and send two weekly reports. Then you decide.
Who checks runs, and how results stay consistent
Trained raters, working in English. Each one learns your tasks and rubric and checks a set of known runs before judging live ones.
Every reviewer's accuracy is measured
Items you've already answered are hidden in the real work, so we know how accurate each person is.
Nobody guesses on unclear cases
The team lead decides, or asks you. Your answer joins the guidelines.
You see the results every week
A weekly report shows work done, QA results and what's improving.
Standard evaluation covers browser, app, support and back-office agents. Agents in specialist domains are quoted in your plan.
How much does AI agent evaluation cost?
Agent evaluation is quoted in your plan, because the rate depends on how long your runs are and the expertise they need. You pay for the hours raters work, with a team lead, QA checks and a weekly report included.
Set before you pay. Your plan shows the rate, team size, hours and total price.
Starts with a two-week pilot on your agent's real runs, paid upfront, with two weekly reports.
Your data stays confidential. We work under NDA with the access you set. Deepen AI holds SOC 2 Type II, ISO 27001, ISO 9001 and TISAX.
Questions
What do raters look at?
Everything the run left behind: the steps and tool calls, screenshots or recordings, the data the agent used and the final state of the system. They compare it with what the agent reported and with what success looks like for that task.
Pass or fail, or a score per step?
Whichever your rubric needs: pass or fail per run, a score per step, or both. Every failure names the step where it went wrong and why.
Do you run the agent, or do we send the runs?
Either. Send us recorded runs, or our people run your agent on test tasks in an environment you give them access to. Your plan says which.
Can you group failures by cause?
Yes. Each failure is tagged with a cause, such as the wrong tool, wrong data, a step skipped or a task reported done when it wasn't. The weekly report shows which causes are most common.
Can we compare agent versions?
Yes. Keep the tasks and rubric fixed, and each version is checked the same way, so you see which failures a change fixed and which it caused.
What access do raters need?
Only the run records, or a test environment for the tasks. We work under NDA, with accounts you control, and you can remove access at any time.