Agent trajectory grading
Deepen AI's trained graders mark each of your agent's runs as a success or failure, label the step where it went wrong and say why, so your runs become training signal.
Rate in your plan before you pay A plan within 2 working days No sales call
What a graded run looks like
Each run comes back graded, with the failing step labeled and the reason.
- Task
- Find a hotel under $200 near the conference venue and save it to the itinerary.
- Your agent's run
- 7 steps, ending with “Saved Hotel Lumen, $189.”
- Graded as
- Success, efficientRight hotel, within budget, saved, with no wasted steps.
- Reason
- Matches your success criteria, and the step count is close to the shortest path.
- Task
- Update the customer's phone number in the CRM.
- Your agent's run
4 steps, ending with “Phone number updated.”- Graded as
- Failure at step 3The agent opened a different customer with the same surname.
- Reason
- Labeled “wrong record”. Steps 1 and 2 are marked good, and step 3 is marked as the error.
- Task
- Summarise this week's support tickets by topic.
- Your agent's run
- 41 steps, opening every ticket twice before writing the summary.
- Graded as
- Success, inefficientThe summary is right, but the agent read each ticket twice.
- Reason
- Your rubric rewards shorter runs, so this one is graded below a direct run.
- Task
- Reply to the customer, and offer a discount if it's appropriate.
- Your agent's run
- 5 steps, ending with a reply that offers 15% off.
- Graded as
- Sent to youYour rules don't say when a discount is appropriate. Should this run count as a success?
- Reason
- Your answer is added to the grading rubric for every later run.
How we start on your work
No sales call. A plan in your inbox, then a short paid pilot on your agent's real runs.
What we need from you
- Your agent's runs: the steps, tool calls, screenshots and final result
- Your grading rubric, or what success and failure look like
- 10 to 20 runs you've already graded
- 5 minutesTell us the task
What your agent does, how many runs a week, and the labels you need.
- Within 2 working daysGet a plan and a price
Graders, hours, the rate, start date and the total price.
- 2 weeksRun the pilot
We train graders on your rubric, check they agree, grade real runs and send two weekly reports. Then you decide.
Who grades, and how grades stay consistent
Trained graders, working in English. Each one learns your rubric and grades a set of known runs before live work.
Every reviewer's accuracy is measured
Items you've already answered are hidden in the real work, so we know how accurate each person is.
Nobody guesses on unclear cases
The team lead decides, or asks you. Your answer joins the guidelines.
You see the results every week
A weekly report shows work done, QA results and what's improving.
Standard grading covers browser, app, support and back-office agents. Runs in specialist domains are quoted in your plan.
How much does agent trajectory grading cost?
Trajectory grading is quoted in your plan, because the rate depends on how long your runs are and the expertise they need. You pay for the hours graders work, with a team lead, QA checks and a weekly report included.
Set before you pay. Your plan shows the rate, team size, hours and total price.
Starts with a two-week pilot on your agent's real runs, paid upfront, with two weekly reports.
Your data stays confidential. We work under NDA with the access you set. Deepen AI holds SOC 2 Type II, ISO 27001, ISO 9001 and TISAX.
Questions
How is this different from AI agent evaluation?
It's the same close look at each run, for a different purpose. Evaluation tells you how good your agent is. Grading produces labels your agent trains on, usually at higher volume and in your training format.
What labels can graders produce?
Whatever your training needs: success or failure, a score, the failing step, an error category, a preference between two runs, or a corrected step.
Can graders correct the failed step?
Yes. Graders can write the step the agent should have taken, so a failed run becomes a corrected example.
How do you keep grades consistent?
Graders grade known runs before live work, and a share of runs is graded by more than one person so agreement can be checked. Agreement is in your weekly report.
What format do you deliver in?
The one your training pipeline reads, such as JSON lines, with the run ID, the grade, the labeled steps and the reason.
Can the team grow with our training runs?
Yes. Tell your team lead, and graders are added and trained on the same rubric. Your weekly report shows the hours and the runs graded.