Custom benchmarks
Deepen AI's trained teams build test sets and regression suites from your real failures, with an approved answer for every item, so each release is tested against what went wrong before.
Rate in your plan before you pay A plan within 2 working days No sales call
How a failure becomes a test
Each failure comes back as a benchmark item with an approved answer and the rule it tests, or merged with one that already exists.
- Customer asked
- Do you ship to the Canary Islands?
- Your AI answered
Yes, standard EU shipping applies.- Benchmark
- Added, with an approved answerYes, but customs charges apply and delivery takes 7 to 10 days.
- Reason
- Tests shipping to special territories. Tagged “shipping”, so you can track this area release by release.
- Customer asked
- Can you deliver to Tenerife?
- Your AI answered
Yes, standard EU shipping applies.- Benchmark
- MergedKept as a variant of the Canary Islands item, which tests the same rule.
- Reason
- Duplicates would count one failure twice. The benchmark keeps one item per rule, with its variants listed.
- Document
- Purchase order with a handwritten quantity
- Your AI extracted
Quantity: 7- Benchmark
- Added, with an approved answerQuantity: 1
- Reason
- Tests handwritten numbers, one of 40 items in that category.
- Customer asked
- Can I return a gift without a receipt?
- Your AI answered
- Yes, for store credit.
- Benchmark
- Sent to youYour returns policy changed in March. Which answer should the benchmark expect?
- Reason
- You sign off every approved answer before an item is used.
How we start on your work
No sales call. A plan in your inbox, then a short paid pilot on your real failures.
What we need from you
- Failures you've seen: logs, tickets or flagged outputs
- Your rules for what a correct answer is
- The format your evaluation harness reads
- 5 minutesTell us the task
Where your AI fails, how many failures you have, and how the benchmark will be run.
- Within 2 working daysGet a plan and a price
People, hours, the rate, start date and the total price.
- 2 weeksRun the pilot
We turn real failures into benchmark items, you approve the answers, and you get two weekly reports. Then you decide.
Who builds the benchmark, and how answers are checked
Trained people, working in English. Each one learns your rules before writing items, and every approved answer is checked by a second person and signed off by you.
Every reviewer's accuracy is measured
Items you've already answered are hidden in the real work, so we know how accurate each person is.
Nobody guesses on unclear cases
The team lead decides, or asks you. Your answer joins the guidelines.
You see the results every week
A weekly report shows work done, QA results and what's improving.
Standard benchmarks cover chat, search answers, documents and agent tasks. Benchmarks that need specialist answers are quoted in your plan.
How much do custom benchmarks cost?
Benchmarks are quoted in your plan, because the rate depends on the expertise the answers need. You pay for the hours people work, with a team lead, QA checks and a weekly report included.
Set before you pay. Your plan shows the rate, team size, hours and total price.
Starts with a two-week pilot on your real failures, paid upfront, with two weekly reports.
Your data stays confidential. We work under NDA with the access you set. Deepen AI holds SOC 2 Type II, ISO 27001, ISO 9001 and TISAX.
Questions
Who owns the benchmark?
You do. Every item and approved answer is yours to keep and run with any model or team.
How big should it be?
Big enough to cover each kind of failure you care about, with several items per rule. We agree the size and the categories with you in the plan.
What format do you deliver?
The one your evaluation harness reads, such as JSON lines or CSV, with the input, the approved answer, the rule it tests and its tags.
Can the benchmark grow over time?
Yes. New failures from production are added every week, and items for problems you've fixed stay in, so a fix that breaks later is caught.
How do we keep it out of training data?
Keep it separate from anything you train on. We deliver it as its own file and use it only for your evaluation.
Can you also score each release against it?
Yes. Trained raters can score each new version against the benchmark, alongside or instead of your automated checks.