LLM evaluation
Deepen AI's trained raters score your model's outputs against your rubric, and give a reason for every score.
Rate in your plan before you pay A plan within 2 working days No sales call
What a scored output looks like
Each output comes back with a score on your rubric and the reason behind it.
- Customer asked
- Can I change my delivery address after the order has shipped?
- Your model answered
Yes, you can change it any time in your account settings.- Rater scored
- Fails on accuracyAccuracy 1 of 5. Helpfulness 3 of 5.
- Reason
- Once an order ships, address changes go through the carrier. The answer points to a setting that no longer works.
- Document
- 12-page supplier contract
- Your model summarised
- Three-year term, renews automatically, 60 days' notice to end it.
- Rater scored
- Passes5 of 5 on all four rubric lines.
- Reason
- The term, renewal and notice period match clauses 2.1, 2.3 and 9.4.
- Task
- Reply to a customer whose order is a week late.
- Your model wrote
Unfortunately, delays happen. Your order will arrive as soon as possible.- Rater scored
- Fails on toneTone 1 of 5. Accuracy 4 of 5.
- Reason
- Your rubric asks for an apology and a delivery date. The reply has neither.
- Customer asked
- What's the best laptop for video editing under $1,500?
- Your model answered
- Recommends a laptop your store doesn't sell.
- Rater scored
- Sent to youYour rubric doesn't say whether recommending products you don't sell is a failure. How should it be scored?
- Reason
- Your answer is added to the rubric, and earlier items are rescored to match.
How we start on your work
No sales call. A plan in your inbox, then a short paid pilot on your real outputs.
What we need from you
- Your rubric, or the qualities you care about
- 20 to 30 outputs you've already scored, or examples of good and bad answers
- Your outputs, in your eval tool or a shared sheet
- 5 minutesTell us the task
What your model does, how many outputs each round, and when you need results.
- Within 2 working daysGet a plan and a price
Raters, hours, the rate, start date and the total price.
- 2 weeksRun the pilot
We train raters on your rubric, check they agree, score real outputs and send two weekly reports. Then you decide.
Who rates, and how scores stay consistent
Trained raters, working in English. Each one learns your rubric and scores a set of known answers before rating live outputs.
Every reviewer's accuracy is measured
Items you've already answered are hidden in the real work, so we know how accurate each person is.
Nobody guesses on unclear cases
The team lead decides, or asks you. Your answer joins the guidelines.
You see the results every week
A weekly report shows work done, QA results and what's improving.
Standard evaluation covers chat, search answers, summaries, extraction and generated copy. Raters with specialist domain knowledge are quoted in your plan.
How much does LLM evaluation cost?
LLM evaluation is quoted in your plan, because the rate depends on the expertise and turnaround the work needs. You pay for the hours raters work, with a team lead, QA checks and a weekly report included.
Set before you pay. Your plan shows the rate, team size, hours and total price.
Starts with a two-week pilot on your real outputs, paid upfront, with two weekly reports.
Your outputs stay confidential. We work under NDA with the access you set. Deepen AI holds SOC 2 Type II, ISO 27001, ISO 9001 and TISAX.
Questions
Can you check our automated judge?
Yes. Raters score the same items as your LLM judge, and the report shows where the two agree and where the judge goes wrong, so you know how far to trust it.
How many outputs should we score?
It depends on how small a difference you need to see between versions. We agree the number per round with you, and it's written into your plan.
How do you keep scores consistent?
Every rater scores the same known items before live work, and we check how often they agree. Known items stay mixed into the work, and agreement is in your weekly report.
Our rubric is loose. Is that a problem?
The team lead flags rubric lines that raters read differently, and proposes clearer wording for you to approve. The rubric gets sharper with every round.
Can we compare releases over time?
Yes. Keep the test set and rubric fixed, and every new version is scored the same way, so the results compare from one release to the next.
Do our outputs stay confidential?
Yes. We work under NDA, inside your tools, with the access you set, and you can remove it at any time.