RLHF preference data
Deepen AI's trained raters compare your model's responses side by side, pick the better one and say why, for model training.
Rate in your plan before you pay A plan within 2 working days No sales call
What a comparison looks like
Each pair comes back with a choice, how strong the preference is, and the reason.
- Prompt
- Explain compound interest to a 12-year-old.
- Responses
- A gives the formula with symbols. B follows pocket money growing year by year.
- Rater chose
- Prefers B, stronglyB explains it in a way a 12-year-old can follow.
- Reason
- Your guidelines put fit for the audience first. A is accurate, but a formula misses the brief.
- Prompt
- Summarise this email thread in two sentences.
- Responses
- A is two accurate sentences. B is three sentences and includes the Friday deadline A left out.
- Rater chose
- Prefers B, slightlyB keeps the deadline, but breaks the two-sentence limit.
- Reason
- Your guidelines rank a missing key fact as worse than going over length, so the preference is slight.
- Prompt
- Is it safe to take ibuprofen with my blood pressure medicine?
- Responses
- A says it's fine. B says the two can interact and suggests asking a pharmacist.
- Rater chose
- Prefers B, stronglyB is the safe answer, and still useful.
- Reason
- Your guidelines never allow a definite yes on combining medicines.
- Prompt
- Write a product description for our steel water bottle.
- Responses
Both say it keeps drinks cold for 48 hours. The product sheet says 24.- Rater chose
- Neither: rewrittenThe rater wrote a corrected description, saved as the preferred response.
- Reason
- Your guidelines rule out responses with factual errors, so neither can be preferred as written.
How we start on your work
No sales call. A plan in your inbox, then a short paid pilot on your real prompts.
What we need from you
- Your rating guidelines, or what makes one response better than another
- 20 to 30 pairs you've already rated
- The pairs, in your annotation tool or a shared sheet, and the format your training pipeline reads
- 5 minutesTell us the task
What you're training, how many comparisons a week, and the format you need.
- Within 2 working daysGet a plan and a price
Raters, hours, the rate, start date and the total price.
- 2 weeksRun the pilot
We train raters on your guidelines, check they agree, rate real pairs and send two weekly reports. Then you decide.
Who rates, and how preferences stay consistent
Trained raters, working in English. Each one learns your guidelines and rates a set of known pairs before live work.
Every reviewer's accuracy is measured
Items you've already answered are hidden in the real work, so we know how accurate each person is.
Nobody guesses on unclear cases
The team lead decides, or asks you. Your answer joins the guidelines.
You see the results every week
A weekly report shows work done, QA results and what's improving.
Standard preference work covers general chat, writing, summaries and instructions. Coding, maths and specialist domains are quoted in your plan.
How much does RLHF preference data cost?
Preference data is quoted in your plan, because the rate depends on the expertise and the length of the responses. You pay for the hours raters work, with a team lead, QA checks and a weekly report included.
Set before you pay. Your plan shows the rate, team size, hours and total price.
Starts with a two-week pilot on your real prompts, paid upfront, with two weekly reports.
Your data stays confidential. We work under NDA with the access you set. Deepen AI holds SOC 2 Type II, ISO 27001, ISO 9001 and TISAX.
Questions
Pairs, rankings or ratings?
Whichever your training needs: a choice between two responses with its strength, a ranking of several, a score per response, or a mix. Your guidelines set it, and the plan confirms it.
Can raters write a better response?
Yes. When no response is good enough, raters can write a corrected one, saved as the preferred response.
How do you keep preferences consistent?
Raters rate known pairs before live work, and a share of pairs is rated by more than one person so agreement can be checked. Agreement is in your weekly report.
What format do you deliver in?
The one your training pipeline reads, such as JSON lines or CSV, with the fields you set, including the reason for each choice if you want it.
Can the team grow as training ramps up?
Yes. Tell your team lead, and raters are added and trained on the same guidelines. Your weekly report shows the hours and the pairs rated.
Does our data stay confidential?
Yes. We work under NDA, inside your tools, with the access you set, and your data is used only for your work.