Glossary
Plain definitions of the terms you'll meet in human review, annotation and data work, and how a Deepen program uses them.
Quality
How the accuracy of human work is set, measured and reported.
- Annotation guidelinesAlso called labeling guidelines, instructions, rubric
The written rules for a task: what each label or decision means, how to handle edge cases, and examples of right and wrong answers. Consistent work starts with clear guidelines.
In a Deepen programIf you have finished examples but no guidelines, the team lead drafts them with you during the pilot.
Writing annotation guidelines- Gold setAlso called golden set, gold standard, ground truth set, gold items
A set of items with known-correct answers, agreed in advance. Gold items are mixed into live work, so each person's accuracy is measured rather than estimated.
In a Deepen programEvery program builds one during the pilot from your guidelines and examples, and you keep it.
How it works- First-pass accuracy
The share of items done correctly the first time, before any correction or rework. It shows how well a team knows the rules, not how much checking happened afterwards.
In a Deepen programQA samples finished work against known answers, and the weekly report shows what it found.
- QA samplingAlso called quality assurance, spot checks, audit sample
Re-checking a sample of finished items against the guidelines, instead of checking every item twice. The sample's results show the quality of the whole batch.
In a Deepen programThe weekly report shows items done, items sampled and the acceptance rate on the sample.
- Acceptance rateAlso called pass rate
The share of reviewed items that pass a quality check without changes. Tracked week by week, it shows whether quality is holding as volume grows.
- Inter-annotator agreementAlso called IAA, consensus, agreement rate
How often two or more people give the same answer to the same item. Low agreement usually points to unclear guidelines rather than careless work.
- Escalation
Sending an item that the guidelines don't clearly answer to someone who can decide, instead of guessing. Each decision is added to the guidelines so the case is settled for everyone.
In a Deepen programUnclear items go to your team lead. Questions only you can answer appear in the weekly report.
Kinds of work
The human work behind AI models, agents and data.
- Human-in-the-loopAlso called HITL
A process where people review, correct or approve some of a system's outputs, often the uncertain ones, before they are used. The corrections can also become training data.
Human review- Human review
People checking AI outputs or data items against clear rules and approving, correcting or flagging each one. It catches what automated checks miss and gives you a measured error rate.
Human review of LLM outputs: a playbook- AI evaluationAlso called model evaluation, evals
Measuring how well a model or agent performs on tasks that matter to you, often by having trained people score its outputs against a rubric. It shows whether a change made things better or worse.
AI evaluation- Preference dataAlso called pairwise comparisons, RLHF data, ranking data
Judgements from people about which of two or more model responses is better, and why. It is used to train models toward the answers people prefer.
- Red teamingAlso called adversarial testing
Deliberately trying to make a model or agent fail: unsafe answers, leaked data, broken rules. The failures found become test cases and training data.
- Agent trajectoriesAlso called demonstrations, agent traces
Step-by-step records of a task being done: each action, tool call and result. People write them as examples to train agents, or review an agent's own trajectories to find where it went wrong.
Agent training data- Data operationsAlso called data ops
Recurring work that keeps data usable: cleaning records, enriching catalogs, matching duplicates, checking documents. It runs on clear rules, like review work, and can be measured the same way.
Data operations
Teams and pricing
Who does the work, and how it's bought.
- Managed teamAlso called managed workforce, dedicated team
A team supplied and run by a vendor, who trains the people, checks the work and answers for the result. You set the rules and see the reports.
In a Deepen programEvery Deepen team is in-house, trained and managed by Deepen AI.
- CrowdsourcingAlso called crowd work, microtasks
Splitting work into small tasks that a large, open pool of registered workers can pick up. Who does each task changes from one item to the next.
In a Deepen programDeepen programs use the same in-house people throughout, so the team learns your rules and keeps them.
- Team lead
The person who runs the team day to day: trains people, answers their questions, decides escalated items and reports to you.
In a Deepen programEvery program has a named team lead from the pilot onwards, who sends the weekly report.
- Person-hour
One hour of one person's work. Pricing by the person-hour makes cost easy to plan: the price follows the people and hours in your plan.
In a Deepen program$5 per person-hour for human review and data operations, with the team lead, QA sampling and the weekly report included.
Pricing- PilotAlso called trial, proof of concept
A short, real run of the work on your own items, to measure quality, speed and cost before committing to full volume.
In a Deepen programTwo weeks on your real items, with two weekly reports. Then you decide.
AWS workflows
The Amazon services many teams use to send work to people.
- Amazon Mechanical TurkAlso called MTurk
Amazon's crowdsourcing marketplace, where requesters posted small tasks (HITs) for registered workers. It closed permanently on 30 September 2026.
Moving from Mechanical Turk- SageMaker Ground TruthAlso called Ground Truth
The AWS service for running labeling jobs on data in S3, with task templates, a labeling portal and output written back to S3. The work is done by a workforce you choose.
Ground Truth and A2I migration guide- Amazon A2IAlso called Amazon Augmented AI, human review workflow
The AWS service that sends some model predictions to people for review, for example when the model's confidence is low, and returns their answers to your application.
Moving from Amazon A2I- Private workforce
In Ground Truth and A2I, a workforce of people you invite, such as your own staff or a vendor's team, instead of a public crowd.
In a Deepen programDeepen's people can join your private workforce, so your templates, data and code stay as they are.
Integrations- Work team
A named group inside a private workforce. Labeling jobs and review workflows are pointed at a work team, which decides who sees the tasks.
Sensor data
Terms from autonomous vehicles, robotics and physical AI.
- LiDAR
A sensor that measures distance with laser pulses and builds a 3D picture of its surroundings. Vehicles and robots combine it with cameras and radar.
- Point cloud
The set of 3D points a LiDAR or depth sensor produces in one scan. Annotators draw boxes or label points in it to mark objects.
- Sensor fusionAlso called multi-sensor data
Combining data from several sensors, such as cameras, LiDAR and radar, into one view of a scene. Fused data can only be labeled accurately if the sensors are calibrated to each other.
- Sensor calibrationAlso called extrinsic calibration, intrinsic calibration
Measuring exactly where each sensor sits and how it sees, so their data lines up. Small calibration errors become large errors in labels and models.
Robotics and physical AI- Targetless calibration
Calibrating sensors from ordinary driving or scene data, without placing special targets such as checkerboards in front of them.
In a Deepen programInvented and patented by Deepen AI.
- 3D bounding boxAlso called cuboid
A box drawn around an object in a point cloud, with its position, size and heading. It is the most common label for vehicles and pedestrians in 3D data.
- ASAM OpenLABELAlso called OpenLABEL
An open standard file format for labels on multi-sensor data, so annotations can move between tools and teams without conversion.
In a Deepen programDeepen AI is one of the standard's authors, and its tools export to it.
Security
The certifications buyers ask about in a security review.
- SOC 2 Type IIAlso called SOC 2
An independent audit of a company's security controls over a period of months. Type II shows the controls worked in practice, not only on paper.
In a Deepen programDeepen AI holds SOC 2 Type II; the report is shared under NDA.
Security- ISO 27001Also called ISO/IEC 27001
The international standard for an information security management system: how a company finds, controls and reviews its security risks.
In a Deepen programDeepen AI is certified to the 2022 standard.
- ISO 9001
The international standard for quality management: documented processes, measured results and continual improvement.
In a Deepen programDeepen AI is certified to the 2015 standard.
- TISAX
The information security assessment used across the automotive industry. Carmakers and their suppliers often require it before sharing data.
In a Deepen programDeepen AI holds TISAX.
- GDPRAlso called General Data Protection Regulation
The EU law on personal data: what may be collected, why, for how long, and who may process it.
In a Deepen programDeepen AI handles personal data in line with GDPR, under a Data Protection Addendum.