Glossary

Plain definitions of the terms you'll meet in human review, annotation and data work, and how a Deepen program uses them.

Quality

How the accuracy of human work is set, measured and reported.

Annotation guidelinesAlso called labeling guidelines, instructions, rubric

The written rules for a task: what each label or decision means, how to handle edge cases, and examples of right and wrong answers. Consistent work starts with clear guidelines.

In a Deepen programIf you have finished examples but no guidelines, the team lead drafts them with you during the pilot.

Writing annotation guidelines
Gold setAlso called golden set, gold standard, ground truth set, gold items

A set of items with known-correct answers, agreed in advance. Gold items are mixed into live work, so each person's accuracy is measured rather than estimated.

In a Deepen programEvery program builds one during the pilot from your guidelines and examples, and you keep it.

How it works
First-pass accuracy

The share of items done correctly the first time, before any correction or rework. It shows how well a team knows the rules, not how much checking happened afterwards.

In a Deepen programQA samples finished work against known answers, and the weekly report shows what it found.

QA samplingAlso called quality assurance, spot checks, audit sample

Re-checking a sample of finished items against the guidelines, instead of checking every item twice. The sample's results show the quality of the whole batch.

In a Deepen programThe weekly report shows items done, items sampled and the acceptance rate on the sample.

Acceptance rateAlso called pass rate

The share of reviewed items that pass a quality check without changes. Tracked week by week, it shows whether quality is holding as volume grows.

Inter-annotator agreementAlso called IAA, consensus, agreement rate

How often two or more people give the same answer to the same item. Low agreement usually points to unclear guidelines rather than careless work.

Escalation

Sending an item that the guidelines don't clearly answer to someone who can decide, instead of guessing. Each decision is added to the guidelines so the case is settled for everyone.

In a Deepen programUnclear items go to your team lead. Questions only you can answer appear in the weekly report.

Kinds of work

The human work behind AI models, agents and data.

Human-in-the-loopAlso called HITL

A process where people review, correct or approve some of a system's outputs, often the uncertain ones, before they are used. The corrections can also become training data.

Human review
Human review

People checking AI outputs or data items against clear rules and approving, correcting or flagging each one. It catches what automated checks miss and gives you a measured error rate.

Human review of LLM outputs: a playbook
AI evaluationAlso called model evaluation, evals

Measuring how well a model or agent performs on tasks that matter to you, often by having trained people score its outputs against a rubric. It shows whether a change made things better or worse.

AI evaluation
Preference dataAlso called pairwise comparisons, RLHF data, ranking data

Judgements from people about which of two or more model responses is better, and why. It is used to train models toward the answers people prefer.

Red teamingAlso called adversarial testing

Deliberately trying to make a model or agent fail: unsafe answers, leaked data, broken rules. The failures found become test cases and training data.

Agent trajectoriesAlso called demonstrations, agent traces

Step-by-step records of a task being done: each action, tool call and result. People write them as examples to train agents, or review an agent's own trajectories to find where it went wrong.

Agent training data
Data operationsAlso called data ops

Recurring work that keeps data usable: cleaning records, enriching catalogs, matching duplicates, checking documents. It runs on clear rules, like review work, and can be measured the same way.

Data operations

Teams and pricing

Who does the work, and how it's bought.

Managed teamAlso called managed workforce, dedicated team

A team supplied and run by a vendor, who trains the people, checks the work and answers for the result. You set the rules and see the reports.

In a Deepen programEvery Deepen team is in-house, trained and managed by Deepen AI.

CrowdsourcingAlso called crowd work, microtasks

Splitting work into small tasks that a large, open pool of registered workers can pick up. Who does each task changes from one item to the next.

In a Deepen programDeepen programs use the same in-house people throughout, so the team learns your rules and keeps them.

Team lead

The person who runs the team day to day: trains people, answers their questions, decides escalated items and reports to you.

In a Deepen programEvery program has a named team lead from the pilot onwards, who sends the weekly report.

Person-hour

One hour of one person's work. Pricing by the person-hour makes cost easy to plan: the price follows the people and hours in your plan.

In a Deepen program$5 per person-hour for human review and data operations, with the team lead, QA sampling and the weekly report included.

Pricing
PilotAlso called trial, proof of concept

A short, real run of the work on your own items, to measure quality, speed and cost before committing to full volume.

In a Deepen programTwo weeks on your real items, with two weekly reports. Then you decide.

AWS workflows

The Amazon services many teams use to send work to people.

Amazon Mechanical TurkAlso called MTurk

Amazon's crowdsourcing marketplace, where requesters posted small tasks (HITs) for registered workers. It closed permanently on 30 September 2026.

Moving from Mechanical Turk
SageMaker Ground TruthAlso called Ground Truth

The AWS service for running labeling jobs on data in S3, with task templates, a labeling portal and output written back to S3. The work is done by a workforce you choose.

Ground Truth and A2I migration guide
Amazon A2IAlso called Amazon Augmented AI, human review workflow

The AWS service that sends some model predictions to people for review, for example when the model's confidence is low, and returns their answers to your application.

Moving from Amazon A2I
Private workforce

In Ground Truth and A2I, a workforce of people you invite, such as your own staff or a vendor's team, instead of a public crowd.

In a Deepen programDeepen's people can join your private workforce, so your templates, data and code stay as they are.

Integrations
Work team

A named group inside a private workforce. Labeling jobs and review workflows are pointed at a work team, which decides who sees the tasks.

Sensor data

Terms from autonomous vehicles, robotics and physical AI.

LiDAR

A sensor that measures distance with laser pulses and builds a 3D picture of its surroundings. Vehicles and robots combine it with cameras and radar.

Point cloud

The set of 3D points a LiDAR or depth sensor produces in one scan. Annotators draw boxes or label points in it to mark objects.

Sensor fusionAlso called multi-sensor data

Combining data from several sensors, such as cameras, LiDAR and radar, into one view of a scene. Fused data can only be labeled accurately if the sensors are calibrated to each other.

Sensor calibrationAlso called extrinsic calibration, intrinsic calibration

Measuring exactly where each sensor sits and how it sees, so their data lines up. Small calibration errors become large errors in labels and models.

Robotics and physical AI
Targetless calibration

Calibrating sensors from ordinary driving or scene data, without placing special targets such as checkerboards in front of them.

In a Deepen programInvented and patented by Deepen AI.

3D bounding boxAlso called cuboid

A box drawn around an object in a point cloud, with its position, size and heading. It is the most common label for vehicles and pedestrians in 3D data.

ASAM OpenLABELAlso called OpenLABEL

An open standard file format for labels on multi-sensor data, so annotations can move between tools and teams without conversion.

In a Deepen programDeepen AI is one of the standard's authors, and its tools export to it.

Security

The certifications buyers ask about in a security review.

SOC 2 Type IIAlso called SOC 2

An independent audit of a company's security controls over a period of months. Type II shows the controls worked in practice, not only on paper.

In a Deepen programDeepen AI holds SOC 2 Type II; the report is shared under NDA.

Security
ISO 27001Also called ISO/IEC 27001

The international standard for an information security management system: how a company finds, controls and reviews its security risks.

In a Deepen programDeepen AI is certified to the 2022 standard.

ISO 9001

The international standard for quality management: documented processes, measured results and continual improvement.

In a Deepen programDeepen AI is certified to the 2015 standard.

TISAX

The information security assessment used across the automotive industry. Carmakers and their suppliers often require it before sharing data.

In a Deepen programDeepen AI holds TISAX.

GDPRAlso called General Data Protection Regulation

The EU law on personal data: what may be collected, why, for how long, and who may process it.

In a Deepen programDeepen AI handles personal data in line with GDPR, under a Data Protection Addendum.

See it on your own work

A plan within 2 working days. Two-week pilot on your real items.