How to set up an Amazon A2I private workforce

Checked against the AWS SageMaker documentation on 2026-10-02. AWS changes console labels from time to time. Where this guide names a page or button, check your console if yours looks different.

Amazon Augmented AI (A2I) sends model predictions to people for review. Since the Amazon Mechanical Turk worker type was removed on 2026-09-30, every review loop needs either a private workforce or a vendor workforce. This guide covers the private route: people you choose, signing in to a review portal for your AWS account.

First, check that this applies to you

As of 2026-10-02, AWS's A2I documentation says A2I is "no longer open to new customers," that existing customers can continue to use it as normal, and that no new features are planned. Ground Truth carries the same notice. This guide is for teams that already run A2I. If you are starting from scratch, compare the effort here with having reviewers work directly in your own tools.

The parts you are setting up

• Private workforce. Everyone who may sign in. AWS allows one private workforce per account per Region, and it is shared between Ground Truth and A2I.
• Work team. A group inside the workforce. Each review workflow sends tasks to a work team. One person can belong to several teams.
• Human review workflow (flow definition). Says when a review starts, which team gets it, which task template reviewers see, and where results go.
• Worker task template. The page reviewers see.
• Human loop. One review of one item.

Before you start

Gather these first. Most stalled set-ups are missing one of them.

• [ ] The AWS Region your A2I workflows run in
• [ ] An IAM role A2I can use, with access to your output bucket
• [ ] An S3 output bucket in the same Region as the workflow
• [ ] Your existing worker task template, or a new one
• [ ] Reviewer work email addresses
• [ ] Your sign-in choice: Amazon Cognito or your own OIDC identity provider
• [ ] IP ranges, if your security policy limits where reviewers can sign in
• [ ] A decision on personal data: which fields reviewers may see, and what is masked

Step 1: choose how reviewers sign in

AWS supports two ways to manage a private workforce: Amazon Cognito or your own OpenID Connect (OIDC) identity provider. When you create a private workforce in the console, SageMaker uses Cognito. To use your own OIDC provider, AWS's documentation says you must create the workforce with the CreateWorkforce API instead.

Cognito is usually simpler for an outside group: invitations go by email, and you can disable people from the console. Choose OIDC when your security team requires sign-in through your own identity provider. We could not confirm from AWS's documentation that an existing workforce can switch between the two, so treat this as a one-time choice until you have checked for your account.

Step 2: create the workforce and a work team

AWS's documentation tells you to create A2I work teams from the Ground Truth section of the SageMaker console, on the page it calls Labeling workforces:

1. Open Labeling workforces and choose Private.
2. Create a private team. Check your console for the exact button label.
3. Invite people by email. The documented limit is 50 addresses per invitation list.
4. Enter an organization name and a contact email that reviewers will see.

If this is your first private team, the same step creates the workforce and a default team containing everyone. Make a separate, named team for each group of reviewers, so you can route work and remove access per team.

Each person gets an invitation to your labeling portal. AWS notes that email addresses are case sensitive, so ask people to sign in with exactly the address you entered.

Step 3: restrict access if you need to

AWS documents a way to limit private workforce access to tasks by IP address. Set it if your policy requires it, then test sign-in from inside and outside the allowed range. Disabling a person stops them receiving work without removing them from the workforce, which is useful between projects.

Step 4: create a new human review workflow

The SageMaker API lists CreateFlowDefinition, DescribeFlowDefinition, ListFlowDefinitions and DeleteFlowDefinition, but no update operation (checked 2026-10-02). To move a workflow from MTurk to a private team, you create a new one:

1. In the Augmented AI section of the console, open Human review workflows and create a new workflow.
2. Name it. AWS's rules: lowercase letters, numbers and hyphens, up to 63 characters, unique in the Region.
3. Set the S3 output location, in the same Region.
4. Choose the IAM role.
5. Choose the task type. For the Amazon Textract and Amazon Rekognition built-in types, set the conditions that start a review, such as a confidence threshold, and an optional random sample. Custom task types have no activation conditions: your code decides when to call StartHumanLoop, and every call creates a review.
6. Choose your worker task template.
7. For workers, choose Private and select your new work team.

Then update your application to use the new flow definition ARN.

Step 5: plan how reviewers find new work

AWS's documentation states that Amazon SNS notifications are supported for Ground Truth but not for A2I. Reviewers will not get an email when A2I tasks arrive. Agree a schedule for portal checks, and watch loop status on your side: A2I sends events to Amazon EventBridge when a human loop is Completed, Failed or Stopped.

Step 6: test with known answers

1. Pick 50 to 200 items with known-correct answers, checked by someone who knows the task.
2. Send them through the new workflow mixed with ordinary items.
3. Read results with DescribeHumanLoop or from the S3 output.
4. Compare reviewer answers with the known ones. Where they differ, decide whether the instructions or the answer key is wrong, and fix it.
5. Move the rest of the volume.

Worked example

Illustrative example. Invented names and numbers, not from a customer.

A team reviews invoice fields that Textract reads with low confidence. Their old workflow, invoice-review-v3, pointed at the MTurk ARN ending in public-crowd/default.

• They create a work team, ap-reviewers, in us-east-1 and invite six reviewers.
• They create invoice-review-v4 with the same task template, output bucket and confidence condition, and the private team.
• They change one configuration value from the v3 ARN to the v4 ARN.
• Results now land under a new prefix. AWS writes A2I output to a path that includes the flow definition name: s3://<bucket>/<flow-definition-name>/YYYY/MM/DD/hh/mm/ss/<human-loop-name>/output.json. Their reporting job read only invoice-review-v3/, so they update it to read both prefixes.
• They send 120 invoices with known answers. Reviewers and the key disagree on 9. Five come from a date-format rule the instructions never stated, so they add it. Four were errors in the key.

Pitfalls

• Missing the output path change. A new workflow name means a new S3 prefix. Downstream jobs go quiet without failing.
• Old content classifier logic. Code written for MTurk may set FreeOfPersonallyIdentifiableInformation. Review it, and decide on personal data explicitly for the new team.
• One team for everything. Separate teams per task keep access reviews and offboarding simple.
• No answer key. Without known answers, you cannot tell a careful team from a fast one.
• Assuming email alerts. A2I does not send SNS notifications to reviewers.

Checklist

• [ ] Confirm your account is an existing A2I customer
• [ ] Choose Cognito or OIDC
• [ ] Create the private workforce and a named work team
• [ ] Invite reviewers and confirm each can sign in
• [ ] Set IP limits if your policy requires them
• [ ] Create a new human review workflow with the private team
• [ ] Update the flow definition ARN in code
• [ ] Update anything that reads the old S3 prefix
• [ ] Run a known-answer batch and fix the instructions
• [ ] Move full volume

Sources

AWS SageMaker Developer Guide pages checked on 2026-10-02: Private workforce (sms-workforce-private.html), Create a Private Workforce (sms-workforce-create-private-console.html), Manage a Workforce (sms-workforce-management-private-console.html), Create a Human Review Workflow (a2i-create-flow-definition.html), Monitor and Manage Your Human Loop (a2i-monitor-humanloop-results.html), and the SageMaker API operations list. All under https://docs.aws.amazon.com/sagemaker/.

If you want a trained team in that work team, with a team lead and QA, see our Ground Truth and A2I private workforce page. For the wider move, including Ground Truth labeling jobs, read the MTurk migration guide.