Configuring evaluation, rubrics and reviewers in AI roleplay

Introduction

The Evaluation tab defines how the AI evaluates learner performance in an AI roleplay. By default, every roleplay uses a qualitative rubric based on general communication criteria. You can replace it with a custom rubric that reflects your own standards.

This article explains how to choose a rubric type, configure a qualitative or quantitative rubric, generate criteria with AI, replace a rubric, assign reviewers and write effective criteria. It also describes how the AI-generated report is structured for each rubric type. For the other tabs, see Creating and managing AI roleplays.

Requirements and limitations

• The AI evaluates the conversation transcript only. It assesses what was said or written, not how it sounded or looked.
• A custom rubric has up to 5 criteria.
• A rubric becomes active when you publish the roleplay.
• The configuration panel is available in English only.
• Reviewers can see submissions only if the AI roleplay page is added to their user menu. See the Assigning reviewers section.

Choosing a rubric type

Use a qualitative rubric when the learner needs to know what to work on. Use a quantitative rubric with a pass score when the result decides something, such as certification readiness or access to the next step.

Configuring the rubric

First, open the roleplay and go to the Evaluation tab. In the rubric section, the rubric shows Default until you customize it. Click Create custom rubric, and then choose Qualitative rubric or Quantitative rubric. Define your criteria, or click Generate criteria to have AI draft them. Finally, click Save changes.

The rubric is saved as part of the roleplay and becomes active when you publish the roleplay.

Qualitative rubric

Define up to 5 criteria. Each criterion has four fields:

• Criterion name: a short, specific name for the skill or behavior, such as "Active listening" or "Objection handling".
• Description: one or two sentences about what the criterion covers. Write it as if you are briefing a new reviewer.
• What Good looks like: concrete, observable behaviors that indicate strong performance. You can include example phrases, actions or required elements.
• What Developing looks like: concrete, observable behaviors that indicate areas for improvement.

To add a criterion, click Add criterion at the bottom of the list. To delete one, click the ellipsis icon on the criterion card and select Delete. At least one criterion must remain.

Quantitative rubric

Define up to 5 criteria. Each criterion is scored from 1 to 5. For each criterion, provide a criterion name, a description and the following three levels:

• 5 - High: how the criterion is demonstrated effectively.
• 3 - Medium: how the criterion is partially met.
• 1 - Low: how the criterion is not met.

The Scoring section has two optional settings:

• Use weights: each criterion shows a weight field. Weights determine how much each criterion contributes to the overall score. Without weights, all criteria count equally.
• Set pass score: select the minimum overall score, from 1 to 5, that a learner must reach to pass. The learner's report then shows a Pass or Fail label next to the overall score.

Generating criteria with AI

Both rubric types support AI-generated criteria.

First, in the rubric, click Generate criteria. Next, in the text field, describe what you want the rubric to evaluate, for example "Evaluate whether the learner handles pricing objections by acknowledging the concern before presenting value." Optionally, upload up to 3 reference documents, such as a sales playbook, coaching guide or competency framework. Each document can be a PDF or DOCX file of up to 100 MB. Images in the documents are not processed. Then click Continue. If the AI needs more context, it asks one to three clarifying questions. Select a suggested answer or type your own, and then click Generate. Finally, review and edit the generated criteria, and click Save changes.

Please note: The roleplay description, goal, avatar role and learner role are used as context for generation. Fill them in before you generate criteria. Generated criteria replace any existing criteria, and a confirmation prompt appears before existing criteria are overwritten. Uploaded documents are used only to generate the criteria. They are not used later to evaluate learners.

If the AI cannot generate criteria from your input, an error message explains why. Revise your description and try again, or write criteria manually. For tips on writing the description, see the section on writing prompts for AI-generated criteria below.

Replacing a rubric

First, in the rubric panel, click Replace rubric. Next, in the window, select the other custom rubric type (Qualitative or Quantitative) or Reset to default to use the default evaluation criteria. Then click Replace to confirm.

Please note: Replacing a rubric deletes all existing criteria. This action cannot be undone.

Assigning reviewers

Reviewers are users who review learner submissions and validate the AI-generated evaluation. By default, no reviewers are assigned.

In the Settings tab, go to Submission and reviewer and select Assign reviewer. Then choose the type of reviewer: Group, Branch, User or Manager, and select the items. You can select more than one item, up to 20.

Please note: When at least one reviewer is assigned, submission becomes required. Learners must complete and submit the roleplay so that it can be reviewed, and submission is required to complete the training.

Before reviewers can see submissions, you must add the AI roleplay built-in page to the user menu that your reviewers use. To do this, open Customization (shapes icon) in the navigation menu, click Menus, and add the AI roleplay page to the menu. The page is visible only to users who are reviewers for at least one roleplay. Learn more about making pages and menus visible to users. If the corresponding notification is active, reviewers are notified when a new submission is available.

For the reviewer experience, see Reviewing AI roleplay submissions.

The AI-generated report

At the end of each simulation, the AI analyzes the full transcript, whether spoken or written, and generates a structured report. The structure depends on the rubric type. For what learners see and how they submit a report, see Practicing with AI roleplay and reading your report.

The default rubric report includes the following:

• Overall feedback with strengths and areas for improvement.
• Tone of voice, which is the communication style inferred from the transcript: clarity, confidence and energy.
• Key quotes that highlight strong and weak moments.

The qualitative rubric report shows the following for each criterion:

• The outcome: Good or Developing.
• The observable behaviors behind the feedback.
• Direct quotes from the conversation as evidence.

The quantitative rubric report includes the following:

• A score for each criterion, such as 4 out of 5, with a rationale based on the conversation.
• An overall score: the weighted average of the criterion scores, or a simple average if weights are not used.
• A Pass or Fail label if a pass score is set.
• An explanation of how the overall score was calculated, including each criterion's weight.

If the conversation does not contain enough evidence for a criterion, the AI assigns the minimum score (1) and flags the rationale as insufficient evidence.

Writing effective rubric criteria

The quality of your criteria determines the quality of the feedback. Precise criteria produce specific, actionable feedback. Vague criteria produce generic feedback. This section summarizes the main guidance. For the full set of examples, see Best practices for custom rubrics in AI roleplay.

The transcript-only rule

The AI reads what was said, not how it was said. Before you save a criterion, ask: "Could I assess this by reading the transcript alone?" If yes, the criterion works. If you would need to hear audio or watch a recording, it does not.

Good example: "Uses ownership language and states a clear next step."

Poor example: "Sounds warm and confident." (Not observable in a transcript.)

Use 3 to 5 focused criteria

Use as few criteria as you need to cover the skills that matter, and no more than 5. Fewer, focused criteria work better for these reasons:

• Each criterion gets clearer evidence in a short conversation. With many criteria, the conversation may not contain enough evidence for each one.
• Overlapping criteria score the same behavior twice, which inflates or distorts the overall result.
• The learner can act on three clear points. A long list is hard to prioritize.

Merge criteria that overlap. For example, "Listening", "Empathy" and "Acknowledging concerns" often assess the same behaviors. Combine them into one criterion, "Active listening", described as "paraphrases, asks clarifying questions and responds directly to stated concerns". If you have a long framework, break it into 2 or 3 criteria that each cover a meaningful part of it, instead of one criterion per step.

Describe observable behaviors

Good example: "Names the observable behavior, describes its impact and proposes a specific change."

Poor example: "Gives good feedback." (No behavior to look for.)

In the performance levels, describe what is present and what is missing. "Somewhat good" does not help the AI calibrate.

Regulated or required language

If learners must use, or avoid, specific language, make that an observable criterion.

Good example: Criterion "Required disclosure". What Good looks like: "States that the call is recorded and that advice is not a guarantee before discussing outcomes." What Developing looks like: "Discusses outcomes without the disclosure, or promises a result."

Poor example: "Complies with regulations." (Does not say what compliant language is.)

Also include the relevant domain facts in the roleplay's Additional context, so the avatar reacts correctly. Learn more in Best practices for writing AI roleplays.

Choose the rubric type for the purpose

• Qualitative for coaching and practice: the learner gets feedback on specific behaviors, with quotes.
• Quantitative with a pass score for certification or gating: the learner gets a score and a Pass or Fail result. Set the pass score high enough to reflect the standard you require, and use weights if some criteria matter more than others.
• If you use a pass score to gate progress, assign reviewers so that a person can validate the result.

Writing prompts for AI-generated criteria

• Describe the behavior, not only the topic. Good example: "Evaluate whether the learner handles pricing objections by acknowledging the concern before explaining the value." Poor example: "Sales skills".
• Prioritize. The AI generates up to 5 criteria. If your request or documents cover more skills, name the 5 that matter most. Good example: "Generate criteria based on our sales playbook. Focus on needs discovery, objection handling, value communication, active listening and closing."
• Request only supported criteria. Avoid vocal delivery or body language.
• Tell the AI how to use a document. For example: "Use our customer service standards as the basis for the criteria." Keep documents focused. For long documents, summarize the relevant points in the text field instead.

Related articles

• Creating and managing AI roleplays
• Best practices for writing AI roleplays
• Practicing with AI roleplay and reading your report
• Reviewing AI roleplay submissions
• Best practices for custom rubrics in AI roleplay