Read the report
5 min read
When a run finishes, its report opens on the run page. The same page is where you review results, re-run and print.
The score card
Section titled “The score card”| Item | What it shows |
|---|---|
| Overall score | 0 to 100: the average of the dimension scores below, over the answers that were scored |
| Dimensions | Five scores, one for each thing a good answer gets right. See The five dimensions. |
| Failures | Failed answers, by severity |
| Response time | How long your chatbot typically took to answer (the median: half the answers were faster). The tooltip adds the average and the slowest. |
| Answered | How many questions your chatbot answered, and how many errored |
| Not scored | Answers waiting on a person (Needs review) and answers left out of the score (Excluded) |
The score’s colour shows how it reads: 90 and above is teal, 75 to 89 yellow, 50 to 74 orange, and below 50 red. The PDF puts the same bands in words: Strong, Good, with gaps to fix, Needs work and Not ready for customers.
The five dimensions
Section titled “The five dimensions”Each dimension looks at one thing a good answer gets right, and scores it from 0 to 100: the percentage of answers that got it right. Each one only counts the questions it applies to.
- Accuracy. Does your chatbot stick to what your documents say? This score drops each time an answer says something your documents say differently.
- Grounded in your docs. Does your chatbot avoid making things up? This score drops when an answer states a specific fact, such as a price or a time limit, that your documents don’t mention.
- Completeness. Does your chatbot give the whole answer? This score drops when an answer leaves out something important, or when your chatbot says it can’t help with a question your documents do answer.
- Knows its limits. Does your chatbot say so when it doesn’t know? This score only counts questions your documents don’t cover, and drops when your chatbot answers them anyway instead of declining.
- Corrects false premises. Does your chatbot put a customer right? This score only counts questions that assume something wrong, such as asking how to turn on a feature that doesn’t exist, and drops when your chatbot plays along.
If a run has no questions for a dimension, for example no questions that assume something wrong, that dimension isn’t shown, rather than showing a perfect score.
Notes on the score card
Section titled “Notes on the score card”- “N answers use facts this project’s docs don’t cover.” Shown when two or more answers failed as Not in your docs. Your chatbot may be answering from documents you haven’t added. Add them on Knowledge, then re-judge.
- “Judged on the passages most relevant to each answer.” Shown when your documents were too large to read whole for every answer, so each answer was judged against the parts most relevant to it. If a result says something isn’t in your documents when it is, mark it Disagree in the review.
Failure types
Section titled “Failure types”| Label | Means |
|---|---|
| Contradicts your docs | The chatbot said something your documents say differently. |
| Not in your docs | The chatbot stated specific facts your documents don’t mention. They may be true, but your documents can’t back them up. |
| Answered outside its knowledge | Your documents don’t cover the question, but the chatbot answered anyway instead of saying it doesn’t know. |
| Accepted a false premise | The question assumed something your documents contradict, and the chatbot didn’t correct it. |
| Left out key information | The answer missed one or more things a good answer includes. |
| Refused an answerable question | The chatbot said it couldn’t help, but your documents answer the question. |
Severity
Section titled “Severity”| Severity | Means |
|---|---|
| Critical | A customer could be badly misled |
| High | A customer could act on this |
| Moderate | Worth fixing |
| Low | Minor |
How severity is set: see How serious it is.
The answers table
Section titled “The answers table”Every question in the run is listed under Answers. Filter by Failed, Passed, Needs review, Excluded or All. The failed list starts with the most serious. The columns are Question, Result, Severity, Response time and Voice.
Click an answer to open its details:
- Question, with its kind, what a correct answer does, and its importance.
- Expected answer beside the Chatbot’s answer, so you can compare them directly.
- Reasons, in tabs:
- What’s wrong: claims that contradict your documents, with the passage from your documents they contradict, and claims that aren’t in your documents;
- What’s right: claims your documents support, with the supporting passage;
- Not counted: side details that aren’t in your documents, and citations or remarks that are never scored.
- How it sounds: see below.
Review results
Section titled “Review results”Click Review verdicts to go through results one at a time. For each one, choose Agree or Disagree with the judge, and add an optional note about what the judge missed or why it’s right.
Keyboard shortcuts: A agree, D disagree, J or → next, K or ← back.
Reviewers, including people with the Viewer role, can label results. Your team’s labels are kept with the run.
How it sounds
Section titled “How it sounds”Every answer also gets a tone reading, separate from whether the answer is right. Tone is descriptive only and never changes a result or the score.
- When the chatbot has a voice set (see Connect your chatbot), each answer is marked On voice, Partly on voice or Off voice.
- An answer that sounds off is labelled with one of three categories:
| Category | Covers |
|---|---|
| Unfriendly | Cold, curt, dismissive, condescending or blaming |
| Pushy | Sales pressure nobody asked for |
| Unclear | Jargon, legal wording, filler, scripted lines, repeated apologies |
The How it sounds card on the run also shows the typical answer length, the reading grade, and how many answers contain lines such as “As an AI…”. Reading tone again for a finished run is free.
Save the report as a PDF
Section titled “Save the report as a PDF”On a completed run, open More and choose Print / Save PDF, then save from your browser’s print dialog. The file is named after the chatbot and the date the run finished, in the form Chatbot name — chatbot audit — date.
The first page is a standalone summary for readers who won’t go further. The sections that follow are:
- Failures at a glance
- What it got wrong: every failure in full
- Needs a person to check
- What it got right
- Left out of the score
- How it sounds
- Red team (chatbot security), when the run has red-team results
Reports are kept in the app and printed to PDF. There are no public share links: anyone who reads a report in the app needs a login in your workspace.