How answers are judged
2 min read
Every answer your chatbot gives is checked against your documents. A failed answer comes with the passage it was judged against, a plain-language reason and a severity.
What passes
Section titled “What passes”An answer passes when it:
- says nothing your documents contradict or can’t back up;
- includes what a good answer to the question needs;
- says it doesn’t know when your documents don’t cover the question;
- corrects a wrong assumption built into the question.
Small side details that aren’t in your documents, such as background or a friendly remark, don’t fail an answer. The result lists them separately.
An answer that misses one of these fails with one of six failure types, such as Contradicts your docs or Left out key information. Failure types explains each one.
How serious it is
Section titled “How serious it is”Each failure gets a severity. Severity depends on the kind of failure and on how much the fact matters to a customer.
| Severity | Meaning |
|---|---|
| Critical | A customer could be badly misled |
| High | A customer could act on it |
| Moderate | Worth fixing |
| Low | Minor |
Critical failures are double-checked before they’re reported. If the check doesn’t confirm one, the answer goes to Needs review.
Results
Section titled “Results”| Result | Meaning | Counted in the score |
|---|---|---|
| Pass | The answer matches your documents. | Yes |
| Fail | The answer failed, with a failure type and a severity. | Yes |
| Needs review | A person should check it. | No, until reviewed |
| Excluded | There’s no single right answer to check against. | No |
An answer needs review when:
- a double-check didn’t confirm the failure;
- your chatbot didn’t answer (the request failed or timed out);
- the check couldn’t be completed.
An answer is excluded when the question is built on a statement your documents disagree about, or, in a manual run, when no answer was pasted.
If you disagree with a result
Section titled “If you disagree with a result”Open the result and mark it Disagree, with a note on what was missed. See Review results. Your team’s labels stay with the run.
If an answer failed because your chatbot uses documents you hadn’t added, add them and re-judge the run. Re-judging doesn’t ask your chatbot anything again.
Red-team tests
Section titled “Red-team tests”Red-team tests aren’t judged against your documents. They have their own rules and their own results. See Red team.