Snoowl Audit

Snoowl Audit finds out what your support chatbot gets wrong. You add your help documents and connect your chatbot. Snoowl Audit reads the documents, writes questions from them, asks your chatbot each question, and checks every answer against the documents. Each failure comes with the passage it was judged against, a plain-language reason and a severity.

Snoowl Audit also checks whether your chatbot can be talked into breaking its rules, for example into revealing private information, and reports those results separately from accuracy. See Red team.

Create an account at audit.snoowl.com/signup, or sign in at audit.snoowl.com. See Getting started.

How an audit works

1. Your documents are read. Snoowl Audit breaks your documents into individual statements: facts, rules, exceptions, limits and procedures. Each statement is marked critical, major or minor depending on what a customer stands to lose if a chatbot gets it wrong.
2. Questions are written from the statements. Questions come in several kinds: direct questions, questions with a condition, questions at the edge of a limit, questions about things the documents rule out, questions that need two facts combined, questions built on a false premise, and a few that your documents don’t cover at all. Each question has an expected answer taken from your documents.
3. Your chatbot is asked each question. Every question is sent on its own, with no earlier conversation, the way a customer’s first message would arrive.
4. Every answer is checked against your documents. Each failure comes with the passage it was judged against, a plain-language reason and a severity. Critical failures are double-checked before they’re reported.
5. You get a report. The report has an overall score, five dimensions, every failure with its reason, and a printable PDF.

See How answers are judged.

Projects

A project holds one set of documents and the questions written from them. You can connect several chatbots to the same project, such as your production and staging chatbots, and test them against the same documents and questions. A run can ask up to five of them side by side. A project has these sections:

Section	What you do there
Knowledge	Add your documents and have them read. See Add your documents.
Questions	Generate, write, import and edit questions. See Questions.
Test sets	Group questions into named sets to run on their own.
Chatbot	Connect the chatbots you want to test. See Connect your chatbot.
Runs	Start runs and open their reports. See Run an audit.
Red team	Add and run the red-team tests. See Red team.
Schedules	Run audits daily or weekly with alerts. See Keep running.

Where to start

• New to Snoowl Audit: Getting started walks through your first report and the free demo project.
• Connecting a chatbot: Connect your chatbot covers both connection types and what to do without API access.
• Reading results: Read the report explains every number and label.