Run an audit

A run asks your chatbot a set of questions and judges every answer. On a project’s Runs page, click Start run.

The Start a run dialog

How should we get the answers?

• Automatic: Snoowl Audit asks your connected chatbot each question and judges the answers.
• Manual: you ask your chatbot yourself and paste its answers, one at a time or from a spreadsheet. See Paste the answers.

Chatbots (Automatic only). Pick one chatbot, or up to five. With more than one, each chatbot gets the same questions at the same time as its own run, and you land on a comparison of them when they finish. This is how you test two environments or two prompts side by side.

If a chatbot’s last connection test failed, the dialog warns you and offers Test again. A chatbot that fails every one of the first few questions stops its run.

Questions

Choice	What is asked
All active questions	Every question that isn’t removed
A test set	The questions in that set that are still active
Choose questions…	The questions you tick

A run asks at most 5,000 questions. Red-team tests run from the Red team page instead (see Red team).

Price. The dialog shows the most the run can cost: $0.02 per answer judged, times the number of questions and chatbots. Answers that error aren’t charged. The amount is set aside from your balance when the run starts, and you’re charged only for answers actually judged. The rest goes back. A manual run charges nothing until you submit answers for judging.

Click Start run (or Start 3 runs for three chatbots). For a manual run, click Open answer sheet.

Note

An account runs one audit at a time. While a run is going, starting another one, including a re-judge, is refused with “Another audit is already running on this account”. Scheduled runs wait and try again 15 minutes later.

While a run is going

Questions are sent six at a time. Each answer is saved before it is judged, so if something interrupts the run, it picks up without asking your chatbot the same question twice. The run page shows progress as answers come in.

Every question is sent on its own, with no earlier conversation.

To stop a run, click Cancel run. The run stops within a few questions, and a cancelled run has no report.

A run that can’t continue is marked This run failed, with the reason. The most common reason is a chatbot that failed every question; fix the connection on Chatbot and start a new run.

When it finishes

The run’s report opens on the run page. See Read the report.

If a run you started takes more than two minutes, you get an email when it finishes. Turn this off in Account settings under Email: When a run I started finishes.

Add a Note on this run from the run page, such as “switched to a new prompt”. Notes show up when you compare runs.

From a finished run you can also re-ask the failed questions or re-judge the answers. See Keep running.