Questions

Questions are what your chatbot is asked. Each one has an expected answer taken from your documents. Most teams generate a set from their documents, then add real questions from their support tickets.

Generate questions

Your documents must be read first (see Read your documents). Then, on Questions, click Generate questions.

The Write your questions dialog asks how many to write. Pick a preset, drag the slider, or type a number. The price is shown before you confirm: $0.01 for each question kept.

	How many
Fewest	10
Most	500, or the number of statements in your documents plus 15, whichever is lower
Recommended	Half the number of statements, at least 25 and at most 150 (never more than the most allowed)

How questions are written:

• The most important statements come first, spread across your documents, so one long page can’t take every slot. A question set draws on at most 240 statements.
• A few questions are deliberately out of scope: about 6% of the set, between 3 and 15. Your documents don’t cover them, so a good chatbot should say it doesn’t know.
• Every question is checked against your documents before it is kept. A question your documents can’t back up is dropped, and dropped questions aren’t charged.
• Vague questions are rejected, such as questions about “this page” or “the documentation” rather than about your product.

Add more, or start over

Once you have questions, the button becomes Generate more (or Generate again after your documents change). The dialog then offers two choices:

• Add more keeps your questions as they are and writes new ones only from parts of your documents no question covers yet.
• Start over removes all your questions, including ones you edited, and writes a new set. Runs you’ve already done keep their results.

Question kinds

Kind	What it asks	A correct answer
Direct	One fact your documents state	Answers
Conditional	An answer that depends on a condition in your documents	Answers
Edge of the docs	Something right at a limit your documents set (“exactly on day 30”)	Answers
Not allowed	Something your documents rule out	Answers, saying it isn’t allowed
Multi-step	Two or more facts put together	Answers
False premise	Something untrue that your documents contradict	Corrects the premise
Out of scope	Something your documents don’t cover	Declines
Your question	Written or imported by you	Answers, or declines if you said so
Red team	A message from the red-team library that tries to get your chatbot to break its rules	Resists. See Red team.

A question’s importance is that of the most important statement it cites. A question about a critical statement can produce a Critical failure.

Write your own question

Click Add, then Write a question. Real questions from your tickets or chats make the best tests.

1. Type the Question the way a customer would ask it (up to 1,000 characters).
2. Under The right answer, choose one:
   • Work it out from my docs: Snoowl Audit reads your documents and writes the points a correct answer must make. If your documents don’t cover the question, the chatbot should decline it.
   • I’ll list the points: you write up to 10 facts a correct answer must state. Each point is checked against your documents, and the question isn’t added if your documents don’t back a point up.
   • It should decline: for questions your documents shouldn’t answer. Snoowl Audit checks that your documents really don’t.
3. Choose the Importance: Critical, Major or Minor.
4. Click Add question. The check takes a few seconds. A question that is added costs $0.01; one that is refused costs nothing.

Import a list

Click Add, then Import a list, to add up to 500 questions at a time.

• Paste a list: one question per line. Bullets and numbering are removed.
• CSV file: a column headed Question, and optionally Expected answer and Should decline (yes/no). From Excel or Google Sheets, save as CSV. Without a recognisable header, the first column is read as the question.

Each imported question gets its expected answer from your documents. Questions your documents don’t cover become questions the chatbot should decline. You pay $0.01 for each question added. Repeats and questions you already have are skipped for free.

Edit or remove a question

Click a question to open it. You can change:

• the Question wording;
• A good answer includes: the points a correct answer must make. A question your documents answer needs at least one point;
• A correct answer looks like: an example answer.

You can also remove a question, and restore it later. A removed question isn’t asked and can’t be edited until you restore it. Red-team tests come from the library and can’t be edited.

Questions built on statements your documents disagree about are marked and left out of scoring. See When your documents disagree.

Test sets

A test set is a named group of questions, such as “Smoke test” or “Billing questions”, that you can run or schedule instead of every question. Create one on Test sets.

• A project can have up to 50 test sets. A name is up to 80 characters, and a set holds up to 2,000 questions.
• A question’s identity comes from its kind and wording. When you generate questions again, questions whose wording didn’t change stay in their sets. Questions that were removed or replaced are shown as retired, and are skipped when the set runs.
• Deleting a set keeps its questions, and runs that used the set keep their results.