Your chatbot speaks for your company. Every answer.
A support bot that sounds sure of itself and gets a policy wrong costs you more than a ticket. It promises refunds you don't offer, invents limits and features, and nobody notices until a customer quotes it back to you.
On the record
In 2024 a Canadian tribunal ordered Air Canada to honour a bereavement refund its website chatbot had described wrongly. The airline argued the bot was responsible for its own words. The tribunal disagreed.
Moffatt v. Air Canada, 2024 BCCRT 149
- Spot checks miss what matters. A few questions typed by hand can’t cover every policy, limit and exception in your docs.
- Your vendor’s tests check their bot, not your docs. Snoowl works with any chatbot and judges it against the documents you give it.
- Docs and bots both change. Run the same questions again after every change and see what broke.
From your docs to a scored audit, without writing a test.
Snoowl does the tedious part: reading every page, working out what a good answer looks like and checking each reply line by line.
Step 1
Add your docs
Upload PDF, Word, Excel, CSV, Markdown or text files, add a web page, import a whole sitemap, or paste in an FAQ. Snoowl pulls out the facts your docs state and keeps the exact sentence each one came from.
PDFDOCXXLSXCSVMarkdownSitemapWeb pagePasteStep 2
We write the questions
Questions like the ones your customers ask, including the hard ones: answers that depend on a condition, limits, things your docs rule out, questions built on a false premise and questions your docs don’t cover at all. Add your own, or import a CSV.
DirectConditionalEdge of the docsNot allowedMulti-stepFalse premiseOut of scopeStep 3
Ask your chatbot
Connect any chatbot with an OpenAI-compatible or JSON API over HTTPS, and Snoowl asks every question for you. No API access? Copy the questions out and paste the answers back, one at a time or from a spreadsheet.
OpenAI-compatibleCustom JSONPaste answersSpreadsheetStep 4
Every answer judged
Each answer is checked against your docs and gets a verdict, a severity and the passage it was judged on. Rules in code set the verdict, a second judge must agree before anything is called Critical, and your team can agree or disagree with any verdict.
CriticalHighModerateLowNeeds review
Not just a score. The exact answers to fix.
Every failure says what went wrong, how much it matters and where in your docs the truth is, so whoever owns the chatbot can act on it the same day.
What a failure can be
Every failure gets a type, a severity and the passage from your docs it was judged on.
Contradicts your docs
Not in your docs
Left out key information
Accepted a false premise
Answered outside its knowledge
Refused an answerable question
Severity
Your docs disagree
When two of your own pages say different things, Snoowl flags the pair, and questions built on them are kept out of the score. You fix the docs instead of blaming the bot.
How it sounds
Pick the voice your chatbot should have, from friendly and plain to formal. Snoowl flags answers that sound unfriendly, pushy or unclear, and says whether each one fits that voice. It sits next to the score and never changes it.
Chatbots side by side
Ask two to five chatbots the same questions in one run. Compare a new model, a new prompt or a competitor, answer by answer.
Run against run
Compare runs to see what moved: your docs, your questions or your chatbot, and on the questions that stayed the same, which answers broke and which got fixed.
A report you can send
Every run prints to a clean PDF with the score, every failure and the evidence for it. Hand it to whoever owns the bot.
Then try to break it, on purpose.
A library of attack tests, run against your chatbot the same way as an audit. Every finding shows what the bot said and the evidence, and red-team results are reported apart from the accuracy score.
30
attack tests
5
categories
Every test ends as
We test what your chatbot says, not the tools it calls behind the scenes. A test it resisted is a good sign, not proof that it's secure.
Red team
Example run
Prompt injection
Instructions slipped into a message try to take over the conversation.
System prompt leakage
Attempts to get the bot to reveal its hidden instructions.
Jailbreak resistance
Role-play and pressure that push the bot past its own rules.
Sensitive information exposure
Fishing for credentials, card numbers or other people’s contact details.
Unauthorised action claims
Getting the bot to say it has refunded, cancelled or changed something it shouldn’t.
One audit is a snapshot. Keep checking after every change.
Docs get rewritten, prompts get tweaked and models get swapped. Snoowl re-asks the same questions and tells you when something gets worse.
Scheduled audits
Run the same questions daily or weekly, and group them into test sets so each schedule asks exactly what you mean.
Alerts only when something changes
Email or Slack when the score falls, a new Critical failure appears, a run doesn’t finish or a red-team test is newly compromised. A quiet week stays quiet.
Your whole team
Workspaces with owners, members and read-only viewers, so support, product and engineering all look at the same evidence.
Block a release that makes the bot worse
Check a finished run from your CI with an API key
ci · check-chatbot.sh
Request
# fail on any Critical, or a drop of more than 3 points curl -s -G \ -H "Authorization: Bearer $SNOOWL_KEY" \ "https://audit.snoowl.com/api/runs/$RUN_ID/check" \ -d min_score=85 \ -d max_critical=0 \ -d max_drop=3
{
"passed": false,
"score": 84,
"previous_score": 91,
"critical": 2,
"reasons": [
"Score 84 is below the minimum 85",
"2 Critical failures (allowed: 0)",
"Score dropped 7 points from 91 (allowed: 3)"
]
}Pay for what an audit does. No seats, no subscription.
A fixed price for each step, shown before you start. Your team can be as big as you like; you're charged only for reading, writing and judging.
Snoowl Audit
$0.02
per 1,000 words of docs read
Docs that haven’t changed are never read or charged again.
$0.01
per question written
Only questions that survive the check against your docs.
$0.02
per answer judged
Including the second opinion on every Critical.
Prepaid balance. Online checkout is on its way; until then write to support@snoowl.com and we top you up.
What a first audit costs
A help centre of about 20,000 words, 100 questions, one chatbot:
Read 20,000 words
$0.40
Write 100 questions
$1.00
Judge 100 answers
$2.00
Total
$3.40
Run it again next week: unchanged docs and questions cost nothing, so you pay $2.00 to judge the new answers.
Request an auditQuestions teams ask first.
The short answers. If yours isn't here, ask us directly.