Keep running

5 min read

An audit is most useful when you run it again after every change to your chatbot or your documents. Snoowl Audit has four ways to do that: quick re-runs, comparisons, schedules and a deploy check. All of them run on demand or on a timetable; none of them watches your chatbot’s live conversations.

Re-ask the failed questions

On a finished run, click Re-ask N failed. This starts a new run of the same chatbot with only the questions that failed and are still active. It’s the quick check after a fix, and it opens a comparison with the original run.

Re-asking needs a connected chatbot, so it isn’t available for runs with pasted answers.

Re-judge

Re-judging judges a finished run’s answers again without asking your chatbot anything. Use it after you add documents your chatbot was answering from.

When the judging rules have changed since the run, the run page offers Re-judge free.
Otherwise, choose Re-judge from the More menu. It costs $0.02 per answer, like a run.

Compare runs

On a run, open More and choose Compare with previous. Or tick between 2 and 5 finished runs on Runs and click Compare. Comparing is free.

A comparison has three parts:

What changed between the runs: your documents (added, updated or removed), the questions, and the chatbot’s connection, including a model change in an OpenAI-compatible request. The Note on this run shows here too.
Metrics, as rates, so runs of different sizes can be compared.
Same questions: answer-by-answer changes, over questions whose expected answer is provably the same in every run.

Each shared question is labelled:

Label	Means
Broke	Passed before, fails now
Got worse	Still fails, and more seriously
Fixed	Failed before, passes now
Less serious	Still fails, but less seriously
Flip-flops	Passed and failed in different runs
Still failing	Failed both times, just as seriously
Still passing	Passed every time
Not scored	Needed review or was left out in a run

The comparison also says how far the runs can be compared:

Label	Means
Like for like	Same documents, same questions, same judging. The differences are real.
Partly comparable	Something besides your chatbot changed. The reasons are listed.
Not comparable	The runs can be shown side by side, but their differences don’t mean much.

The history chart

The Runs page charts your latest 200 finished runs, one line per chatbot. Switch between Score, Failures, Critical or high, Not in your docs, Response time, Sounds like your voice, Sounds off and Red-team findings. Markers show where your documents or your questions changed, so a jump in the line can be traced to its cause.

Schedules

A schedule runs an audit daily or weekly and alerts you when something changes. On Schedules, create a schedule with:

Setting	Options
Chatbots	One to five. More than one runs them side by side on the same questions, and alerts once for the group.
What to run	Questions (all active, or test sets and picked questions) or Red-team tests (all, or picked). A schedule runs one or the other, not both.
When	Daily, or weekly on a Day, at a Time in your time zone
Name	Optional. Defaults to the chatbots’ names.
Alerts	Up to 10 email addresses and a Slack incoming-webhook URL

A scheduled run costs the same as a run you start yourself. A project can have up to 10 schedules. Use the switch on a schedule to pause or resume it.

Alerts

Alerts are sent only when something changed against the chatbot’s previous run. An alert that always fires gets ignored.

Alert	When
The score falls	By at least the points you set (5 by default)
A new Critical failure	A question fails as Critical that didn’t before
The run fails or can’t start	The run didn’t finish, or didn’t start
Fewer answers sound like the voice	Optional: the share of on-voice answers falls by at least the points you set
A test becomes compromised	Red-team schedules: a test is compromised that wasn’t in the previous red-team run

The Slack webhook must be a public https:// address. It is stored encrypted and never shown again.

When a scheduled run can’t start

Another run is going: the schedule tries again 15 minutes later.
Your balance doesn’t cover the run: that time is skipped, and an alert is sent if The run fails or can’t start is on.

Fail a deploy when your chatbot gets worse

You can start a run from your deploy pipeline and fail the build when the result isn’t good enough.

API keys

An owner creates keys in Workspace settings under API keys. Click New key, give the key a name and click Create key, then copy it from the Copy your key now dialog. The key is shown only once.

Keys start with audk_. A workspace can have up to 20.
A key can start runs and read results, like a member. It can’t manage the team or other keys.
Revoking a key takes effect on its next request.

Send the key as Authorization: Bearer audk_… to https://audit.snoowl.com/api/….

Start a run

POST https://audit.snoowl.com/api/projects/{project_id}/runs
Authorization: Bearer audk_…
Content-Type: application/json

{"target_id": "{chatbot_id}"}

Add "set_ids": [...] to run test sets, or "probe_ids": [...] to run picked questions. Without either, every active question is asked. The reply includes the new run’s run_id. Project and chatbot IDs are in their page addresses in the app.

Check the run

GET https://audit.snoowl.com/api/runs/{run_id}/check?max_critical=0&max_drop=5
Authorization: Bearer audk_…

Parameter	Meaning	Default
`min_score`	Fail when the score is below this	No minimum
`max_critical`	Fail when there are more Critical failures than this	`0`
`max_drop`	Fail when the score dropped more than this many points since the chatbot’s previous finished run	No limit

While the run is going, the reply is {"status": "running", "passed": null, "judged": 12, "total": 120}. Poll until status changes. When the run has finished:

{
  "run_id": "…",
  "report_url": "https://audit.snoowl.com/projects/…/runs/…",
  "status": "completed",
  "passed": false,
  "score": 81.4,
  "previous_score": 88,
  "critical": 1,
  "previous_run_id": "…",
  "reasons": ["1 Critical failure (allowed: 0)"]
}

A run that failed or was cancelled returns "passed": false with the reason. A run where no answers were scored fails with “No answers were scored”.

GitHub Actions

The app gives this step under API keys. Put the key in your repository’s secrets as AUDIT_API_KEY and replace the project and chatbot IDs:

- name: Audit the chatbot
  env:
    AUDIT_KEY: ${{ secrets.AUDIT_API_KEY }}
    AUDIT_API: https://audit.snoowl.com
  run: |
    RUN=$(curl -sf -X POST "$AUDIT_API/api/projects/<PROJECT_ID>/runs" \
      -H "Authorization: Bearer $AUDIT_KEY" -H "Content-Type: application/json" \
      -d '{"target_id": "<CHATBOT_ID>"}' | jq -r .run_id)
    while :; do
      R=$(curl -sf "$AUDIT_API/api/runs/$RUN/check?max_critical=0&max_drop=5" -H "Authorization: Bearer $AUDIT_KEY")
      [ "$(echo "$R" | jq -r .status)" = "running" ] || break
      sleep 20
    done
    echo "$R" | jq .
    [ "$(echo "$R" | jq -r .passed)" = "true" ]

The step fails on any Critical failure or a score drop of more than 5 points. Change max_critical or max_drop, or add min_score, to suit.