Connect your chatbot

Snoowl Audit asks your chatbot questions over HTTPS, so the chatbot needs an API it can call. On a project’s Chatbot page, click Connect a chatbot and choose one of two connection types.

A project can have several chatbots, such as your production and staging chatbots, or two versions you’re choosing between. Connect each one the same way. Every chatbot in a project is tested against the same documents and questions, and a run can ask up to five of them side by side. See Run an audit.

Type	Use it for
OpenAI-compatible	Any chat-completions API: OpenAI, Azure, OpenRouter, vLLM, or your bot behind one.
Custom JSON	Any HTTPS endpoint that takes JSON and answers with JSON.

Tip

Test a staging copy of your chatbot if you have one. A run sends a few hundred automated questions.

Fields for both types

Field	What to enter
Name	How the chatbot is named in reports, such as “Support bot (production)”.
Endpoint URL	The address Snoowl Audit sends questions to. It must start with https://.
Voice	Optional. How the chatbot should sound: Friendly and plain, Professional, Formal, Short and direct or Warm and caring. Each answer is then marked on voice, partly on voice or off voice. The voice never changes the score.
I’m authorised to test this chatbot	Required. Snoowl Audit records who confirmed it and when.

OpenAI-compatible

Field	What to enter
Endpoint URL	The full chat-completions address, such as https://api.openai.com/v1/chat/completions.
Model	The model name your API expects.
API key	Sent as Authorization: Bearer ….
System prompt	Optional. Use the same one your chatbot runs with in production, so the audit tests what customers get.

Each question is sent as a standard chat-completions request: your system prompt (if any), then the question as the user’s message. The answer is read from choices[0].message.content in the reply.

Giving the System prompt has a second use: the red team can then confirm when the chatbot leaks it. See Red team.

Custom JSON

Field	What to enter
Request body	The JSON your endpoint expects. Put "{{question}}" where the question goes.
Answer path	Where the answer text is in the JSON reply, as dotted keys. answer, data.reply and choices.0.message.content are all valid.
Headers	Up to 12 headers, for API keys or tokens.

The request body starts as:

{
  "question": "{{question}}",
  "history": "{{history}}"
}

Rules for the request body:

• "{{question}}" must be a whole JSON value, in quotes and on its own. "Q: {{question}}" is refused. Because the question replaces a whole value, a question containing quotes or braces can’t break your JSON.
• "{{history}}" is optional and stands for the conversation so far. Inside a list, its messages are spliced in place. Snoowl Audit asks every question on its own, so the history is always empty.

Rules for headers:

• Put a token in a header named Authorization with the value Bearer <your token>. A header named Bearer, Basic or Token is refused, because the scheme belongs in the value.
• Each header name can appear once.

For example, a bot that takes {"message": "..."} and replies {"data": {"reply": "..."}} needs the request body {"message": "{{question}}"} and the answer path data.reply.

How keys and headers are stored

API keys and header values are encrypted before they’re stored, and they’re never shown again in the app. To keep a saved value when you edit a connection, leave its field empty.

Deleting a connection (Delete in the chatbot’s menu) removes the stored URL, headers and keys. Past run reports stay.

Test the connection

Click Test connection. Snoowl Audit sends the question “Hi! What can you help me with?” and shows the result:

Status	Meaning
Connected	The chatbot answered. The time it took is shown.
Not answering	The request failed. The error says why.
Not tested yet	Test the connection before starting a run.

Network rules

Every request to your chatbot follows these rules:

Rule	Detail
HTTPS only	The address must start with https:// and can’t contain a user name or password.
Public addresses only	The host must resolve to public IP addresses. Private and internal addresses are refused.
No redirects	A redirect counts as a failed answer. Use the final address.
Timeout	60 seconds per question
Reply size	Up to 1 MB
Reply format	JSON, with text at the answer path

When a question fails, the run records the error and carries on. Errors you might see:

Error	What to check
http 401, http 403	The API key or token header
http 404	The endpoint URL
reply is not JSON	The endpoint returns JSON
nothing at reply path '…'	The Answer path matches the reply
reply over size cap	The reply is under 1 MB
refused url: …	The address is HTTPS and public

A failed question is never judged or charged. It is listed under Needs review in the report.

Caution

If the first six or more questions of a run all fail, the run stops with the message “The chatbot failed every question”. Fix the connection on Chatbot, then start a new run.

No API access? Paste the answers

If your chatbot can’t be reached by API, you can still audit it by hand. Start a run with Manual selected (see Run an audit) and click Open answer sheet.

• Ask your chatbot each question yourself and paste exactly what it said. Answers save as you type. Each answer can be up to 8,000 characters.
• Or click the More menu, choose Download as CSV, fill in the answer column, and use Import answers from CSV. Rows are matched by question_id first, then by the question’s wording, so a sheet still imports after the ID column is deleted. Rows that can’t be matched are listed.
• When you’re done, click Judge answers. Only answered questions are judged and charged; unanswered ones are left out of the report. Answers can’t be changed after this.

Red-team tests can’t be run by hand.