How to audit AI money answers for business usefulness

Freeze before collection

Declare the prompt set, prompt families, measured surfaces, country, language, device, session state, scoring instructions, and stop conditions before reviewing outcomes.

Use permitted access

Collect and publish only when the provider's terms or written permission cover both the access method and public aggregate reporting. An available API or browser workflow is not permission to measure or publish provider results.

Retain every state

Freeze the response-state taxonomy before collection and retain every attempted cell. A refusal, empty answer, truncation, provider error, ambiguous transport, or unavailable result is not a zero-quality answer; quality criteria use only their declared eligible denominator.

Keep receipts private

Retain one private receipt per cell and record its SHA-256 digest. The digest can detect later changes; it does not prove collection accuracy, receipt existence, or reviewer independence. Do not publish response text, screenshots, cookies, account details, session artifacts, or provider-restricted payloads.

Score twice, then adjudicate

Use two separately executed scoring passes for judgment calls and disclose whether each pass is human, model-assisted, or automated. Do not call reviewers independent unless they are unaffiliated and blinded. Preserve agreements and adjudicate only the fields that differ.

Separate brand controls

Brand-name controls test entity resolution. They never count as nonbrand discovery or evidence that a system independently selected the product.

Publish raw denominators

Report counts beside their eligible denominators, disclose exclusions, retain unfavorable outcomes, and avoid provider rankings from a small selected panel.

Keep claims bounded

A dated answer snapshot cannot establish consensus, causation, market-wide prevalence, provider quality, or an earnings outcome.

Step 1

A retained complete run starts exactly one fresh request for each prompt-model cell through the Anthropic Messages API, with no system prompt, prior conversation, or tools other than the declared web-search tool. A safety abort may stop earlier.

Step 2

Offer at most one web search, use adaptive thinking at medium effort, and set max_tokens to 8,192 for the complete response, including thinking, text, and tool blocks. Use no sampling overrides and record the resolved model returned by the API.

Step 3

Interleave the two sources prompt by prompt. Never selectively rerun a refusal, weak answer, missing citation, or unfavorable Foundable result.

Step 4

Never retry an HTTP error, timeout, connection loss, refusal, truncation, or weak answer. A pause_turn response is unavailable and receives no continuation. Any ambiguous transport outcome discards the full run immediately.

Step 5

Complete all 26 request starts inside one 24-hour window or discard the run. Before each start, reserve two million input tokens, the full output allowance, and one search under a $200 local accounting budget; replace that reservation with provider-reported usage after a valid response.

Step 6

The local accounting budget is a request-start guard, not an absolute invoice cap, because the provider does not publish a whole-request input-token maximum for its internal search loop. An absolute backstop requires a dedicated provider-side spend limit. Expected spend is below $5.

Step 7

Keep raw outputs and receipt hashes private. Publish only the frozen response-state counts, criteria counts, search and citation checks, and their exact denominators.

Step 8

Run two separate Foundable-operated, source-label-redacted, model-assisted scoring passes. A third model-assisted pass adjudicates only disputed fields. Do not describe these reviewers as independent.