Case study: Stamp AI as a guardrail for LLM answers

The obvious guardrail is a second model asked "is this right?". It shares the first one's problem: it answers from memory, so a confident error gets a confident second opinion. Stamp AI never rules from memory. For each claim it searches the live web for the claim and for any correction or denial, screens what comes back (including for prompt injection planted in pages), and rules on that evidence alone. Then every citation it relies on is checked against what the source says.

It is also measured. On real-world claims that professional fact-checkers had settled, its high-confidence verdicts were right 97.2% of the time, and it called 89.8% of all of them right (the full numbers). That is what lets you write a policy around its output instead of hoping.

1. Generate the answer as you do today.
2. Check it: send the text to the API with tiers: ["stamp"]. We extract the checkable claims (opinions and pleasantries cost nothing), run them in parallel, and return a verdict, a confidence band and sources for each.
3. Decide in code: show it, show it with sources, regenerate it with the refuted claims fed back, or route it to a person.

// guard.ts — check an LLM answer's facts before it ships
const API = "https://fact.engineering/api/v1";
const auth = { Authorization: `Bearer ${process.env.FACT_ENGINEERING_TOKEN}` };

type Band = "high" | "moderate" | "low";
type Verdict = "true" | "mostly_true" | "mixed" | "misleading"
  | "mostly_false" | "false" | "unverifiable";

export async function checkAnswer(answer: string) {
  const res = await fetch(`${API}/checks`, {
    method: "POST",
    headers: { ...auth, "Content-Type": "application/json" },
    body: JSON.stringify({
      text: answer,
      tiers: ["stamp"],
      // Bots repeat themselves: an unchanged claim verified in the last
      // 24 hours keeps its verdict, free.
      reuseVerdicts: true,
    }),
  });
  const check = await res.json();
  if (!res.ok) {
    // 422 no_claims: nothing checkable in the answer (pleasantries, opinions).
    if (check.error?.code === "no_claims") return [];
    // 402: out of credits; 429/422: rate or extraction limits. Fail open or
    // closed here, your call; never show the error to the reader.
    throw new Error(`fact check failed: ${check.error?.code ?? res.status}`);
  }

  // Poll (or pass a callbackUrl and get a signed webhook instead).
  for (;;) {
    const r = await fetch(`${API}/checks/${check.id}/report`, { headers: auth });
    const report = await r.json();
    if (["completed", "failed"].includes(report.check.status)) {
      return report.claims.map((c: any) => ({
        claim: c.text,
        verdict: c.consensus[0]?.verdict as Verdict | undefined,
        band: c.consensus[0]?.confidenceBand as Band | undefined,
        sources: c.evaluations.flatMap((e: any) => e.sources),
      }));
    }
    await new Promise((ok) => setTimeout(ok, 2000));
  }
}

// The policy is yours, and it belongs in code, not in a prompt.
const FALSEISH = ["false", "mostly_false", "misleading"];

export function decide(results: Awaited<ReturnType<typeof checkAnswer>>) {
  const refuted = results.filter(
    (r) => r.verdict && FALSEISH.includes(r.verdict) && r.band !== "low",
  );
  if (refuted.length) return { action: "regenerate", refuted } as const;

  const shaky = results.filter((r) => !r.verdict || r.band === "low");
  if (shaky.length) return { action: "show-with-sources", shaky } as const;

  return { action: "show" } as const;
}

Treat unverifiable and a low band as "we don't know", not as "false". Regenerating on every unknown just buys a different unchecked answer.

A Stamp claim takes about 8 seconds at the median, and every claim in a text runs at once, so an answer is checked in roughly the time of its slowest claim. That is too slow to block a live chat reply on, and fast enough for almost everything else:

• Anything published or sent: generated articles, product pages, newsletters, support macros, help-centre answers. Check before it goes out; a webhook tells your pipeline when.
• Live chat, after the fact: stream the reply as usual, check it in the background, and annotate or correct it a few seconds later. The reader sees the answer immediately and the sources shortly after.
• High-stakes answers only: route just the answers that state numbers, dates, prices, policies or medical, legal and financial facts through the check.

Stamp AI costs 5 credits a claim plus a per-check fee of 10 credits a claim (capped at 100). A typical 4-claim bot answer is 60 credits ($0.60); a 20-claim generated article is 200 credits ($2.00).

That is cheap next to a published error and expensive next to the tokens that wrote a chat reply. So don't check every message of a high-volume bot. Three patterns keep the bill proportional:

• Re-use. Support bots say the same things all day. With reuseVerdicts on, a claim your organization verified in the last 24 hours keeps its verdict for free, so an answer with one new claim and three repeats costs 45 credits instead of 60.
• Sample to measure. Check one answer in fifty. You get a hallucination rate for your bot, broken down by topic, for a fiftieth of the price, and you know where to tighten retrieval or prompts.
• Check what ships, not what's drafted. In a generation pipeline, check the final text once, not every intermediate draft.

One limit to plan for: finding the claims in a new text runs before anything is charged, so it is capped at 100 new texts an hour per organization. That covers sampling and most publishing pipelines; if you need to check every answer of a busy bot, talk to us about raising it.

Stamp AI checks facts about the world. It can't know your private data ("your order shipped Tuesday") unless it is in your Knowledge Bank, and it rarely answers "unverifiable" on thin or contested claims: it tends to pick a side, so a low band is your cue to look. For claims you'll defend in public, the Research Panel has models from different providers check each one independently, at a higher price. The API docs have the full request and webhook reference.