How we benchmark a fact-checker (and why our numbers aren't up yet)

"95% accurate" with no dataset attached is unverifiable — exactly the kind of statement our pipeline would flag. For a fact-checking product the bar is higher still, because the failure is recursive: if you can't trust the accuracy claim, you can't trust anything downstream of it. So the rule we set ourselves is simple: publish measurements, not projections — and until a measurement exists, say so plainly. That's why our benchmarks page currently shows the shape of the results rather than numbers.

Two kinds of data, because each catches what the other misses:

• Public claim-verification datasets with ground-truth labels. These make our results comparable to published research and impossible to cherry-pick — the labels were assigned by someone else, before we showed up.
• Curated documents seeded with known claims — a controlled mix of true, false, and deliberately misleading statements woven into realistic prose. Public datasets test claim-level judgment; these test the whole pipeline, including the step most benchmarks skip: whether the system even extracts the right claims from a messy document before judging them.

• Verdict accuracy. Agreement with ground-truth labels. Our verdicts live on a 7-point scale, but public labels are usually binary or ternary, so for scoring we collapse to supported / refuted / unverifiable and say so — comparability beats flattering granularity.
• Calibration. When we say 80% confidence, are we right about 80% of the time? A fact-checker that is 70% accurate and knows it is more useful than one that is 85% accurate and claims 99. Our confidence scores are computed from panel agreement, source authority, and citation faithfulness rather than self-reported — calibration is the test of whether that formula earns its keep.
• Stability. The same document, submitted twice, should yield the same claims, the same price, and the same verdicts. We already engineer for this — deterministic settings, majority-voted claim extraction — and we measure it, because a fact-checker that changes its mind between identical runs is a random-number generator with good manners.

Every number that lands on the benchmarks page will carry its dataset, its date, and the exact panel configuration that produced it — because "the product" isn't one fixed system: panels are configurable and the model lineup evolves. A number without its configuration is marketing; a number with it is a measurement someone else could check.

What we won't publish: aggregate scores that blend tiers with different evidence standards, results on datasets we can't name, or a single headline percentage stripped of its error cases. Our methodology page documents the pipeline's known failure modes; the benchmark's job is to quantify them, not to hide them.

Two reasons, and they compound. First, we're iterating on the pipeline rapidly — panels, judges, and adjudication logic improve week to week — and our first major quality milestone is planned for October 2026. A benchmark of last month's pipeline would be a precise measurement of a product that no longer exists.

Second, running the full suite honestly is expensive. Every claim voted on by every panel model, judged, adversarially challenged, across tiers and datasets, repeated enough times to measure stability — that's an inference bill that can reach into the tens, potentially hundreds, of thousands of dollars. For a startup, that's not money you spend measuring a moving target. We'd rather spend it once, on the milestone build — a configuration worth freezing, measuring, and standing behind.

When the results land on /benchmarks, we'll write up what surprised us — including whatever doesn't flatter us.