How we benchmark a fact-checker (and why our numbers aren't up yet)

"95% accurate" with no dataset attached is unverifiable. It is the kind of statement our own pipeline would flag. For a fact-checking product the bar is higher still, because the failure is recursive. If you can't trust the accuracy claim, you can't trust anything downstream of it. So the rule we set ourselves is simple: publish measurements, not projections, and until a measurement exists, say so. That's why our benchmarks page currently shows the shape of the results rather than numbers.

Two kinds of data, because each catches what the other misses:

• Public claim-verification datasets with ground-truth labels. These make our results comparable to published research and impossible to cherry-pick, because someone else assigned the labels before we showed up.
• Curated documents seeded with known claims. A controlled mix of true, false, and misleading statements written into realistic prose. Public datasets test claim-level judgment. These test the whole pipeline, including the step most benchmarks skip: whether the system extracts the right claims from a messy document before judging them.

• Verdict accuracy. Agreement with ground-truth labels. Our verdicts use a 7-point scale, but public labels are usually binary or ternary, so for scoring we collapse to supported, refuted, or unverifiable and say so. Comparability beats flattering granularity.
• Calibration. When we say 80% confidence, are we right about 80% of the time? A fact-checker that is 70% accurate and knows it is more useful than one that is 85% accurate and claims 99. We compute confidence scores from panel agreement, source authority, and citation faithfulness rather than asking the model. Calibration tests whether that formula works.
• Stability. The same document, submitted twice, should yield the same claims, the same price, and the same verdicts. We already engineer for this with deterministic settings and majority-voted claim extraction, and we measure it. A fact-checker that changes its mind between identical runs is returning noise.

Every number that lands on the benchmarks page will carry its dataset, its date, and the exact panel configuration that produced it. "The product" isn't one fixed system. Panels are configurable and the model lineup changes. A number without its configuration is marketing. A number with it is a measurement someone else could check.

What we won't publish: aggregate scores that blend tiers with different evidence standards, results on datasets we can't name, or a single headline percentage stripped of its error cases. Our methodology page documents the pipeline's known failure modes. The benchmark's job is to quantify them, not to hide them.

Two reasons, and they compound. First, the pipeline changes week to week. Panels, judges, and adjudication logic all still move, and our first major quality milestone is planned for October 2026. A benchmark of last month's pipeline would be a precise measurement of a product that no longer exists.

Second, running the full suite is expensive. Every panel model votes on every claim, a judge and a challenger run, across tiers and datasets, repeated enough times to measure stability. That inference bill can reach into the tens of thousands of dollars, and possibly hundreds of thousands. For a startup, that's not money you spend measuring a moving target. We'd rather spend it once, on the milestone build, a configuration we will freeze, measure, and stand behind.

When the results land on /benchmarks, we'll write up what surprised us, including whatever doesn't flatter us.