JavaScript is needed for interactive evidence views. Download the frozen data from the public JSON export, or enable JavaScript to inspect results.

Explore the results

Compare quality, cost, and latency across search APIs. Inspect individual judgments or download the data. Every chart has a table view.

Snapshot

Coverage and publication notes

how that was measured

queries

Quality by vendor and category

Each cell is the mean of the three-judge median. Queries without a complete ensemble are dropped.

Score matrix

0–10

Where each vendor's queries land

share of scored queries

The score column below is a mean; this is the spread behind it. Almost everything lands in the top three bands, which is why the query set is listed as a known limitation.

Vendor summary

this run

Overall score is the equal-weight mean of all required category scores. Vendors missing a publishable category are not ranked. Intervals describe uncertainty in the preserved judged sample; unresolved differences are not evidence of equivalence. "Won outright" counts queries where one vendor alone took the highest ensemble median; "among best" counts queries where it was level with the highest. Those differ a great deal here, because of queries ended in a tie, which is itself the most useful thing this table says.
Vendor	Score & evidence	Cost / query	Run cost	p50 latency	Won outright	Among best	Scored	Returns

Routing headroom

Per query it is thinner still: of queries end with two or more vendors tied for the best median, so on most of them there is no pick to make.

Cost against quality

Cost is on a log scale. Volume discounts are not modelled.

Cost per query vs score

list price per query

Cost is what the vendor reported for this run where the API returns it, and list price otherwise. Volume discounts are not modelled.

Share of the best score, by category

choose a vendor

How much of the category-leading rubric score this vendor reaches. Bars start at 70%. This ratio is not measured agent-task success.

How long each API takes to answer

Median wall-clock time from request to response, measured by the harness.

Median response time

p50, measured

Measured over a residential connection during a single run. Treat the ordering as the signal and the absolute values as indicative.

How much the judges disagree

A single judge family would produce different absolute scores for the same responses.

How far apart the three judges were

per response

Only complete three-judge panels are compared. The mean disagreement elsewhere on this page summarizes this distribution; partial panels are excluded.

Mean score by judge family

same responses

Same responses, same rubric, same prompt. The gap is the judge, not the vendor.

family spread

points, highest to lowest family mean

mean disagreement

points, max minus min per response

differ by > 3 points

of responses

Drop any one judge family and the top of places hold. Scored by a single family alone, of rank the vendors differently. How that was measured.

Every query, every score

Search or filter the questions, then inspect an individual query for judge scores, missingness and API timing.

category search

Incomplete cells did not receive a complete judge ensemble. Inspect the query to distinguish a vendor failure from missing judge scores. They are excluded from every aggregate on this page. Breaking-news queries carry no gold answer by design, and are graded on source recency.