How a score is made.
What is measured, how, and where the measurement is weak. — the calibration. Raw scores are on the data page.
Tracked in open gaps.
Six categories that fail differently
queries in categories, chosen so that the categories fail differently rather than all measuring the same thing.
| Category | Queries | What it tests |
|---|
Where the queries come from
Every query is authored for this benchmark rather than lifted from an existing dataset. That is a deliberate trade: an academic set would carry more external validity, but the licences on the obvious candidates are either restrictive, unresolved, or built around a fixed pre-indexed corpus rather than the live web. Authoring the set means it ships with the export under an open licence from day one, and it means nobody has to take on trust that the questions were not chosen to favour a result.
The general-facts and multi-hop sets are written in the style of SimpleQA and HotpotQA respectively, without reusing either corpus. The trade-off worth naming: an authored set is easier to accuse of being unrepresentative, and the honest answer is that the whole set is published, so that accusation is checkable rather than rhetorical.
Breaking-news queries carry no expected answer
A freshness question with a fixed answer decays into a general-knowledge question within weeks, which quietly turns the freshness column into a second facts column. So those queries are written in perpetually-current form ("what is the latest…", "who currently holds…") and the judge grades them on source recency and internal consistency instead of answer match.
The published set does not rotate. An earlier version of this page said those queries were flagged to rotate every cycle so vendors could not overfit to them. The flag exists in the data and nothing reads it — no code in this repository rotates anything, and every run so far has used the same query-set hash. The sentence described an intention as though it were a mechanism, which is the failure this project exists to be an alternative to, so it is gone rather than softened.
Overfitting to the published list is instead what the withheld set is for, and it is the honest place to look: the published questions are all visible in the export, so a vendor that wanted to tune against them could. The withheld set has not run yet, so it currently detects nothing — that is stated there too.
Scoped by contract, not by popularity
Several search APIs have terms of service that restrict third-party benchmarking, and this project's position is that the rule applies to it too.
| Vendor | Returns | Notes |
|---|
- Tavily and Brave
- Held back deliberately. Both have terms broad enough to be read as covering third-party benchmarking, and no written consent has been sought. Both are better known than several vendors that are included, so their absence costs this benchmark reach. That is the intended trade.
- Vendors with explicit bars
- Two vendors surveyed have unambiguous contractual prohibitions on benchmarking or competitive analysis without prior written consent. They will not appear here without that consent in hand.
- Adding a vendor
- A vendor enters the benchmark only after its terms have been read and recorded. In the code, that gate is a single registry: adding an entry to it is the act of adding a vendor to a published comparison, and it is commented as such.
- What gets published about a vendor
- Derived scores, timings and costs. Never the URLs, titles, snippets or synthesized answers a vendor returned. That reduces copyright and terms-of-service exposure at the same time, and it is enforced in the export code rather than left to care.
Exactly how each vendor is called
A wrong endpoint, tier or parameter is the most likely reason a published score is unfair, and the cheapest kind of dispute to settle — so the configuration is stated here rather than left in the source for a vendor to find.
- Exa —
/search, default search type, withcontents.textrequested. That is a billed add-on and it is asked for deliberately: every other vendor returns snippet text in its default response, and Exa does not, so not requesting it would grade Exa on an empty payload. Its cost is vendor-reported on every call, so the add-on is already inside the published price. - Perplexity —
sonar, notsonar-deep-research, which is 10–30× the cost and a different product. Output capped at 2048 tokens; it was 512, which truncated 10 of 150 answers and cost Perplexity about half a point on those. - Serper —
/search, default settings. It is the one vendor that sometimes returns fewer than the requested depth. - Linkup —
standarddepth, notdeep, which is a different and dearer product tier. - You.com — the search endpoint, not the Research tier.
Every response is truncated to the same top results and the same snippet length before a judge sees it. If any of this is the wrong way to call your API, that is a correction worth making and rerunning — say so in public and it gets one.
Vendors: written consent is the only route into the published set. Get in touch.
Making responses comparable
Vendors answer in different shapes and different lengths. Four rules make the comparison fair before a judge ever sees a response.
Uniform depth
Vendors return anywhere from eight to twenty results by default. Scoring whatever each one hands back would reward verbosity, so every response is truncated to the top before it reaches a judge. The rubric separately instructs the judge not to reward length.
Two shapes of answer
Most of these APIs return a ranked list. One returns a prose answer and a ranked list. The shape of each response is recorded and published alongside its score. The judge is not told which shape it is looking at: an earlier version added the words "plus a synthesized prose answer" to the prompt, which named the response shape for exactly one vendor in the set and nothing for the rest — a blinding hole no amount of rubric wording compensates for. The prose is still visible in the payload under its own heading, so nothing is hidden from the judge; it is just no longer flagged.
Uniform snippet length
Snippets reach the judge truncated at characters, which is at least what any adapter asks a vendor for. It used to be 400, below what two vendors return, so the cap silently discarded about a fifth of their text and none of anyone else's — a per-vendor handicap dressed as a normalisation.
Cost
Where a vendor reports the cost of a call in its response, that reported figure is used; where it does not, a published price is used. Which of the two applies is a column in the responses export rather than something to infer, and the tier behind every derived price, with a source link, is in pricing.json. It matters: the headline cost ratio is at the tiers this benchmark is billed at and with both sides undiscounted.
Latency
Wall-clock time from request to response, over an ordinary internet connection, with modest request concurrency. Two known confounds, both real: concurrency means these are not isolated measurements, and vendors cache. A run repeated within minutes of another can be served warm — that happened once here, and the affected run is identified in the changelog. Treat ordering as the signal and absolute milliseconds as indicative.
Does a bigger response score better?
The rubric tells the judges not to reward length, and saying so does not make it true. The judge-visible payload differs by roughly a factor of two between the largest and smallest vendor here, so the question deserves a number rather than an assurance.
Within a single vendor — the clean test, with no vendor differences in it — the correlation between payload size and score runs from to . That is nothing. The clearest single illustration is in the table itself: the second-largest payload in the set belongs to the lowest-scoring vendor. Per-vendor figures are in the export, so this can be recomputed rather than believed.
Three models, three labs, one rubric
Three models from three different vendors score every response independently against one rubric. The published score is the median.
| Family | Model | Mean score, this run | Coverage |
|---|
Why three families
LLM judges systematically prefer text that reads like their own output. The effect is driven by fluency rather than literal self-recognition, so it fires even on text the judge did not write. Search APIs increasingly synthesize their answers with the same class of model, which makes a single-judge design structurally unsafe here.
The results page publishes the measured gap between the three families for exactly this reason: on this run they differ by points on the same responses. The ensemble does not remove that bias. It makes it visible and stops any one family from setting the number alone.
Median, and only when complete
The score for a response is the median of the three judges, not the mean, so one outlier cannot move a published figure. Responses missing any judge are dropped entirely rather than averaged over the survivors: rate-limit failures cluster in time rather than falling randomly, so mixing three-way medians and two-way means in one column would introduce bias, not just thin the sample.
Salvaging truncated replies
Judges sometimes truncate mid-sentence in the optional rationale field, after emitting complete numeric scores. Discarding those replies wholesale would shrink the sample non-randomly, because longer vendor payloads truncate more often. The parser recovers the scores field-wise and drops only the rationale.
The rubric, as the code sends it
read from the exportHow often the judges disagree, and where
A mean spread of points reads as broad agreement. It is equally consistent with every response splitting the judges slightly and with most agreeing exactly while a tenth split by five. Those are different benchmarks, so the distribution is published rather than the average.
It concentrates by category
Where the ensemble is split, the ordering within that category is carrying less information than the number of decimal places suggests, and the honest response is to discount that category — not to discount the whole run, and not to hide the figure.
Which two families
The widest pair on this run is of mean absolute difference. Three judges from three labs is the bias mitigation; it is not a claim that the three agree. The full pairwise matrix is in the export.
Spread by category
mean max−min, this run| Category | Mean spread | Split > 2 pts | n |
|---|
How well the judges track a person
Judge disagreement is agreement between models, which is a weaker and different quantity: three models can agree with each other and be wrong together. This is the other measurement — one human, labelling blind, against the ensemble's own choice.
What the number is
What it is not
How the labeller was blinded
The set was drawn and its id committed before any of it was labelled, and the labels
themselves are in the repository — labels/ — so the figure above can be
recomputed rather than taken on trust. Pairwise, not absolute: a person asked to put a
number on a search result is unreliable at it, while a person asked which of two is
better is not, and the earlier absolute attempt is in the history for anyone who wants
to see it fail.
Questions nobody can tune against, committed before they run
The published query set is public, which is right, and it means a vendor could in principle tune against those exact questions. Alongside every run goes a second set that is not published — and a hash of it that is, dated, before it is ever used.
Withheld, not hidden
A secret question set with a secret score would deserve exactly as much trust as a vendor's own benchmark. Three things make this one checkable instead. Its SHA-256 is committed to the repository before it first runs, so the questions cannot be swapped after a result is seen. It retires on a fixed rotation — every weeks — and is published in full at that point, hash included. And only the text is withheld: every withheld response's scores, latency and cost are in the export already, flagged held_out=1.
What it measures
Not a second leaderboard. The withheld set is a fraction of the size of the public one, so its absolute scores are too noisy to rank anyone by. What it measures is the gap within one vendor: the same API, in the same run, on the same day, through the same judges, scored on published questions against questions that were never published. A vendor that has tuned for the public list scores better on it. Both sides are averaged per category first, so a harder bucket cannot masquerade as a vendor-specific gap.
What it cannot do
A withheld question is not invisible to the vendor being asked it — vendors receive every query this benchmark sends and can see them in their own logs. What this defeats is cheap optimisation against a published list. What it does not defeat is a vendor that deliberately identifies benchmark traffic and treats it differently; that would need traffic mixing, which is not implemented. Stated here rather than left for someone else to point out.
The current commitment
read from the export- Questions
- , across all six categories
- Committed
- , before first use
- SHA-256
- Status
Public minus withheld
positive = better on the published set| Vendor | Public | Withheld | Gap |
|---|
From scores to a published table
Four stages, each of which can drop a number rather than publish a weak one.
-
Response score
Median of the three judges' overall scores. Dropped if any judge is missing.
-
Category cell
Mean of the response scores for that vendor in that category. A cell whose coverage falls below of its queries is published as missing rather than as a number computed from a thin sample.
-
Vendor score
Mean of that vendor's category cells, so every category weighs the same regardless of how many of its queries scored cleanly.
-
One run per cycle is published
A cycle can contain smoke tests and aborted runs. The published run is chosen explicitly: it must cover at least queries in every category, and among those that qualify the one with the most complete judge ensembles wins. Every candidate run and its completeness ships in the export.
Open gaps
Known distance between the design and what currently runs. If one of these blocks your use of the data, say so.
- Short track record
- of data.
- Scores cluster high
- Most results land between 7 and 10. Either these APIs are all good at these questions, or the questions are too easy to discriminate at the top. Harder queries are the next change to the set, and it will move the numbers.
- Positional bias is untested
- The standard mitigation is to run every comparison in both orders and count a preference only when both agree. This benchmark scores each response independently rather than comparing pairs, so positional bias does not arise in the current design. If any pairwise sub-metric is added later, that mitigation has to come with it.
- Single geography, single run
- All queries were issued from one location at one time. Search results vary by region and by hour, and this benchmark does not yet measure either.
- Not a legal review
- The vendor scoping described above is based on reading published terms of service, not on advice from counsel. It reflects a deliberately cautious reading, which is not the same thing as clearance.
Who pays for this, and what that could distort
This benchmark is free and intends to fund itself from products built on top of it. That creates real incentives, and the useful time to write them down is before anyone has a reason to ask.
The conflict that exists
The plan is to sell three things: a live score feed, private evaluations run against a customer's own queries, and a routing client. One proposed pricing tier charges a share of the spend a customer saves by routing — which would mean earning more when traffic moves to cheaper vendors, while also publishing the ranking that says which vendors are good enough to move to. That is a genuine conflict. It is disclosed here rather than denied, the flat per-decision tier that pays the same whichever vendor is chosen is the default, and the conflict will be stated on the pricing page itself.
The conflicts that do not
No vendor pays anything, for any reason — no sponsorship, no placement, no listing fee, no advertising. No equity, employment, advisory role or consulting with any benchmarked vendor or its parent. No vendor sees a score before it is published: no embargo, no preview, no advance right of reply. No vendor influences the query set, the rubric or the judge models. Every vendor account is paid for at list price.
How to dispute a number
Publicly, and specifically. Every individual judge score is already in the export, so a dispute can name a query and a judge rather than object to a total. Configuration disputes — wrong endpoint, wrong tier, wrong parameters — are the most likely kind and the fastest to fix; the exact request made to each vendor is in the source. A correction that moves a published number gets a dated changelog entry, never a silent edit. No NDAs, and no private resolution of a public dispute.
What should make you distrust this
each one externally checkable- The cheap vendors' advantage grows over time in a way that tracks the introduction of savings-share pricing.
- The methodology changes shortly after a vendor relationship changes.
- The withheld set stops being published on retirement, or its retirement is postponed indefinitely.
- The disagreement figures above disappear, or stop being broken out by category.
- A vendor leaves the benchmark without a stated reason.
- The disclosures stay empty after the commercial products start earning.
Run the whole thing yourself
You need your own API keys for the vendors and the judge models. A full run of queries cost in vendor calls.
git clone && cd vannaris python -m venv .venv && .venv/bin/pip install -r requirements.txt cp .env.example .env # add your keys .venv/bin/python scripts/check_keys.py # verify all eight, one cheap call each .venv/bin/python -m src.runner --queries src/queries/full-v1.json .venv/bin/python -m src.export # rebuild site/data and site/export
Or skip the run entirely: every number on this site can be recomputed from the published CSVs with no API keys at all. Worked example.
Methodology changelog
A judge model swap or a rubric edit changes scores without anything changing about the vendors. Those changes belong in a log, not in a silent commit.
- 2026-08-04 · methodology v2 — scores from here are not comparable with the 31 Jul 2026 run
-
The largest change so far, and the reason it is one entry rather than five: everything in it alters what a judge sees or how it answers, so week-over-week comparison across this line is not valid. It is made now, against a single published week, because that is the cheapest moment it will ever be.
- The judges are told today's date. The rubric never stated it, and one judge family was reading post-cutoff results as fabricated — marking vendors down for returning current information, in the column that exists to reward exactly that. It showed up as a 2.4-point family gap on breaking news that is absent everywhere else.
- Sampling is pinned. Judges now run at temperature 0, with a fixed seed where the provider offers one. Previously each score was a temperature-1 sample: on responses whose vendor payload was byte-identical, the ensemble median reproduced only about 70% of the time. Pinning a model version bought nothing without this.
- The snippet cap no longer binds below what adapters request. It was 400 characters against 500 requested, so it discarded about a fifth of two vendors' text and none of anyone else's.
- The response shape is no longer named in the prompt. It was announced for the vendors returning prose and not for the others, identifying one vendor to an otherwise blind judge.
- One vendor's output cap was raised from 512 to 2048 tokens. The harness imposed it, the vendor did not, and it truncated 10 of 150 answers.
The 31 Jul 2026 table stands as published. It is not being rescored — a corrected week is published beside the original, never over it.
- 2026-08-04 · reporting corrections, scores unchanged
- Three published figures were wrong or unsupported and none of them were scoring changes. The queries-won column resolved ties by internal ordering, so it credited vendors with wins they had not had; it is replaced by outright wins and shared-best, with the tie rate published beside them. Two pages asserted that the ranking survives the choice of judge, which this run's own per-judge data contradicts; that is replaced by the measured depth of agreement. The cost ratio was quoted at tiers that were not like for like; both bases are now published with a source per vendor.
- 2026-08-04 · withheld set added
- A rotating set of questions now runs alongside the public one without being published. Its hash is committed before it runs and its questions are published when it retires (above). Scoring is unchanged: no published cell includes a withheld question, so every number on this site is still recomputable from the published query set. The first set is registered and has not run yet.
- 2026-08-04 · disagreement published as rates
- Judge disagreement was published as a single mean. It is now published as the share of responses over each threshold, per category, with the pairwise family differences — the mean cannot distinguish broad mild disagreement from occasional severe disagreement, and on this run the difference is large. No scores changed; the reporting did.
- 2026-08-04 · coverage threshold displayed correctly
- This page and the data page stated the cell-coverage publication threshold as 1%. The threshold enforced in code was, and remains, ; a formatter was rendering a 0–1 share as though it were already a percentage. Display only — no published score was affected.
- 2026-07-31 · initial
- First published run. Three-judge cross-family ensemble, median aggregation, top-10 normalisation, -query set ().
- 2026-07-31 · judge pin reverted
- A newer Google judge model was trialled and reverted before publication: it truncated its JSON on roughly a third of calls under the production-length judge prompt, having passed a short probe. Judge models are now validated against a full-length prompt before pinning.
- 2026-07-31 · aggregate layer guarded
- A two-query smoke test overwrote one category's published cells, and the result was indistinguishable from a real one. Aggregation now refuses to overwrite a cell computed from more queries, records which run produced each cell, and the site recomputes from the per-judge scores of one explicitly chosen run.