Methodology
The workload, API configurations, judge ensemble, and coverage rules behind the published scores.
Coverage and publication notes
Query workload
authored public queries across categories, each covering a different kind of search task.
| Category | Queries | What it tests |
|---|
Where the queries come from
Every query is authored for this benchmark. Existing datasets considered for reuse had restrictive or unresolved licences, or targeted a fixed corpus rather than the live web. The authored public workload ships with the export under an open licence so its selection can be inspected.
The general-facts and multi-hop sets are written in the style of SimpleQA and HotpotQA respectively, without reusing either corpus. Publishing the full workload makes its scope inspectable; it does not establish that these questions represent production traffic.
Breaking-news queries carry no expected answer
A freshness question with a fixed answer decays into a general-knowledge question within weeks, which quietly turns the freshness column into a second facts column. So those queries are written in perpetually-current form ("what is the latest…", "who currently holds…") and the judge grades them on source recency and internal consistency instead of answer match.
The published set does not rotate. An earlier version of this page said those queries were flagged to rotate every cycle so vendors could not overfit to them. The flag exists in the data, but no code uses it to rotate queries. Every published run has used the same query-set hash. Query rotation remains an unimplemented proposal.
The withheld set provides a diagnostic for differences between public and withheld questions. Its observed run coverage is recorded below; registration alone is not evidence that it ran, and a score gap does not prove overfitting.
API scope
Included APIs, request configurations, and the terms governing evaluation and publication.
| Vendor | Returns | Notes |
|---|
- Tavily and Brave
- Held back deliberately. Both have terms broad enough to be read as covering third-party benchmarking, and no written consent has been sought. Both are better known than several vendors that are included, so their absence costs this benchmark reach. That is the intended trade.
- Vendors with explicit bars
- Two vendors surveyed have unambiguous contractual prohibitions on benchmarking or competitive analysis without prior written consent. They will not appear here without that consent in hand.
- Adding a vendor
- A vendor enters the benchmark only after its terms have been read and recorded. In the code, that gate is a single registry: adding an entry to it is the act of adding a vendor to a published comparison, and it is commented as such.
- What gets published about a vendor
- Derived scores, timings and costs. Never the URLs, titles, snippets or synthesized answers a vendor returned. That reduces copyright and terms-of-service exposure at the same time, and it is enforced in the export code rather than left to care.
Request configuration
Endpoints, product tiers, and response limits affect both quality and cost. These are the configurations used in the study.
- Exa —
/search, default search type, withcontents.textrequested. That is a billed add-on and it is asked for deliberately: every other vendor returns snippet text in its default response, and Exa does not, so not requesting it would grade Exa on an empty payload. Its cost is vendor-reported on every call, so the add-on is already inside the published price. - Perplexity —
sonar, notsonar-deep-research, which is 10–30× the cost and a different product. Output capped at 2048 tokens; it was 512, which truncated 10 of 150 answers and cost Perplexity about half a point on those. - Serper —
/search, default settings. It is the one vendor that sometimes returns fewer than the requested depth. - Linkup —
standarddepth, notdeep, which is a different and dearer product tier. - You.com — the search endpoint, not the Research tier.
Every response is truncated to the same top results and the same snippet length before a judge sees it. If any of this is the wrong way to call your API, that is a correction worth making and rerunning — say so in public and it gets one.
Vendors: written consent is the only route into the published set. Get in touch.
Response normalisation
Common result-depth and snippet limits reduce format differences. Output shape, cost basis, and latency confounds remain part of the comparison.
Uniform depth
Vendors return anywhere from eight to twenty results by default. Scoring whatever each one hands back would reward verbosity, so every response is truncated to the top before it reaches a judge. The rubric separately instructs the judge not to reward length.
Two shapes of answer
Most of these APIs return a ranked list. One returns a prose answer and a ranked list. The shape of each response is recorded and published alongside its score. The judge is not told which shape it is looking at: an earlier version added the words "plus a synthesized prose answer" to the prompt, which named the response shape for exactly one vendor in the set and nothing for the rest — a blinding hole no amount of rubric wording compensates for. The prose is still visible in the payload under its own heading, so nothing is hidden from the judge; it is just no longer flagged.
Uniform snippet length
Snippets reach the judge truncated at characters, which is at least what any adapter asks a vendor for. It used to be 400, below what two vendors return, so the cap silently discarded about a fifth of their text and none of anyone else's — a per-vendor handicap dressed as a normalisation.
Cost
Where a vendor reports the cost of a call in its response, that reported figure is used; where it does not, a published price is used. Which of the two applies is a column in the responses export rather than something to infer, and the tier behind every derived price, with a source link, is in pricing.json. It matters: the headline cost ratio is at the tiers this benchmark is billed at and with both sides undiscounted.
Latency
Wall-clock time from request to response, over an ordinary internet connection, with modest request concurrency. Two known confounds, both real: concurrency means these are not isolated measurements, and vendors cache. A run repeated within minutes of another can be served warm — that happened once here, and the affected run is identified in the changelog. Treat ordering as the signal and absolute milliseconds as indicative.
Payload size and scoring
The rubric tells the judges not to reward length, and saying so does not make it true. The judge-visible payload differs by roughly a factor of two between the largest and smallest vendor here, so the question deserves a number rather than an assurance.
Within a single vendor — the clean test, with no vendor differences in it — the correlation between payload size and score runs from to . That correlation alone does not rule out length bias or establish causation. Read it alongside the scoring design and sample limitations. Per-vendor figures are in the export, so this can be recomputed rather than believed.
Judge ensemble
Three pinned models score responses independently using the same rubric. Complete panels contribute their median to aggregate quality scores.
| Family | Model | Mean score, this run | Coverage |
|---|
Why three families
LLM judges systematically prefer text that reads like their own output. The effect is driven by fluency rather than literal self-recognition, so it fires even on text the judge did not write. Search APIs increasingly synthesize their answers with the same class of model, which makes a single-judge design structurally unsafe here.
The results page publishes the measured gap between the three families for exactly this reason: on this run they differ by points on the same responses. The ensemble does not remove that bias. It makes it visible and stops any one family from setting the number alone.
Median, and only when complete
The score for a response is the median of the three judges, so an extreme score has limited influence; the ensemble remains sensitive to judge choice. Responses missing any judge are dropped entirely rather than averaged over the survivors: rate-limit failures cluster in time rather than falling randomly, so mixing three-way medians and two-way means in one column would introduce bias, not just thin the sample.
Salvaging truncated replies
Judges sometimes truncate mid-sentence in the optional rationale field, after emitting complete numeric scores. Discarding those replies wholesale would shrink the sample non-randomly, because longer vendor payloads truncate more often. The parser recovers the scores field-wise and drops only the rationale.
Scoring rubric
read from the exportJudge disagreement
The mean score spread is points. The distribution below distinguishes typical variation from larger disagreements on individual responses.
Variation by category
Where the ensemble is split, the ordering within that category is carrying less information than the number of decimal places suggests, and the honest response is to discount that category — not to discount the whole run, and not to hide the figure.
Pairwise family differences
The widest pair on this run is of mean absolute difference. Three judges from three labs is the bias mitigation; it is not a claim that the three agree. The full pairwise matrix is in the export.
Spread by category
mean max−min, this run| Category | Mean spread | Split > 2 pts | n |
|---|
Human calibration
One blinded human labeller compared response pairs. This measures agreement with that person on that sample; agreement between models alone does not establish correctness.
. Sample details and limits follow below.
Measured agreement
Limits of the result
How the labeller was blinded
The set was drawn and its id committed before any of it was labelled, and the labels
themselves are in the repository — labels/ — so the figure above can be
recomputed rather than taken on trust. Pairwise, not absolute: a person asked to put a
number on a search result is unreliable at it, while a person asked which of two is
better is not, and the earlier absolute attempt is in the history for anyone who wants
to see it fail.
Withheld workload
A separately registered workload provides a public-versus-withheld diagnostic. Question texts are withheld while active; its commitment, scores, and observed coverage are published.
Commitment and retirement
A secret question set with a secret score would deserve exactly as much trust as a vendor's own benchmark. Three things make this one checkable instead. Its SHA-256 is committed to the repository before it first runs, so the questions cannot be swapped after a result is seen. It retires on a fixed rotation — every weeks — and is published in full at that point, hash included. And only the text is withheld: every withheld response's scores, latency and cost are in the export already, flagged held_out=1.
What it measures
Not a second leaderboard. The withheld set is a fraction of the size of the public one, so its absolute scores are too noisy to rank anyone by. What it measures is the gap within one vendor: the same API, in the same run, on the same day, through the same judges, scored on published questions against questions that were never published. Both sides are averaged per category first. A gap can reflect differences in question difficulty or optimisation against the public workload; it does not establish overfitting.
What it cannot do
A withheld question is not invisible to the vendor being asked it — vendors receive every query this benchmark sends and can see them in their own logs. What this defeats is cheap optimisation against a published list. What it does not defeat is a vendor that deliberately identifies benchmark traffic and treats it differently; that would need traffic mixing, which is not implemented. Stated here rather than left for someone else to point out.
The current commitment
read from the export- Questions
- , across all six categories
- Committed
- , before first use
- SHA-256
- Status
Public minus withheld
positive = better on the published set| Vendor | Public | Withheld | Gap |
|---|
Aggregation and eligibility
Response medians become category means, then equally weighted vendor scores. Coverage rules can leave a score unpublished.
-
Response score
Median of the three judges' overall scores. Dropped if any judge is missing.
-
Category cell
Mean of the response scores for that vendor in that category. A cell whose coverage falls below of its queries is published as missing rather than as a number computed from a thin sample.
-
Vendor score
Equal-weight mean of every required category cell. If any category is withheld, the overall score is withheld too; partial category averages are not ranked. Fully judged sample sizes and uncertainty remain visible beside publishable scores.
-
One run per cycle is published
A cycle can contain smoke tests and aborted runs. The published run is chosen explicitly: it must cover at least queries in every category, and among those that qualify the one with the most complete judge ensembles wins. Every candidate run and its completeness ships in the export.
Known limitations
Sampling, calibration, and measurement limits to consider when using these results. Report an issue.
- Short track record
- of data.
- Scores cluster high
- Most results land between 7 and 10. Either these APIs are all good at these questions, or the questions are too easy to discriminate at the top. Harder queries are the next change to the set, and it will move the numbers.
- Positional bias is untested
- The standard mitigation is to run every comparison in both orders and count a preference only when both agree. This benchmark scores each response independently rather than comparing pairs, so positional bias does not arise in the current design. If any pairwise sub-metric is added later, that mitigation has to come with it.
- Single geography, single run
- All queries were issued from one location at one time. Search results vary by region and by hour, and this benchmark does not yet measure either.
- Not a legal review
- The vendor scoping described above is based on reading published terms of service, not on advice from counsel. It reflects a deliberately cautious reading, which is not the same thing as clearance.
Funding and disclosures
Vendor relationships, proposed commercial incentives, and the correction policy.
Proposed commercial incentives
The plan is to sell three things: a live score feed, private evaluations run against a customer's own queries, and a routing client. One proposed pricing tier charges a share of the spend a customer saves by routing — which would mean earning more when traffic moves to cheaper vendors, while also publishing the ranking that says which vendors are good enough to move to. That is a genuine conflict. It is disclosed here rather than denied, the flat per-decision tier that pays the same whichever vendor is chosen is the default, and the conflict will be stated on the pricing page itself.
Vendor relationships
No vendor pays anything, for any reason — no sponsorship, no placement, no listing fee, no advertising. No equity, employment, advisory role or consulting with any benchmarked vendor or its parent. No vendor sees a score before it is published: no embargo, no preview, no advance right of reply. No vendor influences the query set, the rubric or the judge models. Recorded pricing tiers are published in pricing.json.
Corrections and disputes
Publicly, and specifically. Every individual judge score is already in the export, so a dispute can name a query and a judge rather than object to a total. Configuration disputes — wrong endpoint, wrong tier, wrong parameters — are the most likely kind and the fastest to fix; the exact request made to each vendor is in the source. A correction that moves a published number gets a dated changelog entry, never a silent edit. No NDAs, and no private resolution of a public dispute.
Signals to scrutinise
each one externally checkable- The cheap vendors' advantage grows over time in a way that tracks the introduction of savings-share pricing.
- The methodology changes shortly after a vendor relationship changes.
- The withheld set stops being published on retirement, or its retirement is postponed indefinitely.
- The disagreement figures above disappear, or stop being broken out by category.
- A vendor leaves the benchmark without a stated reason.
- The disclosures stay empty after the commercial products start earning.
Reproduce the study
Live collection requires your own API keys. The public workload contains queries. The selected run records in vendor costs, including any withheld queries; costs may be reported or estimated. Judge costs are separate.
git clone && cd vannaris make setup cp .env.example .env # add your keys .venv/bin/python scripts/check_keys.py # verify all eight, one cheap call each .venv/bin/python -m src.runner --queries src/queries/full-v1.json .venv/bin/python -m src.export # rebuild site/data and site/export
Published aggregates can also be recomputed from the committed CSVs without API keys. Recomputation commands.
Methodology changelog
Protocol changes and reporting corrections, with original measurement dates preserved.
- 2026-10-04 · publication analytics correction
- Derived analytics are recomputed from preserved public observations. An overall score now requires every category to clear its publication floor; incomplete vendors remain inspectable but are not ranked. Uncertainty uses category-stratified paired query bootstrap intervals. Separation uses paired sign-flip tests with Holm correction across vendor comparisons and requires an interval excluding zero. These tests assume exchangeable paired score signs under the null; unresolved differences do not establish equivalence. Disagreement statistics now use complete judge ensembles, with partial panels reported separately. Original retrieval dates, run identities and source observations are preserved: this is a reporting correction, not a new API evaluation. Historical provenance remains explicitly partial where the original execution metadata is unavailable.
- 2026-08-04 · methodology v2 — scores from here are not comparable with the 31 Jul 2026 run
-
The largest change so far, and the reason it is one entry rather than five: everything in it alters what a judge sees or how it answers, so week-over-week comparison across this line is not valid. It is made now, against a single published week, because that is the cheapest moment it will ever be.
- The judges are told today's date. The rubric never stated it, and one judge family was reading post-cutoff results as fabricated — marking vendors down for returning current information, in the column that exists to reward exactly that. It showed up as a 2.4-point family gap on breaking news that is absent everywhere else.
- Sampling is pinned. Judges now run at temperature 0, with a fixed seed where the provider offers one. Previously each score was a temperature-1 sample: on responses whose vendor payload was byte-identical, the ensemble median reproduced only about 70% of the time. Pinning a model version bought nothing without this.
- The snippet cap no longer binds below what adapters request. It was 400 characters against 500 requested, so it discarded about a fifth of two vendors' text and none of anyone else's.
- The response shape is no longer named in the prompt. It was announced for the vendors returning prose and not for the others, identifying one vendor to an otherwise blind judge.
- One vendor's output cap was raised from 512 to 2048 tokens. The harness imposed it, the vendor did not, and it truncated 10 of 150 answers.
The 31 Jul 2026 table stands as published. It is not being rescored — a corrected week is published beside the original, never over it.
- 2026-08-04 · reporting corrections, scores unchanged
- Three published figures were wrong or unsupported and none of them were scoring changes. The queries-won column resolved ties by internal ordering, so it credited vendors with wins they had not had; it is replaced by outright wins and shared-best, with the tie rate published beside them. Two pages asserted that the ranking survives the choice of judge, which this run's own per-judge data contradicts; that is replaced by the measured depth of agreement. The cost ratio was quoted at tiers that were not like for like; both bases are now published with a source per vendor.
- 2026-08-04 · withheld set added
- A rotating set of questions now runs alongside the public one without being published. Its hash is committed before it runs and its questions are published when it retires (above). Scoring is unchanged: no published cell includes a withheld question, so every number on this site is still recomputable from the published query set. The first set is registered and has not run yet.
- 2026-08-04 · disagreement published as rates
- Judge disagreement was published as a single mean. It is now published as the share of responses over each threshold, per category, with the pairwise family differences — the mean cannot distinguish broad mild disagreement from occasional severe disagreement, and on this run the difference is large. No scores changed; the reporting did.
- 2026-08-04 · coverage threshold displayed correctly
- This page and the data page stated the cell-coverage publication threshold as 1%. The threshold enforced in code was, and remains, ; a formatter was rendering a 0–1 share as though it were already a percentage. Display only — no published score was affected.
- 2026-07-31 · initial
- First published run. Three-judge cross-family ensemble, median aggregation, top-10 normalisation, -query set ().
- 2026-07-31 · judge pin reverted
- A newer Google judge model was trialled and reverted before publication: it truncated its JSON on roughly a third of calls under the production-length judge prompt, having passed a short probe. Judge models are now validated against a full-length prompt before pinning.
- 2026-07-31 · aggregate layer guarded
- A two-query smoke test overwrote one category's published cells, and the result was indistinguishable from a real one. Aggregation now refuses to overwrite a cell computed from more queries, records which run produced each cell, and the site recomputes from the per-judge scores of one explicitly chosen run.