Take the data. Check the work.
Every judge score, latency and cost, licensed CC BY 4.0.
What is in the export
This run, generated , licensed CC BY 4.0.
| File | Rows | Contents |
|---|
What is deliberately not in here
The harness stores raw vendor responses for reproducibility. They are never published.
Publishing derived scores rather than republishing retrieved content reduces two exposures at once: the copyright in the pages a search API points at, and the storage and redistribution restrictions in several vendors' terms of service.
The cost is real: you cannot audit an individual judgement against the exact result list the judge saw. What you can do is recompute every published aggregate from the scores, see how many results each response contained, and see how far the three judges were apart on it. The export code fails the build if a field carrying vendor content ever reaches it.
Column dictionary
Three tables, and what every column in them means.
judge-scores.csv
| Column | Meaning |
|---|---|
| week | Which run this row belongs to. |
| query_id | Joins to queries.csv. |
| category | One of the six taxonomy buckets. |
| vendor | The search API that produced the response being scored. |
| judge_family | anthropic, openai or google. |
| judge_model | Exact pinned model string. A change here is a methodology change. |
| relevance | 0–10. Do the results address the query? |
| freshness | 0–10. Current enough for this specific query? |
| citation_quality | 0–10. Authoritative sources, or content farms? |
| overall | 0–10. The holistic score. This is the one that aggregates. |
| scored_chars | Size of the vendor payload the judge saw. Published so verbosity outliers are visible. |
| prompt_tokens | Judge input tokens, for cost auditing. |
| output_tokens | Judge output tokens. |
responses.csv
| Column | Meaning |
|---|---|
| response_mode | ranked_results, synthesized_answer or both. Recorded per response, not per vendor. |
| n_results | How many results the response carried after top-10 normalisation. |
| latency_ms | Wall-clock request-to-response time measured by the harness. |
| cost_usd | Vendor-reported where available, list price otherwise. |
| complete_ensemble | 1 if all three judges scored this response. Only these contribute to aggregates. |
| median_overall | Median of the three overall scores, or blank. |
| error | Non-empty if the vendor call itself failed. |
weekly-scores.csv
| Column | Meaning |
|---|---|
| score | Mean of the response medians in this cell. Blank if coverage fell below the floor. |
| n_queries | Queries attempted in this cell. |
| n_scored | Queries with a complete three-judge ensemble. The real sample size. |
| coverage | n_scored / n_queries. Publish threshold is . |
| delta_from_best | Points behind the best vendor in the same category. |
| pct_of_best | Share of the category-leading score. This is the routing-relevant number. |
Recompute the headline number
Two of the CSVs and a few lines of Python reproduce the cost-versus-quality finding on the front page, with no API keys and no trust required.
import csv, statistics
from collections import defaultdict
# One row per (query, vendor, judge). Keep only responses all three judges scored,
# because a partial panel is not comparable with a complete one.
panels = defaultdict(list)
for r in csv.DictReader(open("judge-scores.csv")):
panels[(r["query_id"], r["vendor"], r["category"])].append(float(r["overall"]))
cells = defaultdict(list)
for (qid, vendor, category), scores in panels.items():
if len(scores) == 3: # complete ensemble only
cells[(vendor, category)].append(statistics.median(scores))
means = {k: statistics.mean(v) for k, v in cells.items()}
best = {c: max(s for (v, c2), s in means.items() if c2 == c) for _, c in means}
for (vendor, category), score in sorted(means.items()):
print(f"{vendor:<11}{category:<16}{score:5.2f} {100 * score / best[category]:5.1f}% of best")
The output matches weekly-scores.csv to
three decimal places. Download the files from the table above (the names
on disk still include the run). Any discrepancy is a bug —
report it.
Licence and citation
Free to use commercially. Attribution is the only requirement.
Data
Creative Commons Attribution 4.0. Use it commercially, republish it, build on it. The only requirement is attribution, and a link back so a reader can check the methodology and the caveats that come with these numbers.
Code
MIT. The harness, the judge pipeline and this site are all in the repository.
Corrections
A methodology error that changes a published number gets a changelog entry, not a silent edit. The changelog already contains one.