Data and downloads
Download judge scores, response metadata, and aggregate results for each published snapshot. Public data is licensed CC BY 4.0.
Coverage and publication notes
Download files
Export generated . Original retrieval dates remain attached to each snapshot.
| File | Rows | Contents |
|---|
Publication policy
Exports contain derived evidence. Raw retrieved content is retained locally and excluded from publication.
Publishing derived scores avoids redistributing retrieved page content and helps respect the storage and redistribution restrictions in vendor terms.
These exports support aggregate recomputation, response counts, and judge-disagreement analysis. They cannot replay an individual judgement against the result list the judge saw. Export guards reject fields containing vendor content.
Column reference
Key fields in the judge, response, and aggregate CSV exports.
judge-scores.csv
| Column | Meaning |
|---|---|
| week | Which run this row belongs to. |
| query_id | Joins to queries.csv. |
| category | One of the six taxonomy buckets. |
| vendor | The search API that produced the response being scored. |
| judge_family | anthropic, openai or google. |
| judge_model | Exact pinned model string. A change here is a methodology change. |
| relevance | 0–10. Do the results address the query? |
| freshness | 0–10. Current enough for this specific query? |
| citation_quality | 0–10. Authoritative sources, or content farms? |
| overall | 0–10. The holistic score. This is the one that aggregates. |
| scored_chars | Size of the vendor payload the judge saw. Published so verbosity outliers are visible. |
| prompt_tokens | Judge input tokens, for cost auditing. |
| output_tokens | Judge output tokens. |
responses.csv
| Column | Meaning |
|---|---|
| response_mode | ranked_results, synthesized_answer or both. Recorded per response, not per vendor. |
| n_results | How many results the response carried after top-10 normalisation. |
| latency_ms | Wall-clock request-to-response time measured by the harness. |
| cost_usd | Vendor-reported where available; otherwise estimated from the recorded pricing tier. |
| cost_source | reported or estimated. Identifies the basis of the recorded cost. |
| complete_ensemble | 1 if all three judges scored this response. Only these contribute to aggregates. |
| median_overall | Median of the three overall scores, or blank. |
| error | Non-empty if the vendor call itself failed. |
weekly-scores.csv
| Column | Meaning |
|---|---|
| score | Mean of the response medians in this cell. Blank if coverage fell below the floor. |
| n_queries | Queries attempted in this cell. |
| n_scored | Queries with a complete three-judge ensemble. The real sample size. |
| coverage | n_scored / n_queries. Publish threshold is . |
| delta_from_best | Points behind the best vendor in the same category. |
| pct_of_best | Share of the category-leading score. This is the routing-relevant number. |
Recompute the results
The maintained recomputation command validates the published analytics from committed CSVs without API keys.
# From a clone of the repository; uses committed versioned CSV exports. .venv/bin/python scripts/recompute_publication.py --check # Write a separate recomputed publication for inspection. .venv/bin/python scripts/recompute_publication.py --out /tmp/vannaris-recomputed
The check verifies derived scores and uncertainty against the committed publication. It applies full-ensemble and category coverage floors, excludes active held-out questions from public aggregates, and uses the versioned files for each run. It needs no API keys and makes no vendor calls. Public exports support recomputing aggregates; they cannot replay the original grading because retrieved vendor content is excluded.
Licence and citation
Data is available under CC BY 4.0; the code is MIT licensed.
Data
Creative Commons Attribution 4.0. Use it commercially, republish it, build on it. The only requirement is attribution, and a link back so a reader can check the methodology and the caveats that come with these numbers.
Code
MIT. The harness, the judge pipeline and this site are all in the repository.
Corrections
A methodology error that changes a published number gets a changelog entry, not a silent edit. The changelog already contains one.