Source-linked AI summary
Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation
Koutian Wu, Junjie Zhou, Ergan Shang, Jiayu Wang, Pengqian Han, Junkai Wang, Wanghan Xu, Lin Shi
TL;DR
Benchmark researchers need a way to find evaluations, materials, and reporting conditions across fragmented sources. Benchmark Radar provides a living catalog and discovery system with shared provenance and retrieval interfaces, and its v0.11.0 release makes 1,283 source records searchable while preserving 464 unscored records with artifact links.
Problem
Benchmark researchers must search multiple resources to find benchmarks, task materials, reported uses, and the settings behind scores.
Method
Benchmark Radar combines daily discovery with a searchable source-preserving catalog, model-report mentions, score histories, shared identifiers, and web and offline retrieval.
Results
1,283 source records are searchable in v0.11.0, including 464 unscored records with artifact links.
Takeaways & Limitations
Readers can find candidate evaluations, inspect task materials and reporting choices, and follow the citations and settings behind scores.
Takeaways & Limitations
Retrieval precision, task suitability, and time saved were not evaluated, and lexical matching can miss paraphrases and renamed tasks.
Abstract
from arXiv · showhide
Benchmark researchers and developers of large language models (LLMs) and other AI systems need to find relevant evaluations, locate their benchmark datasets and code, and understand the settings behind reported scores. We present Benchmark Radar, a living database and search engine for retrieval and discovery of AI benchmarks, covering LLM evaluation, agentic and tool-use benchmarks, coding, reasoning, safety, and domain-specific evaluations. The system combines daily discovery of benchmark papers, repositories, datasets, and releases with a searchable benchmark catalog, mentions in model cards and technical reports, and score histories. It retains source identities and citations so readers can inspect candidate benchmarks and their evaluation evidence. Daily discovery draws on 37 sources: 13 direct connectors and 24 first-party research and engineering feeds. The catalog contains 1,283 source records drawn from 4 benchmark catalogs and 12,916 numeric observations on 790 records. We describe collection and retrieval, audit the full catalog, and examine benchmark saturation, adoption trends, and the limits of score comparisons. A worked example walks through a complete prior-art search, showing how to query the catalog and inspect benchmark evidence when designing a new evaluation. We release the web dashboard with a benchmark leaderboard, a Pareto frontier view of score against measured use, saturation and trend views, daily feeds, downloadable evidence, a command-line interface (CLI) for offline queries, and reproducible analysis.
1 Introduction
Benchmark evaluation is fragmented across capabilities, resources, and reporting contexts, making it difficult to find suitable tests and interpret scores. Benchmark Radar addresses this gap with a searchable, source-preserving system that combines catalog retrieval, daily discovery, model-report mentions, and documented evaluation evidence.
- Motivation: Evaluation evidence is needed to distinguish capability gains from changes in prompts, data splits, or evaluation procedures when comparing modern language models.The motivation applies beyond general-purpose chat to recommender, scientific, and social or political modeling contexts.
- Evaluation landscape: Benchmarks span broad knowledge and reasoning tests, coding and software-engineering tasks, and command-line agent evaluations with distinct capability and contamination concerns.Examples include MMLU, GPQA, Humanity’s Last Exam, SWE-bench, LiveCodeBench, Terminal-Bench, and Long-Horizon-Terminal-Bench.
- Motivation: Benchmark Radar combines catalog retrieval, daily discovery, model-report mentions, and documented evaluation settings to connect benchmarks with their materials and later use.It is designed to reduce the need to consult multiple separate resources when investigating an evaluation.
- System contribution: The system gathers benchmark artifacts, groups observations by matching identifiers, and exposes source labels, dates, mentions, and scores through a searchable index.Its collection and ranking methods are described so the outputs can be reproduced and audited.
- System contribution: Benchmark Radar preserves one source record per benchmark, including entries without scores, dates, or citations, while reviewed identity links retain separate measurements.This structure supports a living catalog without collapsing distinct source records into a single measurement.
2 Related Work and Scope
Benchmark Radar builds on existing catalogs, evaluation platforms, and documentation practices while focusing on searchable provenance across benchmark records and their reported use. Its scope complements methods that assess benchmark saturation, item quality, and score interpretation.
- Benchmark catalogs and evaluations: Existing resources provide benchmark descriptions, evaluation platforms, leaderboards, model analyses, and artifact metadata, but Benchmark Radar adds discovery histories, artifact links, and source inspection.The system uses records from LLM Stats, OpenCompass, and Artificial Analysis alongside model-report evidence.
- Documenting evaluation evidence: Benchmark Radar retains evaluation conditions, dataset characteristics, and citations when collected sources provide them, while measuring which evidence remains available for inspection.Its catalog spans benchmarks with different task definitions, including GPQA, SWE-bench, and SciBench.
- Evaluating benchmarks themselves: Prior work studies benchmark saturation, item-level variation, and reference-free judging, whereas Benchmark Radar makes benchmark records, measurements, and provenance searchable.The paper uses its command-line client to identify candidate papers and supplements retrieval with recommendations and source review.
3 System and Methods
The system separates a source-based benchmark catalog from dated discovery history, normalizes records and evidence, and serves shared retrieval interfaces for web and offline use. Its census applies explicit eligibility rules while preserving provenance and distinguishing candidate retrieval from suitability judgment.
- System overview and daily discovery: The system separates benchmark-registry and model-report records from dated discovery observations, while both contribute evidence through shared identifiers and record structures.Discovery preserves mentions, releases, and updates; catalog records provide benchmark entries and associated measurements.
- Discovery collection: Daily collection searches a 48-hour window across 13 direct connectors and 24 first-party feeds, records source errors, removes future-dated rows, and requires healthy core sources before publication.The core sources at the cutoff were arXiv, Hugging Face Hub, and GitHub Search.
- Catalog construction: A source record represents one benchmark entry from one contributing source, with normalized names, identifiers, artifacts, scores, model identities, and cited documents.Records remain in the catalog when scores, dates, or citations are absent, while reviewed identity links preserve separate observations and counts.
- Retrieval and interfaces: Lexical BM25F retrieval exposes matched and missing query words, fields, and score components through a shared query service used by the dashboard, CLI, and HTTP interfaces.The CLI can search catalog records, discovery snapshots, or both using a local, versioned dataset with checksum and provenance information.
- Scope of evaluation: Candidate retrieval precedes suitability judgment, and the paper does not evaluate retrieval accuracy or time saved through a controlled baseline.The contributor case illustrates use through local queries and source inspection rather than a measured retrieval comparison.
- Census and measurement rules: The census counts finite observations, scored models, and cited documents using explicit identifiers, while percentage-scale summaries require declared units, known score direction, and values within 0–100.Matching scales alone do not establish matching versions, prompts, tools, attempts, or evaluators.
4 Results
The results audit Benchmark Radar’s catalog, scoring coverage, taxonomy, release-year composition, and daily discovery collection. They show broad source-linked coverage while preserving distinctions between source records, facets, discovery observations, and evidence quality.
- 4.1 Catalog Coverage and Task Materials: 1,283 source records span four catalogs, with 12,916 numeric score observations on 790 records and 493 records lacking numeric scores.Scores count observations rather than distinct models.
- 4.1 Catalog Coverage and Task Materials: 464 of 493 unscored records still link to at least one paper, repository, or dataset, while the full catalog links to 475 papers, 506 repositories, and 293 datasets.Records may contain more than one link type.
- 4.1 Catalog Coverage and Task Materials: The leaderboard lets readers inspect scored records by reported score, scored-model count, and date, while the frontier marks Pareto candidates and flags unverified values or date proxies.The record view also shows model scores within one source record.
- 4.2 Benchmark Taxonomy Across the Full Catalog: 1,279 of 1,283 records receive labels across 11 top-level domains and 63 sub-domains, with interaction paradigm and modality retained as overlapping facets.The four unclassified records lacked publisher-supplied descriptive fields, categories, or modality.
- 4.2 Benchmark Taxonomy Across the Full Catalog: Among 345 agentic records, 128 are classified as Agentic & Tool Use and 117 as Coding & Software Engineering, showing why domain and interaction paradigm require separate axes.The remaining agentic records span six additional classes.
- 4.2 Benchmark Taxonomy Across the Full Catalog: 615 records have benchmark release dates and 668 do not; source-mix reweighting changes the agentic share by at most 1.1 percentage points in reported years.The separate undated column preserves records without release dates, while the source-mix check limits interpretation of year-to-year changes.
5 Worked Example: Checking Prior Art
The worked example uses Benchmark Radar to search prior work on credit assignment in agentic training under a reproducible-small-model requirement. It combines local retrieval and source inspection, while leaving relevance judgments and design comparisons to the researcher or agent.
- 5 Worked Example: Checking Prior Art: A contributor searched August work on credit assignment in agentic training, requiring small Qwen-series models for reproducible baselines.The workflow installed the Benchmark Radar client and public Skill, downloaded the corpus, and searched locally.
- 5 Worked Example: Checking Prior Art: The contributor inspected linked papers, repositories, and datasets, supplemented retrieval with web searches using different terminology, and read source evidence before assembling the comparison.The resulting comparison is retained in the worked-example materials.
- 5 Worked Example: Checking Prior Art: The workflow separates candidate retrieval from comparison: Benchmark Radar exposes candidates and evidence, while the researcher or agent judges relevance and compares evaluation designs.Appendix E retains session screenshots and an earlier manually assembled comparison.
6 Limitations and Future Work
The catalog is bounded by collection coverage, unvalidated classification, and unresolved questions about retrieval quality, benchmark comparability, and evaluation dates.
- Catalog scope: The census reflects its recorded cutoff, while collection limits, failed requests, missing identifiers, and differing snapshot dates constrain coverage.Broader coverage requires additional source collection and review of existing evidence.
- Retrieval and evaluation: Retrieval precision, task suitability, and time saved remain unevaluated, while the worked example lacks a controlled baseline and lexical matching can miss paraphrases or renamed tasks.Semantic retrieval would require reviewed relevance judgments across queries and candidate records.
- Collection coverage: The arXiv route does not backfill earlier papers when later versions appear, and the Section 2 search missed two relevant earlier studies.A coauthor identified and read the missed studies.
- Comparison validity: Saturation and score-stagnation analyses require comparable test versions and settings with dates tied to score reporting or evaluation.Recorded benchmark-release and model-announcement dates do not by themselves establish evaluation dates.
- Classification: Task-capability classification remains unvalidated, and the deterministic null extractor assigns no capability levels to 5,863 discovery-derived tracks.Completing and evaluating those labels is separate from maintaining the benchmark catalog.
7 Conclusion
Benchmark Radar unifies benchmark discovery, catalog search, model-report mentions, and score histories in one living search engine. Its v0.11.0 release makes 1,283 source records searchable while preserving unscored records with artifact links.
- Conclusion: Benchmark Radar brings daily discovery, catalog search, model-report mentions, and score histories into one living search engine.Readers can find evaluations, task materials, reporting choices, settings, and citations behind scores.
- Conclusion: 1,283 source records are searchable in v0.11.0, including 464 unscored records with artifact links.The release distinguishes benchmark records, scored models, cited documents, and numeric observations in its full-catalog analyses.
A Full-Catalog Census
The full-catalog census retains all 1,283 source records and separates benchmark records from documents, models, scores, and discovery artifacts. It shows broad documentation and artifact linkage but warns that these units are not interchangeable.
- Catalog population: 1,283 records comprise 790 scored and 493 unscored entries, with 464 unscored records linking to a paper, repository, or dataset.All records remain in the census rather than treating missing scores as zero.
- Artifact coverage: 464 unscored entries provide artifact links, and all 1,283 source records appear in the artifact-link view without date or score filters.Each mark represents one source record linked to its detail page.
- Catalog structure: The census preserves separate source records and reviewed identity links, while discovery units remain distinct from benchmark records.This separation supports inspection without collapsing separate measurements or inflating the benchmark total.
- Evidence units: The catalog contains 1,208 distinct cited documents attached to 1,278 records, while 868 model identities cover every numeric score in the 790 scored records.Model counts for unscored records are unknown.
- Evidence units: One cited page can document hundreds of scored models, so document, model, and score counts cannot substitute for one another.The GPQA Diamond examples contrast 586 scored models under one registry page with 21 observations for 19 models across 27 cited documents in model reports.
B.2 Score Scales and Comparison Eligibility
Only a small subset of scored records meets the catalog’s percentage-scale rule, and dates, documentation metadata, and evaluation settings limit direct comparison. The system therefore preserves native-scale evidence while marking eligibility and provenance explicitly.
- Score scales: 82 of 790 scored records meet the percentage-scale rule, while 708 use other or unverified scales that do not support shared percentage-headroom calculations.All 1,283 records remain accounted for before comparison.
- Date evidence: Only 615 records have benchmark release dates, and model-announcement or document-publication dates do not directly establish evaluation dates.Progress studies require actual reporting or evaluation dates and comparable settings.
- Documentation: Documentation attaches 1,208 distinct cited documents to 1,278 records, but organization identity is missing for 1,171 cited documents.The export preserves document identities and source URLs despite incomplete organization metadata.
- Score evidence: The catalog retains all 12,916 numeric observations and native-scale maxima, while a numeric maximum need not represent the best result for lower-is-better metrics.Units, score direction, tied observations, settings, and explicit missing values remain attached to records.
- Comparison eligibility: Percentage headroom is computed only for explicitly percent, higher-is-better values within 0–100; otherwise scores remain preserved but headroom is unknown.Test versions, reasoning budgets, tools, attempts, and evaluators still require checking before comparison.
- Discovery provenance: Discovery draws on 37 sources, but connector health and optional-route failures are recorded separately from the benchmark census.The snapshot records malformed Semantic Scholar data, a missing Brave Search API key, and a feed collector with no records.
E Worked Use Case: Prior-Art Check for a New Evaluation
The worked example uses Benchmark Radar to retrieve candidate prior art for a proposed evaluation, then inspects source evidence before assembling a comparison table. The workflow accelerates candidate discovery but still requires reading source papers to populate design dimensions.
- Prior-art search: The contributor used Benchmark Radar to retrieve candidates for a prior-art comparison of credit-assignment evaluations in agentic training.The search targeted work published in August 2026 and required small Qwen-series models for reproducible baselines.
- Offline retrieval: The coding agent installed the command-line client and Skill, downloaded the local corpus, and queried candidate records offline.The workflow was demonstrated in Figure 10.
- Source inspection: Retrieved records exposed paper, repository, and dataset links for deciding which sources to inspect next, but dataset availability still required follow-up checks.One record had a repository link, while another had paper and repository links; neither recorded a dataset link.
- Source inspection: The agent supplemented local queries with web search, cross-checked candidate sets, and inspected source evidence before selecting records for comparison.Figure 13 exposed query-word matches and retrieval-score components for judging one candidate's relevance.
- Comparison table: Each comparison-table row represents a related benchmark and each column a design dimension read from source papers, so retrieval supports but does not replace evidence review.The contributor reported that manual assembly required more effort than any part of the earlier project except producing the benchmark data.
F Reproducibility, Access, and Citation
The release provides versioned software, catalog-build checks, access artifacts, and citation resources for reproducing and inspecting Benchmark Radar. Its scope is bounded by the audited release inputs and discovery cutoff.
- Reproducibility: The v0.11.0 release fixes the software commit and discovery cutoff, preserving previously audited inputs while excluding later data.The dated registry inputs remain the August snapshots registered for the release.
- Reproducibility: The build normalizes registry and model-report inputs, classifies discovery tracks, packages the catalog with checksums, and validates client updates before activation.The manuscript and dated analysis files build independently of the software checkout.
- Access and citation: Benchmark Radar provides permanent locations for the archived paper, release data, source code, dashboard, discovery observations, benchmark catalog, RSS feed, and citation metadata.Table 13 summarizes the access and citation resources.
- Access and citation: The software is MIT-licensed, while the paper and original editorial content use CC BY-NC-SA 4.0 with restrictions on commercial republication and integration.Third-party source material remains under its original terms.