PaperlayerView exact revision on arXiv

Source-linked AI summary

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

Harshita Diddee, Gregory Yauney, Swabha Swayamdipta, Daphne Ippolito

arXiv:2603.18019v2cs.CLcs.AIcs.SE

TL;DR

Benchmark descriptions may not reveal whether their items represent a practitioner’s specific use-case, and validity judgments can depend on the stakeholder’s definition of a capability. BENCHBROWSER retrieves use-case-relevant benchmark items; practitioner evaluations find the evidence relevant, and the paper uses it to identify coverage gaps and conflicting model rankings.

  • Problem

    Benchmarks vary in scope and granularity, making it difficult to determine whether their items adequately represent a practitioner’s use-case and capability of interest.

    Source: §1 · p. 1

  • Method

    BENCHBROWSER retrieves benchmark examples for natural-language use-cases and uses an LLM judge to filter obvious mismatches while leaving relevance decisions to practitioners.

    Source: §1 · p. 2, §2.3 · p. 4

  • Results

    A study with 10 NLP practitioners found retrieved examples relevant across diverse use-cases, and the evidence exposed narrow coverage and conflicting model rankings across benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Surfacing benchmark items across suites lets practitioners inspect how their use-case is represented and diagnose content and convergent validity.

    Source: §8 · p. 9

    The content-validity analysis does not verify facets with annotators, and LLM-generated facets may have biased recall, so its disparities are preliminary signals for human review.

    Source: L. Limitations · p. 30

Abstract

from arXiv · show

Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality of benchmarks: a "poetry" benchmark may never test for haikus, while "instruction-following" benchmarks will often test for an arbitrary mix of skills. This opacity makes verifying alignment with practitioner goals a laborious process, risking an illusion of competence even when models fail on untested facets of user interests. We introduce BenchBrowser, a retriever that surfaces evaluation items relevant to natural language use cases over 20 benchmark suites. Validated by a human study confirming high retrieval precision, BenchBrowser generates evidence to help practitioners diagnose low content validity (narrow coverage of a capability's facets) and low convergent validity (lack of stable rankings when measuring the same capability). BenchBrowser, thus, helps quantify a critical gap between practitioner intent and what benchmarks actually test.

1. Introduction

Benchmarks vary in scope and granularity, so their scores may not represent a practitioner’s specific use-case. BENCHBROWSER retrieves relevant benchmark items to help practitioners assess validity, and the paper uses this evidence to identify coverage and ranking-stability risks.

1. Introduction

  • Benchmark descriptions and contents vary: examples may cover different capabilities or repeatedly test one targeted capability, complicating validity assessment.Content validity concerns whether test items represent the capability’s full domain; convergent validity concerns whether benchmarks measuring the same capability produce consistent model rankings.Source: §1 · p. 1
  • Manual auditing and keyword searches do not scale to modern suites, while static documentation cannot capture practitioners’ changing, stakeholder-dependent definitions of capabilities.Source: §1 · p. 2
  • BENCHBROWSER retrieves semantically related examples across benchmark suites, enabling practitioners to inspect evaluation items and interactively audit what affects their use-case evaluation.The paper frames this as an alternative to relying on rigid annotation schemas; auditing is fast enough to cover tens of thousands of examples in seconds.Source: §1 · p. 2
  • A study with 10 NLP practitioners found that BENCHBROWSER returns examples they deem relevant across diverse use-cases.Source: §1 · p. 2
  • A Python-to-Go use-case change sharply reduces available examples, while relevant examples can yield rankings that diverge from established benchmarks.For Hindu-religion questions, models ranked highest on BIG-bench Hindu Knowledge can fall nearly to the bottom on HellaSwag prompt-completion examples.Source: §1 · p. 2

2. BENCHBROWSER

BENCHBROWSER retrieves top-k benchmark examples for a natural-language use-case through anchor generation, semantic retrieval, and filtering. Its anchor strategies address abstract intents and missing format cues, while the final judge removes only obviously unrelated items.

2. BENCHBROWSER

  • BENCHBROWSER retrieves the top-k most relevant examples from a database spanning many benchmarks through anchor generation, retrieval, and filtering.Source: §2 · pp. 2–3

2.1. Stage 1: Anchor Generation

  • Because use-cases can be abstract and omit example-format cues, anchor generation tests rewrites that better match benchmark content.Rephrasing expands a topic into three subtopics or a capability into three skills; example synthesis creates three diverse-format examples, while shorthand uses skill and topical cues in a bounded vocabulary.Source: §2.1 · p. 3

2.2. Stage 2: Semantic Retrieval

  • Semantic retrieval embeds anchors with baai/bge-en-icl, uses Faiss similarity search to retrieve nearest examples, and compares results with BM25 and random retrieval.Source: §2.2 · p. 3

2.3. Stage 3: Filtering

  • An LLM judge filters for topical and skill overlap and whether an example provides credible evidence of use-case competence, removing obvious mismatches rather than deciding relevance for practitioners.Source: §2.3 · pp. 3–4

3. Evaluation Setup

BENCHBROWSER is evaluated on use-case queries spanning broad topics, cross-domain skills, concrete applications, novel practitioner queries, and cases with existing validation sets. The setup assesses retrieval relevance, ranking quality, and, for topic queries, recall.

3. Evaluation Setup

  • The evaluation includes 58 topic queries drawn from BBH and MMLU to represent broad knowledge domains.Source: §3 · p. 4
  • Skill queries cover capabilities across domains and are synthesized from Clio use-cases, while application queries represent concrete deployed tasks and are also Clio-derived.Source: §3 · p. 4
  • A novel-query set crowdsources 20 use-cases from nine NLP practitioners, including creating Matplotlib visualizations.Source: §3 · p. 4
  • Known-validation queries are paired with existing use-case-specific test sets, such as NutriBench for calorie-intake estimation, to test whether retrieved items predict ground-truth performance.Source: §3 · p. 4
  • The evaluation reports anchor and system precision, NDCG@k for final-set ranking quality, and Recall@k for topic queries.Topic recall uses a restricted 28,000-example database; other metrics use a 70,000-example database.Source: §3 · p. 4

4. BENCHBROWSER Finds Relevant Examples

Rewriting-based retrieval anchors outperform direct and non-neural baselines on topics and skills, while Example Synthesis is consistently among the strongest methods. Human evaluation broadly supports the relevance judgments, with some disagreement and weaker alignment on particular use-case sets.

4. BENCHBROWSER Finds Relevant Examples

  • Rewriting-based anchors outperform non-neural baselines and the Original baseline by more than 10 points in Prec anchor@k and Prec sys@k on topics and skills.Example Synthesis is consistently best or second-best across all three sets, and Shorthand performs comparably.Source: §4 · p. 4
  • Example Synthesis performs best in tie-breaking tests of ranking preservation and overlap, so the authors use it for subsequent experiments.Shorthand and Example Synthesis achieve the highest precision across diverse use-case sets.Source: §4 · p. 5
  • Precision remains largely stable as k grows for topics and skills, while applications show sharper degradation, potentially reflecting fewer relevant examples for contextual queries.All methods have the least precision on applications, potentially due to query context.Source: §4 · p. 5, Table 3 · p. 5
  • Warm-started end-to-end latency averaged 5.28 ± 1.48 seconds across 210 public-site requests, with no failures.The system supports interactive audits of practitioner use-cases.Source: §4 · p. 5

4.1. Human Evaluation of BENCHBROWSER

  • Automatic and human mean relevance closely match within 0.1 for topics, skills, and applications, but rank agreement across these sets is weak to moderate (ρ ∈[0.15, 0.50]).The known validation set is the main outlier, with a mean gap of 0.30 driven by a few broad queries targeting esoteric suites.Source: §4.1 · pp. 5–6
  • Human annotator agreement ranges from slight to moderate, with disagreements mainly between Relevant and Partially Relevant rather than Relevant and Irrelevant.This pattern suggests calibration differences rather than fundamental disagreement.Source: §4.1 · p. 6

5. Measuring Content Validity

The section assesses content validity by comparing retrieval coverage across variants of a capability while holding the broader skill fixed. Closely related variants often yield sharply different coverage, exposing underrepresented facets.

5. Measuring Content Validity

  • BENCHBROWSER makes facet-level coverage inspectable by retrieving examples for use-cases that vary one facet of a broader capability.The study constructs 20 capabilities with six variants each, differing along one interpretable axis, for 120 total use-cases.Source: §5 · p. 6
  • The authors estimate coverage by the percentage of retrieved examples judged Relevant, excluding Partially Relevant examples; large differences across a skill family’s variants signal low content validity.Stakeholder-dependent relevance and overlapping examples make manual assessment expensive and burdensome.Source: §5 · p. 6
  • Closely related variants often yield very different numbers of Relevant examples, with conventional facets such as Python code and fantasy stories better represented than Go code or historical fiction.The disparity also appears under a lexical BM25 baseline, suggesting it reflects benchmark composition rather than the semantic retriever.Source: §5 · p. 6
  • Sparse facet coverage can make a strong aggregate score uninformative about untested deployment-relevant variants, and the evidence can guide benchmark construction for underexplored facets.For debugging, examples concentrated on syntax errors may say little about security vulnerabilities or memory leaks.Source: §5 · p. 6
  • Figure 3 compares the best- and worst-represented facets within each skill family, showing sharp coverage disparities across the 70k-example database.Some facets are heavily overrepresented while others are nearly absent.Source: Figure 3 · p. 7

6. Measuring Convergent Validity

Retrieved sets sometimes preserve gold-set model rankings and sometimes diverge substantially, including inversions, even when examples are relevant to the use-case. The section uses rank divergence to diagnose convergent-validity risks and inspects retrieved items for possible causes.

6. Measuring Convergent Validity

  • The study compares rankings from each use-case’s gold set and retrieved set across eight models, averaging Kendall’s τ over 50 Monte Carlo trials.The gold baseline uses subsets of the gold set to account for the lower achievable correlation with smaller sets.Source: §6 · p. 7
  • Rank divergence is Δ(u) = τgold(u) − τret(u): small values indicate ranking preservation, while large values indicate divergence beyond finite-sample effects.τret compares retrieved-set rankings with matched-cardinality gold subsets; τgold measures the gold-set ranking correlation against smaller gold subsets.Source: §6 · p. 7
  • Rank divergence varies sharply across use-cases: some retrieved sets preserve benchmark-induced comparisons, while others show large divergence or ranking inversions.The analysis is restricted to use-cases whose retrieved examples meet the human-relevance threshold.Source: §6 · p. 8
  • Format or scoring-rubric changes can alter what success means for a nominal capability, creating measurement mismatch and potentially different model rankings.The text notes that win-rate is sensitive to calibration inconsistencies, which can propagate into rank differences.Source: §6 · p. 8
  • Retrieved examples may concentrate on a source benchmark with narrower or differently scoped operationalizations, so benchmarks labeled as the same capability can measure non-equivalent constructs.A shift from definitional recall to applying a concept in context can also redefine the construct under evaluation.Source: §6 · p. 8
  • Inspecting retrieved examples and their source benchmarks can help practitioners assess whether ranking instability reflects construct non-equivalence or divergent operationalization.Manual inspection identifies three recurring drivers of ranking inversions and two practical ways low convergent validity may arise.Source: §6 · p. 8

7. Related Work

Related work addresses customizable benchmark construction and validity-oriented evaluation infrastructure. BENCHBROWSER differs by retrieving existing vetted items for free-form practitioner intent rather than relying on synthetic labels or predefined schemas.

7. Related Work

  • Customizable benchmarking includes synthetic test-set generation from seed corpora or zero-shot, and restructuring existing evaluations for targeted selection.Examples include capability metadescriptions, retrieval of difficult items, and repositories organized by fixed taxonomies.Source: §7 · p. 8
  • BENCHBROWSER retrieves and organizes vetted, labeled benchmark items for free-form intent without synthetic labels or predefined schemas.This preserves the manual supervision applied during original test-set curation.Source: §7 · pp. 8–9
  • Validity-oriented evaluation work uses construct-sensitive tools and costly evidence collection, but practitioners lack scalable support for inspecting item-level evidence for missing coverage and conflicting operationalizations.BENCHBROWSER’s retrieval-based interface addresses this gap.Source: §7 · p. 9

8. Discussion and Conclusion

BENCHBROWSER makes benchmark validity auditable under practitioner-defined use-cases by surfacing evaluation items and differences in how capabilities are measured. This evidence supports diagnosing underrepresented facets and ranking inversions, and can guide where new benchmarks or slices should focus.

8. Discussion and Conclusion

  • BENCHBROWSER surfaces items, sources, rubrics, and framing differences so practitioners can audit what benchmarks measure and diagnose low content and convergent validity.Human evaluations, consistent with LLM-judge results, show it retrieves relevant evidence for assessing coverage and representation.Source: §8 · p. 9
  • Its cross-benchmark view exposes which subsets, outliers, or framings drive ranking inversions and where capability facets are underrepresented.This supports more stable, well-contextualized conclusions and guides where new benchmarks or existing slices should focus.Source: §8 · p. 9

Impact Statement

The paper presents a fast, low-cost workflow for auditing benchmark validity against user-specified intent. By exposing measurement choices that can produce conflicting rankings, it supports more cautious interpretation of competence claims and more trustworthy benchmarking.

Impact Statement

  • The workflow takes a first step toward practitioner-centered validation by auditing benchmark validity under user-specified intent.When user-centric capabilities drive utility, practitioners need access to the evaluation items and rubrics behind system-selection decisions.Source: Impact Statement · p. 9
  • A fast, low-cost, end-to-end workflow exposes rubrics, framing, and construct scope that can produce conflicting rankings, helping practitioners interpret competence claims more cautiously.The work contributes a core ingredient for trustworthy benchmarking by making capability claims auditable.Source: Impact Statement · p. 9

A. Motivation Experiment Details for Figure 1

This section describes how BenchBrowser retrieves benchmark examples for practitioner use cases, including the prompts, use-case sets, evaluation criteria, and retriever configurations used to assess relevance. It reports comparisons with gold-set model performance and distinguishes practical usefulness from broad relatedness.

A. Motivation Experiment Details for Figure 1

  • The motivation experiment evaluates seven instruction-tuned models.The passage names Qwen2.5-7B-Instruct, Ministral-8B-Instruct-2410, Llama-3.1-Tulu-3-8B, c4ai-command-r7b-12-2024, DeepSeek-R1-Distill-Llama-8B, Mistral-7B-Instruct-v0.2, and Meta-Llama-3-8B-Instruct.Source: A. Motivation Experiment Details for… · p. 14

B.1. Shorthand Anchor

  • Because benchmarks often lack datapoint-level skill annotations, BenchBrowser represents each example as a shorthand of its primary skill plus up to three searchable attributes.The format is ⟨skill⟩& ⟨key1⟩& ⟨key2⟩& ⟨key3⟩.Source: §B.1 · p. 14

B.2. Anchor Generation Prompts

  • The refinement prompt generates three brief, non-nested query refinements that operationalize different aspects of an intent and include at least one broad-domain refinement.The prompt adapts refinements to broad capabilities, topics, tasks, scenarios, proper nouns, or questions.Source: §B.2 · pp. 15–16

C.1. In Context Examples for the Embedding Models (used during Original, Example Synthesis and Rephrasing Index Creation)

  • The in-context examples span claim verification, blogging, unit testing, stargazing, and resume writing, under a shared instruction to retrieve records with matching concepts and topics.The examples are used for embedding models during index creation.Source: §C.1 · p. 16

C.2. Comparing Embedding-Models and Retriever Configurations for the Retriever Backbone

  • bge-en-icl achieves Prec sys@k =0.694, Recall@k (0.218), and the highest NDCG@k; its 1.04s latency is lower than the reranker’s 14.03s.The reranker has Prec sys@k =0.664 and recall=0.176, while llm-embedder has Prec sys@k =0.488 and Recall@k =0.012; bge-en-icl is adopted as the default backbone.Source: §C.2 · p. 17, Table 9 · p. 17

D. Filtering selection prompt

  • BenchBrowser’s relevance prompt scores a test case as relevant, possibly relevant, or not relevant based on knowledge overlap, skill transfer, and practical utility.The prompt maps these judgments to 1, 0, and -1, respectively.Source: D. Filtering selection prompt · p. 18

E.2. Evaluation Categories

  • Evaluation use cases cover focused topics, practitioner-facing skills, practical applications, and 21 novel scenarios elicited from NLP practitioners.The topic set uses 58 tasks and subjects from BIG-bench Hard and MMLU; the skills and applications sets each contain 50 use cases, with retrieval over an index of approximately 67k examples from 20 benchmarks.Source: §E.2 · pp. 18–19

E.3. Prompts for Generating Usecases

  • The skill-use-case prompt derives 50 broad, transferable skills from high-level categories found in a random sample of 1M Claude.ai conversations.It asks for atomic skills applicable across multiple use-case categories.Source: §E.3 · p. 19
  • For the applications set, Claude-3.4-Sonnet was asked to generate applications based on ground-truth categories from an evaluation dataset of 19,476 synthetic chat transcripts, avoiding applications that combine too many tasks.The skills should be testable or benchmarkable and granular enough to be distinctive.Source: §E.3 · pp. 19–20

E.4. known validation set use-cases and their corresponding Gold Test Sets

  • The known validation set is specified through use-cases and corresponding gold datasets, with datasets prefixed “bb” included in Big Bench.Source: Table 11 · p. 21, §E.4 · p. 21

E.5. Evaluation Metrics

  • Evaluation metrics include ranking precision before and after relatedness filtering, intersection@k, recall@k, model-ranking correlation, and NDCG@k.NDCG@k accounts for relevance and ranking position, while model-ranking correlation compares rankings on gold and retrieved sets.Source: §E.5 · p. 21

E.6. Evaluation Prompts

  • Prec sys@k assesses whether retrieved examples are useful for a practitioner’s specific intent, and annotators corroborate this distinction from broader relatedness.The authors use Prec sys@k for most comparisons because it reflects the practically useful metric practitioners would expect to maximize.Source: §E.6 · p. 22

F.1. Tie-breaking: Computing Correlation between Model Performances on Retrieved Set and Gold Test Set for topics set

  • Retrieved-set and gold-set model performance show high correlations, highest for the Example Synthesis anchor method.The comparison uses eight models on 58 BBH and MMLU test sets, with bottom-k and random benchmark slices as correlation baselines.Source: §F.1 · pp. 22–23, Figure 5 · p. 23

F.2. Tie-breaking: Estimating Overlap with Approximated Gold Set for applications set and skills set

  • Testcase retrieval consistently has higher intersection@k across all three use-case sets, with comparisons beyond Kcommon treated cautiously because they reflect fewer than 90% of queries.The method estimates an approximated gold set from the union of relevant examples returned at k = 80; Figure 6 marks the common area and the lower-support tail.Source: §F.2 · p. 23

F.3. Precision of BENCHBROWSER as K is increased

Example Synthesis maintains relatively stable precision as retrieval depth grows for topics and skills, while applications are harder; judge comparisons support stable aggregate retrieval ordering, and the demo reports latency over 210 requests.

F.3. Precision of BENCHBROWSER as K is increased

  • For topics and skills, Example Synthesis precision remains relatively stable as k grows, while applications precision declines more sharply.Applications are lower across methods, reflecting the challenge of retrieving examples for highly contextual use-cases; Original becomes the most competitive method in this set.Source: §F.3 · pp. 23–24

G. Automatic Judge Calibration with Another Automatic Judge

  • Both automatic judges rank items similarly by relevance, although GPT-5-mini assigns more RELEVANT labels (≈57%) than Gemini-2.5-flash-lite (≈49%).Their Spearman correlations are consistently positive and statistically significant, supporting stable aggregate retrieval comparisons under judge substitution.Source: G. Automatic Judge Calibration with… · p. 25

H.1. Hosting Infrastructure

  • The service runs on Cloud Run with 4 GiB RAM and 1 vCPU, and requests may incur cold-start overhead because its minimum instance count is zero.Embeddings are outsourced to DeepInfra, and reported end-to-end latency includes that external API call.Source: §H.1 · p. 25

H.2. Latency Profiling

  • The demo’s mean end-to-end latency was 5.28 ± 1.48 s across 210 requests, with p99 latency of 8.95 s and no failures.Testcase generation took 1.67 s (31.7%), embedding generation 0.99 s (18.8%), sample scoring 0.59 s (11.2%), and 2.02 s (38%) was unattributed overhead.Source: §H.2 · pp. 25–26, Figure 9 · p. 26

I. Human Evaluation of BENCHBROWSER

  • The human evaluation involved 10 annotators who were adult, fluent English speakers in the United States with prior NLP-project experience.Participation took approximately one hour, with compensation of a DoorDash food order up to $20 USD.Source: §I · p. 26

I.1. Detailed Analysis of Human–Automatic Disagreement

The disagreement analyses locate most labeling differences at the boundary between relevant and partially relevant, while a lexical BM25 check reproduces content-validity trends. For convergent validity, retrieval-induced ranking divergence often exceeds the instability measured within gold sets.

I.1. Detailed Analysis of Human–Automatic Disagreement

  • Most human–automatic and human–human disagreements occur between Relevant and Partially Relevant labels, rather than between Relevant and Irrelevant.Annotators can give plausible but different interpretations of relevance, such as whether sentiment classification suffices for detecting emotional support.Source: §I.1 · p. 26, Table 13 · p. 27

J. Additional Details for Content Validity

  • A lexical BM25 baseline reproduces the qualitative content-validity trends, indicating that the observed variation in relevant-example counts is not an artifact of semantic retrieval.The results support the conclusion that some use-case facets are underserved relative to dominant facets.Source: J. Additional Details for Content… · pp. 27–28, Figure 11 · p. 28

J.1. Example of Use Cases showing Spread in Relevant Count across Variations

  • Across use-cases, relevant-example counts can vary substantially between facets, with some variants well represented and others barely represented.The examples span facets such as domain, context, platform, style, output format, and programming language.Source: §J.1 · pp. 28–29

K. Additional Details for Convergent Validity

  • For convergent-validity evaluation, the authors use benchmark-specific performance metrics and select a high-precision prompt per use-case for human annotation.The reported model performance estimates aggregate per-sample scores; the metric differs by benchmark, including accuracy, win-rate, and custom protocols.Source: K. Additional Details for Convergent… · p. 29, Table 14 · p. 30

K.1. Correcting Kendall’s Tau

  • The ranking comparison uses retrieved–gold rankings at matched evaluation budgets and gold-set controls to measure ranking stability under sampling.The method compares τret against τgold, which contrasts full-gold and subset rankings, and τsanity, which compares independent gold subsets.Source: §K.1 · p. 30
  • Across tasks, τret is often much lower than τgold and τsanity, sometimes near zero or negative, despite relatively stable rankings within gold sets at the retrieved-set size.The authors posit divergent operationalizations and rubric mismatches as explanations for the low convergent validity.Source: §K.1 · p. 30

L. Limitations

The evidence has bounded generalizability: human evaluation covers a limited set of use-cases and annotators, facet analysis relies on unverified LLM-generated facets, and diagnoses are confined to the indexed items and feasible model evaluations.

L. Limitations

  • The human evaluation’s scale and diversity are limited, so its precision and coverage estimates may not generalize beyond the evaluated use-cases.The study collected 4,200 relevance judgments but constrained use-case and annotator diversity to manage annotation cost.Source: L. Limitations · p. 30
  • Content-validity facets are LLM-generated and not elicited or verified by annotators, so the analysis is intended to flag disparities for inspection, not prescribe equitable facet coverage.Actionable claims require human-in-the-loop facet elicitation and relevance review of retrieved examples.Source: L. Limitations · p. 30
  • Diagnoses are bounded by indexed benchmark items and evaluated models, because expanding suite and model coverage is resource-intensive.Some metrics require commercial-model inferences, further constraining the feasible set of evaluated models.Source: L. Limitations · p. 30
Loading 2603.18019v2…