Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources

1 Introduction

• The framework distinguishes source users, whose content and judgments contribute to resources, from target users, whose needs and contexts results are intended to inform.Documenting where content originates does not establish which populations or settings scores represent.Source: §1 · p. 1
• Seven dimensions connect assumptions in benchmark content, metadata, documentation, and scoring procedures to the scope of evaluation claims.The dimensions cover linguistic and locale conventions, geographic visibility, socioeconomic structure, source-user context, target-user assumptions, annotator visibility, and evaluation normalization.Source: §1 · pp. 1–2
• The audits span 11 text benchmarks and five image-text resources, examining conventions and scoring, geographic visibility and socioeconomic structure, and documentation of user-relevant assumptions.The analyses connect benchmark composition, evaluation rules, and documented user contexts.Source: §1 · p. 2
• The paper contributes a reusable audit framework, systematic audits of 16 resources, and an actionable documentation checklist with reproducible artifacts.The checklist supports examining resource assumptions and interpreting the scope of evaluation claims.Source: §1 · p. 2

• Benchmark design, construction, content, and scoring procedures shape capability measurement, interpretation, and evaluation reliability (Paullada et al., 2021; Raji et al., 2021; Liu et al., 2024; Mousavi et al., 2026).Source: §2 · p. 2
• Data statements, datasheets, and BenchmarkCards provide mechanisms for documenting provenance, intended use, and evaluation conditions (Sokol et al., 2025).Source: §2 · p. 2
• User-centric evaluation research addresses real-world intents, dialects, cultural variation, and population-specific needs (Wang et al., 2024a; Faisal et al., 2024; Shwartz, 2022; Nayeem and Rafiei, 2026, 2024, 2025).This paper extends those insights to evaluation resources by measuring conventions, testing meaning-preserving alternatives, and examining target-user assumptions.Source: §2 · p. 2
• Research identifies uneven geographic coverage in visual datasets and regional performance disparities, while recent work profiles country representation in image-text data (Shankar et al., 2017; DeVries et al., 2019; Basu et al., 2026).This paper integrates geographic visibility and socioeconomic associations with provenance and intended-user contexts.Source: §2 · p. 2

• The audit makes assumptions in benchmark content, metadata, documentation, and scoring explicit, clarifying what scores represent and where their interpretation is supported.A meaning-preserving answer expressed with different spelling, units, or date conventions may score differently when only another convention is recognized.Source: §3 · p. 3
• The audit distinguishes source users, who contribute content or judgments, from target users, whose needs and contexts results are intended to inform.Source users can include exam writers, crowdworkers, and web contributors; target users can include students, clinicians, and multilingual speakers.Source: §3 · p. 3
• Seven dimensions organize inspectable evidence: linguistic and locale conventions, geographic visibility, socioeconomic structure, source-user context, target-user assumptions, annotator visibility, and evaluation normalization.The taxonomy draws evidence from benchmark content, metadata, documentation, and scoring procedures.Source: §3 · p. 3, Table 1 · p. 3
• The framework is applied through audits of text conventions, image-text geographic visibility and socioeconomic structure, and documentation; paired scoring tests examine meaning-preserving answer variants.Figure 1 compares American-centered and other user-valid convention signals across 11 text benchmarks, with aggregate shares shown at right.Source: §3 · pp. 3–4, Figure 1 · p. 4

4 RQ1: Do Text Benchmarks Encode an Implicit Default User?

• The audit tests convention signals in benchmark questions, passages, instructions, options, and gold answers, classifying items as American-centered, other-convention, mixed, or unmatched.It reports coverage and American-centered share, and compares results with spelling matches excluded.Source: §4 · p. 4

4.1 Results and Analysis

• 62.4%–94.4% of matched convention signals are American-centered across the 11 benchmarks, with a majority in every benchmark.The range is 62.4% in MedQA to 94.4% in TruthfulQA; Figure 1 compares convention categories and aggregate shares.Source: §4.1 · p. 4
• 5.97% of WinoGrande items and 7.59% of TruthfulQA items contain audited convention signals, compared with 65.85% in MedQA and 78.08% in DROP.Coverage measures the share of items with a signal, while American-centered share measures the distribution among matched signals; Table 2 also reports spelling-excluded results.Source: §4.1 · pp. 4–5, Table 2 · p. 5
• Excluding spelling, American-centered signals remain a majority in eight benchmarks, but ARC-Challenge, MedQA, and MMLU-Pro show the opposite pattern.The highest spelling-excluded shares are TruthfulQA (90.5%), GSM8K (83.3%), SVAMP (80.0%), and DROP (72.3%); the opposite-pattern shares are 22.5%, 44.3%, and 47.3%, respectively.Source: §4.1 · p. 5

4.2 Scorer Case Study: Do Valid Answer Variants Change Evaluation Scores?

• Paired scoring tests compare original reference-consistent answers with meaning-preserving spelling, written-date, or time-format variants against the same gold annotations.The analysis uses DROP’s official EM/F1 and TruthfulQA’s released BLEU/ROUGE correctness-ranking procedures, with question-clustered bootstrap confidence intervals.Source: §4.2 · p. 5
• DROP official EM falls from 100.0% to 33.3% for meaning-preserving variants, while token-level F1 falls from 100.0% to 77.2%.The EM reduction is 66.7 percentage points (95% CI [−92.3, −36.4]); the F1 reduction is 22.8 points (95% CI [−41.6, −5.2]).Source: §4.2 · p. 5, Table 3 · p. 5
• In TruthfulQA, BLEU and ROUGE-1 correctness remain at 97.0% while ROUGE-1 similarity to a correct reference falls from 100.0 to 88.9.ROUGE-1 similarity change has a 95% CI of [−13.5, −9.0]; ROUGE-2 correctness declines by 6.1 points, with a confidence interval including zero.Source: §4.2 · p. 5, Table 3 · p. 5
• Testing normalization across spelling, date, time, and measurement variants can reveal how answer conventions affect scores and inform user-centric documentation.The paper distinguishes conventions represented in benchmark content from alternatives recognized by scoring procedures.Source: §4.2 · pp. 5–6

5 RQ2: Which Geographies Are Visible in Multimodal Benchmarks?

• The audit defines geographic visibility as the distribution of country assignments among records resolved from explicit references in captions and associated textual metadata.A deterministic resolver covers country names, aliases, demonyms, capitals, and selected cities; assignments require a unique resolution.Source: §5 · p. 6

5.1 Results and Analysis

• The United States leads four resources, while Iceland leads CC12M; the top three countries account for 26.8–37.6% of resolved assignments.Europe, North America, and Oceania collectively account for 60.9–76.5%.Source: §5.1 · p. 6
• Mean Pearson r = 0.780 and Spearman ρ = 0.778 link country visibility with World Bank GDP share across resources.GDP share provides the strongest single-variable fit among socioeconomic indicators; the associations are descriptive rather than causal.Source: §5.1 · p. 7
• In a balanced WIT comparison, the U.S. share falls from 20.0% in English to 8.5% in the non-English sample, while economic concentration persists.The ten largest economies account for 53.5% of resolved assignments in the non-English sample.Source: §5.1 · p. 7
• Country references identify contexts visible in resource text but do not establish intended evaluation users.Documenting geographic coverage, languages, and source contexts makes these distinctions auditable.Source: §5.1 · p. 7

6 RQ3: Are User Assumptions Made Explicit in Documentation?

• The audit examines public documentation for all 16 resources, comprising 11 text benchmarks and five image–text collections.It assesses source-user and target-user context, annotator visibility, locale assumptions, evaluation normalization, and source/geographic metadata.Source: §6 · p. 7
• The audit scores six documentation criteria as absent, explicitly mentioned, or operationalized through inspectable data, metadata, or evaluation procedures.Two annotators independently scored each resource; the highest score supported by documented evidence was assigned.Source: §6 · pp. 7–8

6.1 Results and Analysis

• All 16 resources operationalize source-user context, but none operationalize target-user context despite all mentioning target-user or evaluation context.Documented origins include examinations, Wikipedia, Reddit, web alt-text, and CommonCrawl.Source: §6.1 · p. 8
• Only 6.2% operationalize locale assumptions, while 25.0% mention them; no resources operationalize annotator attributes as auditable metadata.Annotator information is mentioned in 56.2% of resources.Source: §6.1 · p. 8
• Evaluation normalization is mentioned in 75.0% of resources and operationalized in 68.8%, yet documented procedures may treat meaning-preserving variants differently.Source/geographic metadata is mentioned in 37.5% and operationalized in 31.2%.Source: §6.1 · p. 8

7 Discussion: Toward User-Centric Benchmark Design

• Linguistic conventions affect which answers receive credit, geographic references reveal uneven country visibility, and documentation specifies data provenance more clearly than intended evaluation users.Together, the audits connect resource design to benchmark validity.Source: §7 · p. 8
• A six-field documentation checklist translates the audit requirements into practical reporting questions about provenance, intended users, and evaluation assumptions.An extended reporting template supports implementation in Appendix D.Source: §7 · p. 8
• For benchmark users, the checklist helps assess whether scores apply to specific populations, languages, locales, and deployment settings.For creators, it supports specifying intended users, documenting scoring conventions, and identifying validation needs.Source: §7 · p. 9
• Making resource assumptions explicit strengthens score interpretation and clarifies the scope of claims across user contexts.Content, metadata, reference answers, and scoring procedures shape evaluation before models are evaluated.Source: §7 · p. 9

8 Conclusion

• The conclusion applies a user-centric audit framework to 11 text benchmarks and five image–text resources.The framework examines evaluation resources and their assumptions.Source: §8 · p. 9
• The audits reveal American-centered linguistic conventions, geographic visibility associated with national GDP share, and gaps between documented data sources and intended users.Paired scoring experiments show that meaning-preserving answer variants can receive different scores.Source: §8 · p. 9
• Making benchmark assumptions explicit is presented as essential for interpreting scores across user contexts.The conclusion ties these assumptions to how evaluation is shaped and understood.Source: §8 · p. 9

• Linguistic analysis examines observable spelling, date, time, and measurement conventions, while scoring sensitivity is tested with the official DROP scorer and reproduced TruthfulQA evaluator.The analyses characterize linguistic representation and scoring sensitivity within the audited resources.Source: Limitations · p. 9
• Geographic visibility is measured through resolved country references in captions and associated textual metadata, so shares describe textual references rather than image provenance.Socioeconomic correlations characterize associations with national economic indicators.Source: Limitations · p. 9
• The documentation audit covers publicly available benchmark papers, dataset cards, and repositories, and scores the transparency and auditability of released documentation.Its rubric assesses whether user-relevant assumptions are explicitly documented and operationalized through inspectable evidence.Source: Limitations · p. 9
• The study examines 11 text benchmarks and five image–text resources across selected linguistic, geographic, and socioeconomic dimensions.The framework is presented as a foundation for broader audits, including multilingual evaluation, accessibility, diverse user contexts, and richer annotation metadata.Source: Limitations · p. 9

• The study uses publicly available resources and aggregate measurements, respects dataset licenses and usage conditions, and treats country assignments as textual references rather than image provenance or contributor locations.Source: Ethics Statement · p. 9
• Findings describe resource-level properties and evaluation procedures without inferring individual identities or demographic characteristics.Socioeconomic relationships are reported as descriptive associations.Source: Ethics Statement · p. 9
• The framework encourages disclosure of data provenance, intended users, annotator practices, and scoring conventions to help define the scope of evaluation claims.Such disclosures enable assessment of applicability across linguistic, geographic, and user contexts.Source: Ethics Statement · p. 9
• Code and research artifacts are publicly released under stated licenses to support reproducibility and future research while respecting original-resource restrictions.Source: Ethics Statement · pp. 9–10

A.1 Benchmark Selection and Evaluation Text

• The audit treats each benchmark item as its unit of analysis and concatenates questions, passages, instructions, answer options, and gold answers to capture conventions in resource text.It covers 11 benchmarks spanning knowledge, mathematics, commonsense reasoning, reading comprehension, science, truthfulness, and medical question answering.Source: §A.1 · p. 13

A.2 Convention Inventory and Detection Rules

• The convention inventory covers spelling, date formats, time formats, and measurement units, using lexical or pattern-based detection rules for explicit surface forms.The comparison category groups other user-valid forms rather than identifying a single alternative English variety.Source: §A.2 · p. 13

A.3 Item Labels and Aggregation Metrics

• Coverage measures the proportion of items with an audited signal, while American-centered share measures the balance among matched signals; mixed items can contribute to both signal counts.The aggregate comparison combines several spelling traditions, date and time formats, and measurement conventions rather than representing one alternative English variety.Source: §A.3 · p. 15

A.5 Coverage and Spelling-Excluded Robustness

• After spelling matches are excluded, eight of 11 benchmarks retain an American-centered majority, while ARC-Challenge, MedQA, and MMLU-Pro show the opposite pattern.TruthfulQA (90.5%), GSM8K (83.3%), SVAMP (80.0%), and DROP (72.3%) retain high shares; ARC-Challenge (22.5%), MedQA (44.3%), and MMLU-Pro (47.3%) have majorities of other convention signals.Source: §A.5 · pp. 15–16

A.4 Convention-Level Distributions

• Figure 4 separates spelling, date, time, and measurement signals, showing varied convention profiles across benchmarks and prominent spelling contributions to American-centered profiles.Mathematics and reading-comprehension resources also contain date, time, and measurement signals, while some knowledge and commonsense resources are dominated by orthographic differences.Source: §A.4 · p. 15, Figure 4 · p. 16

A.6 Scorer Case Study: Construction and Method

• The paired scorer study tests meaning-preserving spelling, written-date, and time-notation variants against identical gold annotations using DROP and TruthfulQA scoring procedures.DROP contributes 12 pairs from 11 questions; TruthfulQA contributes 33 pairs from 25 questions after four semantic changes are removed.Source: §A.6 · p. 16

A.7 Scorer Case Study: Detailed Results

• On 12 eligible DROP pairs, official EM falls from 100.0% to 33.3% (−66.7 percentage points), while official F1 falls from 100.0% to 77.2% (−22.8 points).Across 33 TruthfulQA pairs, correctness remains 97.0% for BLEU and ROUGE-1 and 100.0% for ROUGE-Lsum, while maximum correct-reference BLEU similarity falls from 97.0 to 69.2 and maximum ROUGE-1 similarity from 100.0 to 88.9 on the 0–100 scale. DROP spelling variants lose an average 45.7 F1 points, while written-date variants retain full F1 even when EM decreases; these results motivate audits of equivalent-answer conventions.Source: §A.7 · p. 17, Table 9 · p. 18

B.1 Image–Text Resources

• The audit analyzes five resources and measures geographic visibility from explicit country references in captions and associated textual metadata, not photo or contributor origin.The resources are DataComp1B, WIT English, RedCaps, CC12M, and COYO-700M.Source: B Geographic Visibility Audit · p. 18, §B.1 · p. 18

B.2 Country Lexicon Construction

• The geographic lexicon contains 3,413 entries: 3,036 country-name entries, 83 demonyms, and 294 city entries.It combines country names, localized names, aliases and demonyms, and city references.Source: §B.2 · p. 18

B.3 Matching and Country Assignment

• The resolver weights explicit country names above demonyms and city references, then counts single-country and uniquely highest weighted-majority assignments while excluding ambiguous and unresolved records.Resolved coverage is reported separately from the distribution of country visibility among resolved assignments.Source: §B.3 · p. 19

B.4 Geographic Concentration

• The United States leads resolved assignments in DataComp1B (13.55%), WIT English (14.59%), RedCaps (21.18%), and COYO-700M (16.30%), while Iceland leads CC12M (10.24%).Frequently referenced countries span North America, Europe, and major Asian economies, while low-frequency lists include countries across Africa, Asia, Europe, the Americas, the Caribbean, and the Pacific.Source: §B.4 · pp. 19–20, Figure 5 · p. 20

B.5 GDP and Country Representation

• Across five resources, Spearman correlations between GDP share and visibility range from 0.735 in RedCaps to 0.823 in WIT English, with mean OLS R2 = 0.614.Count-weighted R2 is 0.748; the analysis excludes countries with populations below one million.Source: Table 12 · p. 21, §B.5 · p. 21

B.6 Socioeconomic Comparisons and Controls

• GDP share has the strongest individual fit among examined socioeconomic indicators, and adding a control raises mean OLS R2 from 0.614 to 0.617–0.630.The comparisons include population share, GDP per capita, Internet usage, and tourism arrivals.Source: §B.6 · p. 21, Table 13 · p. 22

B.7 Language-Composition Stress Test on WIT

• In WIT, the U.S. share falls from 20.0% in the English comparison to 8.5% in the balanced non-English sample.The non-English sample’s ten largest GDP-share countries account for 53.5% of resolved assignments; Pearson r = 0.752 and Spearman ρ = 0.744.Source: Table 14 · p. 22, §B.7 · p. 22
• The main WIT English audit’s U.S. share of 14.59% and the separate English comparison’s 20.0% refer to different samples and are reported independently.Reporting geographic coverage alongside language composition and collection sources makes influences on cross-resource country distributions inspectable.Source: §B.7 · p. 22

C Documentation Audit

• 99.0% exact agreement and Cohen’s κ = 0.984 were recorded before adjudication across criterion-level decisions.The sole disagreement concerned whether CC12M’s released image URLs operationalized source/geographic metadata; adjudication assigned score 2 because the URLs allow direct provenance inspection.Source: Table 15 · p. 23, C Documentation Audit · p. 23
• Documentation of annotator visibility and locale assumptions is less consistent than evaluation normalization, which is comparatively well specified, particularly for text benchmarks.Accepted alternatives and scoring behavior vary across procedures; the audit distinguishes source provenance from the specificity of documented evaluation scope.Source: C Documentation Audit · p. 23

D Example User-Centric Documentation Template

• The documentation template asks creators to report source and target users, annotator provenance, locale conventions, normalization, geographic metadata, coverage gaps, recommended uses, and auditability.It calls for documenting available evidence and assumptions needed to interpret evaluation results.Source: D Example User-Centric Documentation… · pp. 23–24

C Documentation Audit

• All 16 resources operationalize source-user context, but none operationalize the link between evaluation scores and intended user populations or settings.All mention target-user or evaluation context. The benchmark-wise audit scores are reported in Table 17.Source: C Documentation Audit · p. 23, Table 17 · p. 24

E AI Use Statement

• Generative AI tools assisted with text editing and routine code tasks; authors reviewed the assisted work and independently conducted and interpreted the research.AI-assisted code was manually verified and tested before use.Source: E AI Use Statement · p. 25