PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting

Avijit Ghosh, Anka Reuel, Jenny Chim, Wm. Matthew Kennedy, Srishti Yadav, Jennifer Mickel, Yanan Long, Andrew Tran, Anastassia Kornilova, Damian Stachura, Kevin Klyman, Felix Friedrich, Jeba Sania, Jan Batzner, Anoop Mishra, Eliya Habba, Yixiong Hao, Nathan Heath, Shalaleh Rismani, Usman Gohar, Andrea Loehr, David Manheim, Ruchira Dhar, Sree Harsha Nelaturu, Aarush Sinha, Leshem Choshen, Drishti Sharma, Ishan Khire, Amit Saha, Subramanyam Sahoo, Michael Hardy, Michael Alexander Riegler, Kabir Manghnani, Michelle Lin, Yanan Jiang, Yilin Huang, Asaf Yehudai, Jessica Ji, Aris Hofmann, Mubashara Akhtar, Max Lamparth, Nuno Moniz, Yacine Jernite, Stella Biderman, Zeerak Talat, Sanmi Koyejo, Mykel Kochenderfer, Irene Solaiman

arXiv:2606.09809v2cs.AI

TL;DR

AI evaluation reporting is fragmented, making results difficult to interpret, compare, and trace across sources. EVALUATION CARDS builds a unified reporting layer with evidence-based fields, interpretive signals, reader-specific views, and large-scale monitoring, revealing systematic gaps in reproducibility, completeness, and multi-party reporting.

  • Problem

    AI evaluation reports use incompatible formats, omit interpretive fields, and lack a shared basis for comparing or tracing results across sources.

    Source: §1 · p. 2

  • Method

    EVALUATION CARDS composes benchmark metadata, evaluation run data, and reporting fields into a unified record with four interpretive signals and reader-specific views.

    Source: §1 · p. 2, §4.1 · p. 6

  • Results

    96.5% of (model, benchmark, metric-path) triples lack at least one minimal reproducibility field, while 98.2% of (model, benchmark) pairs are reported by only one party.

    Source: §1 · pp. 2–3, §5 · p. 9

  • Takeaways & Limitations

    The deployed instrument surfaces systematic gaps in public evaluation reporting across thousands of models, benchmarks, and reported results.

    Source: §1 · pp. 2–3

    The operationalized schema focuses on artifact-side reporting, leaving process-side decisions to linked mechanisms such as preregistration.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs. The cost is interpretive: readers cannot reliably compare results across sources, identify what a report omits, or trace an aggregate claim to its underlying evidence. Recent efforts address isolated components but leave three gaps: they cover only narrow slices of the evaluation lifecycle and do not compose into a single interpretable record; they specify static representations that do not differentiate the questions different stakeholders bring to the same evidence; and they remain proposals on paper, lacking the extraction infrastructure required for adoption at scale. We present \EvalCards{}, an operational reporting layer that composes benchmark metadata, evaluation run data, and model metadata into a unified record. We (1) derive a reporting schema from a structured review of 52 papers and 10 stakeholder interviews, (2) implement four interpretive signals (reproducibility, documentation completeness, provenance and risk, and score comparability), rendered through reader modes calibrated to research and non-research audiences, and (3) deploy a monitoring tool that applies \EvalCards{} across 5,816 models, 635 benchmarks, and 101,843 results, surfacing systematic gaps in current reporting practice.

1 Introduction

EVALUATION CARDS addresses fragmented AI evaluation reporting by unifying evidence and adding interpretive signals for comparison, provenance, reproducibility, and completeness. Its large-scale audit reveals pervasive reporting gaps and limited multi-party coverage.

Problem

  • Fragmented formats omit interpretive context, making scores difficult to compare and leaving deployment, regulatory, and scientific readers to reconstruct assumptions.The relevant context includes what was measured, under which assumptions, and how results should be interpreted.Source: §1 · p. 2

Contribution

  • EVALUATION CARDS unifies evaluation infrastructure and surfaces interpretive signals across inconsistent reporting sources.The system is intended to make evaluation evidence more reproducible, complete, traceable, and comparable for readers with different needs.Source: §1 · p. 2

System design

  • The backend ingests and standardizes sources, computes reproducibility, completeness, provenance, and comparability signals, and renders reader-specific views.Figure 1 organizes these operations into ingest, standardize, compute-signals, and output stages.Source: Figure 1 · p. 3

Empirical findings

  • 96.5% of (model, benchmark, metric-path) triples lack at least one minimal reproducibility field, while median per-benchmark documentation completeness is 10.7%.Developer self-reporting populates 0.0% of base reproducibility fields on average, compared with 16.6% for third-party reports.Source: §1 · pp. 2–3
  • 98.2% of (model, benchmark) pairs are reported by only one party, while cross-party divergence exceeds the 5% threshold in 51.9% of multi-organization metric groups.Source: §1 · p. 3

2 Related Work

Prior work covers pieces of evaluation documentation and infrastructure, but no prior artifact combines lifecycle reporting, reader-specific rendering, and continuous monitoring at scale.

Related work

  • No prior artifact joins a reporting framework, evaluation run data, and benchmark metadata while differentiating rendering by reader type and monitoring public reporting continuously.EVALUATION CARDS is designed to address this combined gap.Source: §2 · p. 3

3 The EVALUATION CARDS Framework

The EVALUATION CARDS Framework combines a permissive reporting standard with hierarchical evidence organization and an evidence-based derivation process. It covers artifact-side reporting while using partial records to support structured comparison and interpretive signals.

3 The EVALUATION CARDS Framework

  • Table 1 compares EVALUATION CARDS with prior technical reporting documentation and infrastructure.The comparison situates the framework against earlier efforts addressing parts of the reporting problem.Source: Table 1 · p. 4
  • EVALUATION CARDS links reporting documentation, per-run execution data, benchmark metadata, reader modes, and continuous monitoring in one cross-layer artifact.Partial population remains useful because the record still supports the rollout hierarchy and interpretive signals.Source: §3 · pp. 3–4

3.1 An evidence-based reporting framework

  • The framework derives from 52 papers and practitioner interviews, with 730 recommendation items independently coded at high agreement.The review retained 52 papers from 748 candidates and used a best-fit method to derive the framework structure.Source: §3.1 · p. 4
  • The framework operationalizes artifact-side items likely to appear in published reports, while process-side commitments are delegated to linked mechanisms such as preregistration.This boundary avoids recreating adoption barriers caused by requiring fields without a published trace.Source: §3.1 · p. 4
  • Table 2 presents the EVALUATION CARDS reporting framework derived from the literature review and practitioner input.Its full framework-to-field mapping is provided in Appendix D.Source: §3.1 · pp. 4–5, Table 2 · p. 5

1. Design

The design organizes evaluation reporting around goals, validity, protocols, scoring, splits, contamination, and pre-reporting considerations.

1. Design

  • The design covers goals and tested constructs, development and design preregistration, validity, task and item development, and human-subjects or ethics considerations.Source: §1 · p. 5
  • It also covers pre-run protocols, scoring and validation, splits and holdouts, pilots and baselines, contamination and gaming, awareness, and pre-reporting.Source: §2 · p. 5

3. Execution

Execution covers run logging and the reporting of run differences as part of the evaluation lifecycle.

4. Lifecycle

  • Run logging and reproducibility capture are identified as execution-stage reporting concerns.Source: §3 · p. 5
  • The execution stage also records mitigations and adaptations made during evaluation.Source: §3 · p. 5
  • Analysis and differences between runs are included in the execution-stage record.Source: §3 · p. 5

4. Lifecycle

The lifecycle framework organizes reporting choices across evaluation stages, while the rollout hierarchy preserves the structure needed to interpret each score. It replaces flat score records with paths that support drill-down and disambiguation.

4. Lifecycle

  • The lifecycle includes reporting and publication, process reporting, transparency, replication, reproducibility, data access, later use, and maintenance.Source: §4 · p. 5

3.2 A rollout hierarchy for evaluation evidence

  • Every score resolves through Family, Composite, Benchmark, Split, and Metric levels rather than a flat model–benchmark–score triple.The hierarchy covers related benchmark families, named composites, individual evaluations, benchmark subcategories, and scoring rules.Source: §3.2 · p. 5, Figure 2 · p. 5
  • The hierarchy attaches integrity signals to specific metric paths, enabling readers to trace aggregate claims to supporting metrics.Source: §3.2 · pp. 5–6
  • Different subtasks, metrics, and model or benchmark identifiers resolve to distinct paths, preventing conflation across measurements.Source: §3.2 · pp. 5–6

4 EVALUATION CARDS: Pipeline & Interpretive Layer

EVALUATION CARDS composes benchmark, run, and model metadata into standardized records, then exposes interpretive signals through reader-specific views. The system makes omissions, provenance, reproducibility, and comparability visible without imposing pass/fail judgments.

4.1 Composition over existing sources

  • EVALUATION CARDS standardizes Auto-BenchmarkCards, EEE run data, and model metadata into a consistent entity space.Stable identifiers and alias records connect variant model and benchmark names before signals are computed.Source: §4.1 · p. 6

4.2 Signals

  • The four signals address whether results can be re-executed, interpreted, attributed, and compared across setups or reporting parties.The signals are reproducibility, reporting completeness, provenance, and comparability.Source: §4.2 · p. 7
  • Missing fields lower reporting completeness and may trigger reproducibility, provenance, or comparability signals, while omissions are shown without grades or thresholds.Source: §4.2 · p. 7

4.3 Reader modes

  • Research mode foregrounds configurations, missing reproducibility fields, setup differences, and detailed metric settings for technical users.Source: §4.3 · p. 8
  • Summary mode reframes the same records in plain language, emphasizing accountability, provenance, comparability, caveats, and intended use for non-technical readers.It compresses metric configuration and presents fixed notes on what a benchmark measures, its main caveat, and intended audience.Source: §4.3 · pp. 7–8
  • Interview feedback was positive, while systematic usability evaluation remained planned after deployment.Source: §4.3 · p. 8

5 Empirical findings from the EVALUATION CARDS corpus

The corpus audit finds pervasive gaps in reproducibility, benchmark documentation, and multi-source reporting. Joint canonicalization makes these reporting patterns visible across a large public evaluation record, while the corpus remains skewed toward English-language benchmarks and frontier-scale models.

5 Empirical findings from the EVALUATION CARDS corpus

  • The corpus contains 5,816 models, 635 single benchmarks, 62 families, 10 composites, and 101,955 reported results from 30 organizations.Its coverage overrepresents English-language benchmarks and frontier-scale models because of the ingestion sources.Source: §5 · p. 9
  • 48,698 of 50,461 result triples (96.5%) lack at least one minimal reproducibility field, with max_tokens absent from 95.6% and temperature from 93.9%.For agentic benchmarks, eval_plan and eval_limits are missing from 100% of triples.Source: §5 · p. 9
  • First-party rows populate 0.0% of base reproducibility fields on average versus 16.6% for third-party rows across 180 paired model–benchmark reports.Source: §5 · p. 9
  • Median benchmark documentation completeness is 10.7% across 635 benchmarks, with raw score fields populated but provenance and comparison-context fields systematically absent.Score reporting alone does not provide the documentation needed to interpret what a score means.Source: §5 · p. 9
  • 98.2% of 49,865 model–benchmark pairs are reported by only one party, while 51.9% of multi-organization metric groups exceed the 5% divergence threshold.First-party-only reporting is more concentrated in agentic and general benchmarks than in safety benchmarks.Source: §5 · p. 9
  • Joint canonicalization exposes setup variation, reporting disagreement, and schema-level documentation gaps that remain invisible in individual publications or leaderboards.These patterns describe the public reporting record rather than the underlying evaluations.Source: §5 · p. 9

6 Community adaptability and adoption

EVALUATION CARDS is an openly governed reporting layer that composes evaluation evidence and adapts its presentation for different users. Its walkthroughs show how the system exposes missing reproducibility fields, uneven documentation, source provenance, and score divergence across models and benchmarks.

6 Community adaptability and adoption

  • EVALUATION CARDS is designed as a participatory, openly governed, extensible initiative that unifies reporting infrastructure and renders evidence for different audiences.The project supports community governance, self-hosting, schema extensions, signal modifications, and new reader modes.Source: §6 · pp. 9–10

A.2 Corpus

  • The corpus contains 5,816 models, 635 single-benchmarks across 62 families and 10 composites, and 101,955 reported results from multiple ingestion sources.The corpus overrepresents English-language benchmarks and frontier-scale models.Source: §A.2 · p. 21

A.3 Model-level walkthrough: GPT-5

  • 95% of GPT-5’s 213 reported results lack at least one minimal reproducibility field, with only 11 results reporting temperature and max_tokens.The interface surfaces this missingness as a reproducibility gap.Source: §A.3 · p. 21
  • GPT-5’s benchmark documentation completeness is 93% where Auto-BenchmarkCards exist but 11% for the remaining benchmarks.The model has 64 benchmarks with matching Auto-BenchmarkCards; populated examples include livecodebench-pro and global-mmlu-lite.Source: §A.3 · p. 21
  • Only 13% of GPT-5’s reported results are first-party, while 87% are third-party or independent; FrontierMath and HumanEval have first-party-only reporting.The provenance signal distinguishes source types and highlights benchmarks without supplementary reproduced scores.Source: §A.3 · pp. 21–22
  • MATH-500 scores for GPT-5 range from 84.7% to 98.9% across three organizations, exposing setup-dependent divergence across sources.EVALUATION CARDS presents these differences when individual leaderboards may omit them.Source: §A.3 · pp. 21–22

A.4 Evaluation-level walkthrough: MMLU-Pro

  • On MMLU-Pro, 98% of 5,079 reported results lack a minimal reproducibility field, while 11 same-model cases exceed the setup comparability threshold.The benchmark aggregates 401 models from eight organizations and flags potentially setup-dependent results.Source: §A.4 · p. 22
  • MMLU-Pro also reveals cross-organization score divergence: Llama 3.2 is reported at 20.9% by Hugging Face and 61.8% by Arcadia Impact.Six model entries exceed the cross-party divergence threshold.Source: §A.4 · p. 22

A.5 Corpus-level analysis

Corpus-level aggregation shows that reporting gaps are systematic rather than confined to individual walkthroughs. Reproducibility, documentation completeness, and provenance analyses reveal widespread missing context and limited cross-party overlap.

Reproducibility

  • 96.5% of 50,461 reported model-benchmark-metric triples lack at least one minimal reproducibility field.Missingness is concentrated in max_tokens (95.6%) and temperature (93.9%), while agentic-specific fields are missing for 100.0% of agentic triples.Source: §A.5 · p. 22
  • First-party reports populate 0.0% of base reproducibility fields on average, compared with 16.6% for third-party reports across 180 paired cases.This comparison is computed on the same model-benchmark pairs.Source: §A.5 · p. 23

Completeness

  • Median per-benchmark documentation completeness is 10.7% across 635 benchmarks, despite raw score fields reaching 100.0% population.Benchmark-card documentation, provenance, and comparison-context fields are systematically thinner than raw score reporting.Source: §A.5 · p. 23

Provenance

  • 98.2% of 49,865 model-benchmark pairs are reported by only one party, and 51.9% of scored multi-organization metric groups show cross-party divergence flags.Among paired reports, divergence exceeds the threshold for 7.2% of pairs; first-party-only reporting is most prevalent for agentic and general benchmarks.Source: §A.5 · p. 23

B Limitations & Future Work

EVALUATION CARDS inherits limitations from its sources, canonicalization choices, artifact-side scope, and LLM-focused coverage, while future work targets contamination controls and broader integration. The supporting interviews used semi-structured methods and included technical, developer, and policy perspectives.

B Limitations & Future Work

  • Missing or un-ingested results and incomplete benchmark cards limit corpus coverage, while ambiguous benchmark identifiers can distort comparability and multi-source coverage.Auto-BenchmarkCards validates factual accuracy but not comprehensiveness, and EEE is not a complete census of public reporting.Source: B Limitations & Future Work · p. 24
  • The artifact-side scope excludes process decisions such as pre-run commitments and pilot execution, so completeness does not assess the underlying evaluation process.Process-side documentation is intended for complementary mechanisms such as preregistration systems.Source: B Limitations & Future Work · p. 24
  • EVALUATION CARDS does not judge whether scores satisfy safety, regulatory, or deployment-fitness thresholds because those criteria vary by jurisdiction and use case.The research and policy reader modes were also informed by preliminary interviews that may not capture all reader types.Source: B Limitations & Future Work · p. 24
  • Current coverage focuses on LLM evaluation reporting and does not support other AI systems or modalities.Expanding coverage is identified as a development priority.Source: B Limitations & Future Work · p. 24
  • Future work prioritizes structured contamination-control fields and includes longitudinal tracking, non-English benchmark coverage, and links to preregistration and audit systems.The current schema captures contamination only in free-text limitations fields that do not affect reporting completeness.Source: B Limitations & Future Work · p. 24
  • Completeness scores measure documentation adequacy, not the rigor or thoroughness of the underlying evaluation, creating a risk of safetywashing.The authors avoid letter grades and pass/fail thresholds, but misinterpretation remains possible.Source: B Limitations & Future Work · p. 24

C Interview Methodology

  • The study conducted semi-structured interviews with 12 stakeholders across technical, developer, and policy roles, recruited through author networks across 9 organizations and 3 regions.Participants were informed about the study, assured anonymity, and interviewed under institutional ethics approval.Source: C Interview Methodology · p. 25

C.1 Interview Participants

  • Interview participants with both policy and technical roles were tagged primarily as policy stakeholders while contributing insights about both groups’ interpretive needs.The participant table contextualizes roles, sectors, and geographic locations.Source: §C.1 · p. 25, Table 4 · p. 25

C.3 Limitations

The paper notes that its interview sample is concentrated in North America, while the implementation combines evaluation records and benchmark metadata through normalization, entity resolution, and precomputed comparison views. The pipeline preserves unresolved identifiers for review and reports in-domain resolver accuracy by entity type.

C.3 Limitations

  • Ten interviews were conducted, primarily with participants residing in North America, so Global South perspectives may be missing.Participants were recruited through interviewers’ networks, and critical viewpoints may have been missed.Source: §C.3 · p. 26

D.1 Inputs

  • The platform combines per-evaluation run records carrying model, evaluator, provenance, and numeric-result data with benchmark cards describing benchmark structure and scoring.These two upstream stores produce a consumer-facing record.Source: §D.1 · p. 27

D.2 Pipeline

  • Normalization decomposes nested results, canonicalizes model, benchmark, and metric identities, then joins records to benchmark metadata and materializes comparison views.Unmatched strings remain visible rather than being silently dropped.Source: §D.2 · p. 27

D.2.1 Data Cleaning

  • Cleaning lowercases identifiers, collapses separators, separates family and version suffixes, detects multilingual splits, and parses metric variants such as pass@k.These operations create comparable keys while retaining version and parameter information where relevant.Source: §D.2.1 · p. 27

D.2.2 Entity Matcher

  • The entity registry matches aliases through exact, normalized, and narrow fuzzy strategies, while unresolved strings continue downstream for human review.On 200 sampled entities per type, accuracy was 98.3% for models, 77.4% for benchmarks, and 86.7% for metrics, measured on in-domain data.Source: §D.2.2 · p. 28

D.3 Worked Examples

  • Worked examples trace records from the upstream evaluation store through benchmark metadata joins into frontend-served outputs.They illustrate how the assembled records move through the platform into user-facing views.Source: §D.3 · p. 28

D.3.1 Example 1: a single-benchmark, single-metric record

This section illustrates how Evaluation Cards structure and assess evaluation records, preserving provenance, benchmark and metric detail, and model variants while defining stakeholder-specific reading modes. It also operationalizes signals for reproducibility, reporting completeness, provenance, comparability, aggregation, and annotation agreement across multiple scopes.

D.3.1 Example 1: a single-benchmark, single-metric record

  • Single-benchmark records are normalized into canonical model, benchmark, and metric identifiers while preserving raw upstream strings, provenance, and scores for reuse across benchmark, model, and developer views.The GPQA Diamond example resolves to canonical accuracy and moonshotai/kimi-k2-thinking, while retaining raw identifiers and third-party source metadata.Source: §D.3.1 · pp. 30–31

D.3.2 Example 2: a composite benchmark with sub-tasks and a roll-up

  • Composite records are decomposed into canonical leaf results while retaining parent-suite grouping, and aggregate scores are marked for display but excluded from corpus-wide leaderboards and averages.The HELM Capabilities record yields five atomic benchmark results plus a “Mean score” aggregation tagged is_summary_score: true.Source: §D.3.2 · p. 32
  • Metric specificity is preserved during composite normalization: HELM’s MMLU-Pro result remains “COT correct” rather than being collapsed into generic Accuracy, while its leaf key joins the shared benchmark metadata.The row retains helm_capabilities as its family key and mmlu_pro as its canonical leaf key, enabling comparison with other MMLU-Pro results.Source: §D.3.2 · p. 32

Persona 1: Technical Evaluator

G Governance

  • Governance uses role-based review, scope-dependent change procedures, fixed proposal records, semantic versioning with deprecation windows, public decision logs, and a sustainability fallback when fewer than two maintainers remain.Substantive changes require a proposal, a 21-day comment period, and steering-group approval; breaking changes require a major version bump and at least 90 days of dual-version support.Source: §G.1 · p. 35, §G.2 · p. 35, §G.3 · p. 35

H.1.1 Reproducibility

  • Reproducibility is computed per result by checking required execution fields, using temperature and max_tokens as the minimal sub-schema and adding harness, eval_plan, and eval_limits for agentic evaluations.Any missing required field flags the result for reproducibility concerns.Source: §H.1 · p. 37, §H.1.1 · p. 37
  • A result is a reproducibility gap when any required field is missing, with the interface identifying missing fields and reporting aggregate and per-field missingness.Reproducibility is summarized through the share of flagged triples and missingness rates for individual fields.Source: §H.1.1 · p. 37

H.1.2 Reporting Completeness

  • Completeness averages population across 28 schema fields, scoring binary fields as present-or-absent and partial fields by their populated sub-items.The interface also shows fully missing fields, median benchmark completeness, and per-field population rates; completeness remains distinct from reproducibility.Source: §H.1.2 · pp. 37–38

H.1.3 Provenance

  • Provenance records evaluator relationships and multi-party reporting, while benchmark risk categories are surfaced as attention cues rather than included in numerical scores.Each score indicates whether it was reported by a first party, third party, or collaboratively, and whether multiple parties reported the same triple.Source: §H.1.3 · p. 38

H.1.4 Comparability

  • Comparability flags score divergence exceeding 5% of a metric’s range across setup variants or reporting parties, surfacing the underlying setup differences.The overall flag combines variant and cross-party divergence, using the uniform threshold θ = 0.05 across metrics.Source: §H.1.4 · p. 38

H.2 Aggregation Across Views

  • Model, benchmark, and corpus views aggregate the interpretive signals over their respective result sets, including first-party-only shares and comparability-flag counts.The aggregation framework applies analogous operations at model, benchmark, and corpus scopes.Source: §H.2 · p. 39

H.3 Agreement metrics

  • Agreement evaluation uses percent agreement, Cohen’s κ, and Krippendorff’s α for single-label fields, while multi-label fields additionally use exact match, Jaccard similarity, and pooled item-by-tag agreement.Blank ratings are excluded pairwise; pooled multi-label confidence intervals require caution because item-by-tag decisions are clustered, and κ and α may be redundant with two raters.Source: §H.3 · p. 39

H.4 Benchmark Categorization

This section describes EvalCards’ benchmark categorization and situates it within prior reporting artifacts, evaluation infrastructure, and AI-evaluation guidance. It synthesizes heterogeneous recommendations from a systematic review of 52 studies into a consolidated evaluation framework while retaining methodological and viewpoint differences as interpretive limitations.

H.4 Benchmark Categorization

  • The corpus contains 635 unique benchmarks categorized with an 18-category taxonomy using name-only LLM assistance, heuristic rules, and subsequent human corrections and recategorizations.The benchmarks were deduplicated through surface normalization, and four ambiguous entries were retained for future review.Source: §H.4 · p. 40

I Related Work Extended

  • Prior artifacts document isolated models, datasets, benchmarks, audits, evaluations, or governance processes, leaving stakeholders to assemble context across sources without extraction infrastructure.These artifacts generally use a single static representation and impose manual population burdens that hinder adoption at scale.Source: I Related Work Extended · p. 42
  • Evaluation repositories and leaderboards standardize either run data or result displays, but not both, while benchmark information remains underdocumented.Repositories collect runs without reader-facing interpretive interfaces, whereas leaderboards display scores without the inference details and metadata needed for interpretation.Source: I Related Work Extended · p. 42

J Systematic Literature Review

  • The systematic review addresses fragmented and differently framed recommendations by preserving their heterogeneity while building a consensus-oriented framework for consistent AI evaluation practice.The review is intended to inform EvalCards by clarifying differences, supporting consensus, and guiding infrastructure rather than resolving or disputing proposals.Source: J Systematic Literature Review · p. 43, §J.1 · p. 43

J.2.1 Search, Screening, and Inclusion

  • The review used broad searches and supplemental expert, Elicit, gray-literature, and citation sources, then screened documents for evaluation-relevant recommendations using reviewer judgment and expert feedback.The authors note that supplemental sources may favor mainstream, well-known, or highly cited work, while full-paper screening and exclusion decisions can introduce interpretive bias.Source: §J.2 · p. 43, §J.2.1 · pp. 44–45

Extracted Item-Level Characterization

  • Extracted recommendations were coded by assessment class, type, workflow stage, artifact, objective, and supercategory, with independent reviewer categorizations reconciled before best-fit framework synthesis.The synthesis mapped evaluation-framework elements into a shared structure and produced structured tables plus a consolidated framework for later Delphi elicitation.Source: §J.2.3 · p. 45, Extracted Item-Level Characterization · pp. 45–46, §J.2.5 · p. 49

J.2.5 Synthesis & Groupings

  • The synthesis grouped recommendations by workflow stage and objective, then refined a shared best-fit framework through literature mapping, subcategories, and expert feedback.Items could contribute to multiple groupings because codebook dimensions were non-exclusive.Source: §J.2.5 · p. 49

J.2.6 Risk of Bias within studies, in synthesis and in reporting biases

  • The included literature comprised mainly frameworks, guidance documents, position papers, and regulatory or grey literature, so standard empirical risk-of-bias tools were not applicable.The review retained methodological and viewpoint differences and addressed selection, coverage, and interpretive biases in its discussion.Source: §J.2.6 · p. 49

J.3.1 Selected Studies and Characteristics

  • 52 unique papers published from 2020–2025 remained after screening 748 sources, with relevant publications increasing each year.The final exclusions removed papers that were too narrow, outdated, irrelevant, non-research, inaccessible, or duplicated.Source: §J.3.1 · p. 49
  • 38/52 studies (73%) targeted developers, while 13/52 (25%) addressed auditors and 12/52 (23%) addressed policy makers.Providers and users were target audiences in 2/52 (4%) and 1/52 (2%) studies, respectively.Source: §J.3.1 · p. 50

J.3.2 Extracted Item Characteristics

This section presents a structured, lifecycle-spanning framework for coding extracted AI evaluation recommendations and reporting requirements. It defines overlapping characteristics, classification dimensions, and benchmark-, model-, and evaluation-level fields to support reproducibility, auditability, provenance, and comparison across sources.

J.3.2 Extracted Item Characteristics

  • The 730 items primarily concerned evaluations (82%) and pursued Transparency & Accountability (78%) and Performance Fit-for-Purpose (53%), with 82% combining multiple objectives.The most common objective combination was Performance Fit-for-Purpose + Transparency & Accountability (47%, 285/602).Source: §J.3.2 · pp. 50–51
  • Among 730 extracted items, 73% (533/730) explored interactions between multiple workflow stages, with Reporting & Transparency + Scoping & Objectives most discussed at 12% (65/533).Source: §J.3.2 · p. 51
  • Recommendations most often targeted the evaluation Process (73%), with 73% (536/730) discussing multiple artifacts, especially Outputs + Process (32%, 172/536).The most frequent supercategories were Test Design & Task Construction and Metrics & Rubrics, Statistical Validity & Uncertainty, each applying to 12% of items.Source: §J.3.2 · p. 51

J.3.3 Best Fit Framework

  • Among 13 framework papers, seven addressed Evaluation Stages / Needs, and their recurring elements were organized into 11 candidate categories.The seven relevant frameworks were [5] [12] [63] [79] [85] [86].Source: §J.3.3 · p. 53

J.3.4 Synthesis & Groupings

  • Expert feedback consolidated the 11 candidate categories into five higher-level areas: Evaluation Design, Before Evaluation Execution, Evaluation Execution, Evaluation Lifecycle, and Evaluation Reporting & Publication.The final framework is presented as recommended practice areas, with itemized sub-items and explanatory prompts provided separately.Source: §J.3.4 · p. 55

J.4 Discussion

  • The review found that reporting and transparency were the most widely discussed themes, while monitoring, operationalization, and instrumentation received much less attention, suggesting greater concern with disclosure than execution practice.Only 2% (15/730) of items concerned Instrumentation, and the review characterized evaluation recommendations as increasingly focused on socio-technical processes rather than model-only scorecards.Source: §J.3.2 · p. 51, §J.4 · pp. 57–58

J.7 Systematic Review References

  • The systematic review’s references list the 52 sources used to ground the extracted-item synthesis, including recommendations, audits, governance proposals, and evaluation critiques.Source: §J.7 · p. 60

J.8 Codebook for the Classification, Grouping and Synthesis of Extracted Items

  • The codebook classifies extracted items for later Delphi elicitation rather than presenting them as endorsements, correctness judgments, priorities, or a finalized standard.Source: §J.8 · p. 61

1.1 Scope

  • Extractable items are statements that recommend, caution against, constrain, or condition how AI evaluations, audits, or related governance processes are designed, executed, reported, or followed up.Source: §1.1 · p. 61
  • Because AI evaluation practices are value-laden, context-dependent, and heterogeneous, categories are heuristic and overlapping, with consistency and transparency prioritized over forced precision.Source: §1.2 · p. 61

Classification Dimensions

2.1 Item Class

  • Item class distinguishes evaluation practices, audit practices, governance processes, and rare other claims, with evaluation preferred when an item plausibly fits evaluation or audit.Source: §2.1 · p. 62

2.2 Extracted Item Type

  • Extracted-item type records whether a statement is a category, challenge, consideration, requirement, design, performance, reporting, or follow-up guidance.Source: §2.2 · p. 62

2.3 Workflow Stage / Topic

  • Workflow-stage coding captures where an item applies, including purpose, test design, inputs, instrumentation, scoring, human involvement, statistics, sensitivity, safety, and responsible use.Source: §2.3 · p. 63, Workflow Stages / Topics · p. 63

Workflow Stages / Topics

  • Workflow-topic coding assigns items to the stage or topic where the main design decision or obligation arises, while allowing multiple tags for genuine overlap.Topics include reliability and validity, human evaluation, robustness, misuse and safety, monitoring, reproducibility, documentation, and operational constraints.Source: Workflow Stages / Topics · p. 64

2.4 Artifact Discussed

  • Artifact coding identifies what an extracted item concerns, including data or prompts, models, systems, outputs, processes, and environments.Multiple artifacts may be selected; Process is preferred for recommendations about conducting evaluations, while Environment is used when context materially affects interpretation or validity.Source: §2.4 · p. 64

2.5 Objective

  • Objective coding captures why an extracted item exists, covering performance, safety, fairness, privacy, security, transparency, efficiency, and compliance.Coders should record the paper’s stated objective, select multiple objectives when applicable, and use Compliance only for explicit formal obligations.Source: §2.5 · p. 65

2.6 Supercategory (Synthesis Theme)

  • Each item receives one supercategory, a broad thematic classification intended for high-level synthesis and comparison across workflow stages or objectives.Supercategories support sense-making and thematic clustering rather than fine-grained analysis.Source: §2.6 · pp. 65–66

Supercategory Values

  • Supercategory values organize items around evaluation purpose and scope, process use, test design, data and scenarios, human evaluation, and prompting or elicitation.These themes distinguish evaluation aims and use from task structure, inputs, human benchmarks, and how model responses are elicited.Source: Supercategory Values · p. 66
  • The framework defines descriptive, overlapping supercategories covering validity, robustness, fairness, generalization, reproducibility, transparency, interpretation, benchmarking, tooling, operationalization, and out-of-scope items.Coding should reflect what papers emphasize rather than reviewer interest, and sparsity is not required.Source: Supercategory Values · p. 66

3. Decision Rules and Precedence

  • Coding precedence follows stated intent: reporting requirements motivated by validity remain Reporting, while normative disagreement does not justify exclusion.Supercategories are descriptive rather than prescriptive, and overlapping assignments are expected.Source: §3 · p. 67, Coding Guidance · p. 67

4. Worked Examples

  • Worked examples classify preregistration as a requirement in scoping and test design, emphasizing transparency and interpretability rather than governance compliance.The example treats the evaluation process as the primary artifact because the binding decision occurs before execution.Source: §4 · pp. 67–68
  • A second worked example codes contamination checks as a procedure spanning data and process because contamination threatens reported-performance validity across future model versions.The item addresses an action to perform, not merely a consideration, while also implicating reliability, robustness, and security.Source: §4 · p. 68

5. Reviewer Workflow and Disagreement Handling

  • Reviewers flag apparently incorrect codings or missing multi-select options, while lead-reviewer judgment resolves logged disputes without erasing the disagreement record.Resolution uses the codebook and comparisons with similar prior items.Source: §5 · p. 68

6. Versioning and Change Log

  • The codebook records an initial draft and a later version for use, including review, corrections, regenerated sections, and substantial edits.The supplied change log identifies versions 0.1 and 1.0 with dates and rationales.Source: §6 · pp. 68–69

J.9 Delphi Items

  • The Delphi checklist covers stakeholders, purpose and scope, preregistration, task construction, ethics, pre-run validation, contamination controls, run logging, adaptations, analysis differences, access, governance, reporting, costs, and replication.Specific checklist items include metrics and uncertainty, judge specifications and quality control, data partitioning, baselines, reproducibility provenance, artifact access, operational documentation, comparison standards, and replication guidance.Source: §J.9 · pp. 69, 71, 73
  • The evaluation report should state its purpose, intended use, decision linkage, audience, relation to prior evaluations, preregistration, construct definition, scope, threat model, and real-world validity boundaries.It should explain what is measured, which capabilities or failure modes are in scope, how results connect to consequences, and what uncertainties or path-dependence risks remain.Source: §J.9 · pp. 70–71
  • Reports should disclose external-validity gaps, representativeness and coverage limits, construct contamination, item-format effects, and trade-offs among internal, external, and construct validity.These limitations determine how broadly scores can be generalized and how non-target factors may influence them.Source: §J.9 · pp. 71–72
  • Evaluation documentation should explain metric choice, score meaning, baselines, aggregation, subgroup treatment, uncertainty sources, robustness tests, saturation, and statistical-analysis assumptions.Required disclosures include sensitivity to prompting, stochasticity, language, and implementation differences; uncertainty estimators; sample-size justification; multiplicity; researcher degrees of freedom; and distributional or dependence assumptions.Source: §J.9 · p. 72
  • Task and item development should document selection and adaptation logic, expert-informed validation, provenance, legal and ethical sourcing, representativeness, coverage gaps, modality assumptions, and human-subject safeguards.The reporting items also require explaining how formats, languages, and modalities affect measured capability and interpretation, alongside approvals, consent, privacy, risk mitigation, and compensation where relevant.Source: §J.9 · pp. 72–73
  • Before evaluation use, developers should preregister hypotheses, methods, thresholds, and decision rules; verify items, code, and metrics; and register expected baseline performance.The framework distinguishes design preregistration from later protocol commitment and pre-run readiness checks, while requiring correctness definitions, ground-truth validation, malformed-output handling, and human-reference comparability.Source: §J.9 · pp. 73–74
  • Judging and evaluation operations should specify rubrics, judge models or human-judge procedures, training, quality control, blinding, partitioning, holdout governance, revision controls, and pilot calibration.These requirements aim to support reliable scoring, fair comparisons, leakage control, and preserved validity for later or ongoing use.Source: §J.9 · p. 74
  • Evaluation reports should document contamination and gameability controls, preregistered comparison and disclosure plans, runtime configurations, full logs, protocol deviations, and execution-time adaptations.These records should explain pilot diagnostics, elicitation changes, adaptation rationale, and implications for reproducibility and interpretation.Source: §J.9 · pp. 75–76
  • Reports should analyze failure patterns, run differences, execution constraints, and gaps between evaluated models or setups and deployed systems when interpreting conclusions.They should also connect claims to evidence and decision criteria, disclose non-comparable conditions, and document disagreements over interpretation.Source: §J.9 · pp. 76–77
  • Evaluation artifacts should specify access, licensing, replication alternatives, reusable code conditions, governance, versioning, maintenance, and criteria for valid reuse or re-evaluation.Operational documentation should support portability, safe adaptation, issue reporting, updates, deprecation, and retirement as models, risks, and contexts change.Source: §J.9 · pp. 76–77
  • Comparative reporting should standardize datasets, prompts, scoring, temperature, access conditions, and uncertainty while qualifying trend, forecast, model-selection, and setup-based claims.Reports should contextualize results against baselines, provide sensitivity analyses, and explain differences from prior reports where relevant.Source: §J.9 · p. 77

Transparency

  • A replication package and reproducibility guide should be provided, or constraints on reproducibility should be explicitly explained.The transparency requirements also include evaluation usage conditions, release context, and access constraints.Source: Transparency · p. 78
  • Transparency reporting should identify model-version and release context, run timelines, access or publication restrictions, selective-reporting risks, commitment mechanisms, and disclosure trade-offs involving security, privacy, and misuse.Reports should provide structured machine-readable and human-readable artifacts, content safeguards, correction pathways, and current authoritative publication status.Source: Transparency · p. 78
  • Reproducibility requires executable artifacts, clear access conditions, end-to-end reproduction or auditable verification, and secure retention and versioning when artifacts are not public.The checklist specifically covers code, environments, scripts, seeds, dependencies, audit records, independent verification, and long-term artifact management.Source: Transparency · pp. 78–79

RESULTS

  • Study selection uses aggregate exclusion reasons rather than separately listing every excluded report, while included-study characteristics are presented through the selected-studies results and Figure 7.The checklist identifies the selected-studies section and Figure 6 for aggregate exclusion reasons, and Figure 7 and Section J.7 for included-study characteristics.Source: RESULTS · p. 81
  • The review reports no meta-analysis, heterogeneity investigation, sensitivity analysis, or formal certainty assessment.These omissions are explicitly marked as not applicable, not conducted, or not reported across the checklist entries.Source: RESULTS · p. 81, DISCUSSION · p. 81

OTHER INFORMATION

  • The review protocol is publicly available on OSF, but no registry number or protocol amendments are reported.The checklist distinguishes protocol access from registration details and records no described amendments.Source: OTHER INFORMATION · p. 82

K.1 Model view: GPT-5

  • The GPT-5 model view combines model identification, reporting-source ownership, and reported-metric comparisons across organizations.Reported metrics include cross-source scores, score ranges, divergence, result counts, means with 95% confidence intervals, and per-source scores.Source: §K.1 · p. 83, Figure 10 · p. 83, Figure 11 · p. 83

K.2 Benchmark view: MMLU-Pro

  • The MMLU-Pro benchmark view compares model results, documents benchmark coverage and licensing, surfaces issues such as contamination, and shows score progression and distributions over time.The view includes category comparisons, benchmark-card metadata, frontier progression, and score distributions across reported models.Source: §K.2 · p. 83, Figure 12 · p. 84, Figure 13 · p. 84
  • The benchmark view summarizes MMLU-Pro’s purpose, known issues, methodology, and interpretive signals for each evaluation entry.The signals panel covers reproducibility, completeness, provenance, and comparability alongside metric specifications; comparability also reports its threshold basis, model count, and cross-source agreement rate.Source: Figure 15 · p. 85, Figure 16 · p. 86, Figure 17 · p. 86

K.3 Developer View

  • The developer view lists developers with reported models and shows the number of models and benchmarks submitted by each.Source: Figure 18 · p. 87

L Fields Ingested by EVALUATION CARDS

  • EVALUATION CARDS ingests benchmark-card fields spanning details, intended users, data, methodology, ethical and legal considerations, risks, and flagged fields, plus evaluation-level metadata and results.Evaluation-level fields cover top-level and source metadata, evaluation libraries, model information, core results, metric and score configuration, generation configuration, and detailed results.Source: Table 27 · p. 88, Table 28 · p. 88, Table 29 · p. 89
  • The evaluation-level schema further separates source metadata, evaluation libraries, model information, core results, metric configuration, score details, generation configuration, and detailed results.Source: Table 34 · p. 90, Table 35 · p. 90, Table 36 · p. 90

M Mapping from Systematic Literature Review to EVALUATION CARDS Schema and Interpretive Signals

The paper maps systematic-review items to EVALUATION CARDS fields and interpretive signals, while providing public code and a live demo.

M Mapping from Systematic Literature Review to EVALUATION CARDS Schema and Interpretive Signals

N Code and Demo

Related papers

Loading 2606.09809v2…