Preliminary suggestions for rigorous GPAI model evaluations

• GPAI evaluations have been criticised for weak statistical rigour, inadequate real-world risk measurement, poor documentation, and poorly designed constructs and instruments.Source: §I · p. 1
• The document compiles preliminary suggestions intended to improve internal validity, external validity and reproducibility.Source: §I · p. 1
• Suggestions cover human uplift studies, benchmark evaluations and cross-cutting practices, organised across design, implementation, execution and documentation.Source: §I · pp. 1–2
• The field is nascent and lacks consensus on how to robustly measure model capabilities and impacts amid rapid technological advancement.Source: §I · p. 2
• The compilation draws on literature from machine learning, statistics, psychology, economics, biology and other fields.Source: §I · p. 2
• The intended audience includes GPAISR providers, third-party evaluators and academic researchers developing or conducting model evaluations.Source: §I · p. 2

• The document defines high scientific rigour, identifies evaluation types in scope, and organises suggestions across four evaluation life-cycle stages.Source: §II · p. 2

• GPAISR providers must conduct evaluations, and relevant evaluations should demonstrate high scientific and technical rigour through internal validity, external validity and reproducibility.Source: II.a Elements of high scientific and… · pp. 2–3
• Rigorous evaluations may give decision makers and policymakers more accurate information about GPAI risks and expand opportunities for mitigation and preparedness.Source: II.a Elements of high scientific and… · p. 3

• Human uplift studies assess how access to or use of a GPAI model, relative to status quo tools, affects human task performance.Source: II.b Evaluation types in scope · p. 3
• Benchmark evaluations use standardised items to grade GPAI model responses or behaviour.Source: II.b Evaluation types in scope · p. 3

• The four stages are design, implementation, execution and documentation, providing the organising structure for suggestions across GPAI evaluations.Source: II.c Evaluation life cycle stages in… · p. 3, Table 2 · p. 4

Design

• Define the research question, target population, evaluation environment, measured concept, performance metrics and relevant domain expertise before evaluation.Suggestions also call for expert validation of evaluation items and grading criteria, plus assessment of whether methods measure the intended construct.Source: Design · p. 5
• Use power analyses and sufficient sample sizes, with pre-analysis plans and preregistration where possible, to support transparent and statistically rigorous evaluation.Power analyses may guide benchmark runs or questions and human-study group sizes, while novel evaluations may require multiple effect-size scenarios.Source: Design · p. 5
• For human-participant research, use ethical procedures, representative sampling, balanced incentives and complementary qualitative data to understand participant experiences and outcome mechanisms.Recommended practices include informed consent, privacy-preserving data handling, review-board approval and recruitment criteria aligned with the population of interest.Source: Design · p. 6

Implementation

• Specify elicitation techniques and participant tool-access protocols, including available time and compute, so implementation captures results responsive to the research question.For capability ceilings, elicitation may target maximum capabilities, with blinding of elicitation personnel presented as a gold standard.Source: Implementation · p. 6

Documentation

• Document procedures, data sources, participant characteristics, instructions, model parameters, execution environments, code, access conditions, resources, results distributions and interpretation guidance.Documentation should cover recruitment and attrition, task processing and annotation, prompts and grading, software dependencies, randomisation and analysis code, and instance-level outputs where relevant.Source: Documentation · pp. 7–8
• Securely release execution details and evaluation code while recording calendar time, compute, personnel effort and model access to support reproducibility.The recommended releases include evaluation and grading methodology, hardware and software environments, prompting methods, and code for cleaning, statistical testing and analysis.Source: Documentation · pp. 7–8

III.b Suggestions for rigorous human uplift studies

• Human uplift studies compare task performance with GPAI access against status quo tools and often use randomised controlled trials to assess causal impact.Prior examples are described as methodologically limited despite contributing to the literature.Source: III.b Suggestions for rigorous human… · p. 8, Design · p. 8

Design

• Use stratified or clustered reproducible randomisation and keep treatment and control conditions identical except for the GPAI models or tools studied.Randomisation may account for demographics such as domain expertise and prior GPAI experience, with random seeds supporting reproducibility.Source: Design · p. 8
• Keep user interfaces appropriate to the research question and constant across groups, because poor or unequal interfaces can distort measured human uplift.A superior treatment interface may depress control performance and overestimate uplift attributable to the model itself.Source: Design · p. 9

Implementation

• Pilot and validate instruments, protocols and monitoring mechanisms with out-of-sample participants resembling the study sample.Source: Implementation · p. 9
• Define tooling access carefully, monitor non-compliance and contamination, and measure mechanisms linking tool use to outcomes.Control-group access may capture AI diffusion and thereby underestimate human uplift, so access decisions should balance the research question with that risk.Source: Implementation · p. 9

III.c Suggestions for rigorous benchmark evaluations

• Benchmark evaluations provide repeatable, scalable evidence for comparing models or humans and for informing capability-threshold decisions and further risk assessment.Source: III.c Suggestions for rigorous… · p. 10

Design

• Benchmarks vary by output format, from multiple-choice and open-ended questions to long-form tasks requiring multiple steps and extended resources.Examples include MMLU and GPQA, MATH and HumanEval, and long-form evaluations such as PaperBench, SWE-bench and CTF challenges.Source: III.c Suggestions for rigorous… · p. 10
• Grading may be automated, human, model-based or combined, and best practices should reflect the grading strategy and benchmark size.Exact matching may require validation for external validity, while human grading invokes practices relevant to human participants; uncertainty methods also depend on benchmark size.Source: III.c Suggestions for rigorous… · p. 11, Design · p. 11
• The recommendations build on prior benchmark best-practice work, particularly Reuel et al. (2024a).Source: Design · p. 11

Documentation

• Document metrics and measurement methods, reference scores, test and prompt design, representative items and responses, and reusable software packages with replication scripts.Reference points may include floors, ceilings, human baselines and random performance; code and data may require canaries, encryption or other secure-release methods.Source: Documentation · p. 12
• For risk-relevant evaluations, provide representative benchmark items or proxy-task samples alongside summary statistics to support interpretation.Source: Documentation · p. 12

• The suggestions address methodological concerns across design, implementation, execution and documentation, drawing on lessons from several established fields.Source: §IV · p. 12
• The scope covers human uplift studies, benchmark evaluations and cross-cutting practices applicable across evaluation types.Source: §IV · p. 12
• The suggestions are preliminary, and future work should expand evaluation categories, prioritise recommendations and validate their role in the GPAI evaluation ecosystem.Source: §IV · p. 12

• Human uplift studies compare human task performance with and without GPAI access relative to status quo tools, while benchmark evaluations grade standardised item responses or behaviour.The notes distinguish these as two core evaluation categories and provide extended definitions elsewhere in the document.Source: Notes · p. 13
• Documentation and dissemination should support transparency and accountability while managing security, contamination, misuse, vulnerability-disclosure, and access-control risks.Not all documentation must be public; tiered access and secure information-sharing agreements are possible strategies.Source: Notes · p. 14
• Evaluation choices are out of scope and should be prioritised according to risk magnitude, available resources, timeline, and previous evaluation results.Evaluators may structure priorities across evaluation types, tasks, and constructs, including whether and how to run a particular evaluation.Source: Notes · p. 14
• Benchmark scores require construct-specific interpretation because a measure such as HumanEval pass@k may capture functional correctness without assessing maintainability, security, or integrated-system performance.Evaluation constructs should be clearly formulated and mapped to evaluation types so results remain interpretable across different beliefs or definitions.Source: Notes · pp. 14–15
• Control-group AI assistance can underestimate GPAI impacts, so researchers should define the research question and methodology carefully as tools such as search engines incorporate generated answers.Participants may also be asked whether they used AI tools after the study.Source: Notes · p. 15
• Automated grading is not always valid, while manual expert grading is costly and time-intensive and requires careful methodology to produce high-quality, representative data.Naive fuzzy string matching, such as BLEU for code, may fail to measure functional correctness accurately.Source: Notes · p. 16

Notes

• The cited works include Reuel et al. (2024a), Hendrycks & Woodside (2024), Philip & Hemang (2024), and Phan et al. (2025).Source: Notes · pp. 15–16

GPAI evaluations literature review

• The GPAI-evaluations review identified 64 papers through database searching, abstract filtering, and snowball sampling.The initial search returned 85 papers, 35 remained after filtering, and 29 additional papers were added through snowball sampling.Source: GPAI evaluations literature review · p. 17
• The review excluded sources lacking GPAI-evaluation focus, broader methodological scope, or acceptable publication status.Excluded materials included personal blogs or forums and papers focused on executing one specific benchmark rather than broader evaluation methodologies.Source: GPAI evaluations literature review · p. 17

Interdisciplinary knowledge literature review

• The interdisciplinary review used convenience sampling across machine learning, statistics, psychology, economics, biology and related fields, prioritising relevance to validity and reproducibility.It drew on academic papers, industry reports and established frameworks rather than attempting an exhaustive systematic review.Source: Interdisciplinary knowledge literature… · p. 17

Limitations

• The methodology is limited by documented practices, incomplete coverage of practitioner constraints, a December 2024 cutoff, and possible disciplinary overemphasis.The authors suggest practitioner interviews and cost-benefit prioritisation as future research directions.Source: Limitations · p. 17

Appendix B: Tagging criteria

• Internal validity concerns cause-effect capture, external validity concerns generalisation beyond the evaluation environment, and reproducibility concerns independently obtaining consistent results.The criteria respectively emphasise confounding and measurement, representativeness and deployment robustness, and transparent, standardised technical conditions.Source: Appendix B · p. 18

About this paper

• The document presents a preliminary compilation of practices for human uplift studies, benchmark evaluations and other evaluation types, organised across four lifecycle stages.Its intended audience includes GPAISR providers, third-party evaluators, policymakers and academic researchers.Source: About this paper · p. 24