Source-linked AI summary
GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, Natalie S. Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, Amelia Glaese, Jerry Tworek
TL;DR
AI’s economic impact is difficult to assess from adoption and GDP indicators alone, motivating GDPval’s direct evaluation of realistic work. GDPval builds expert-created tasks across economically important occupations and finds that leading models approach industry experts in deliverable quality, while releasing a gold subset and automated grader for further research.
Problem
Existing approaches emphasize AI adoption, usage, and GDP effects, but GDPval addresses the need to evaluate model performance directly on real-world economically valuable tasks.
Method
GDPval constructs expert-created tasks across 44 occupations in 9 sectors, evaluates deliverables through expert comparisons, and releases 220 gold tasks with an automated grader.
Results
Leading models approach industry experts in deliverable quality, with Claude Opus 4.1 winning or tying the human deliverable on 47.6% of gold-subset tasks.
Takeaways & Limitations
GDPval provides a basis for tracking model progress and studying AI capabilities on economically valuable work through realistic tasks and accessible grading.
Takeaways & Limitations
The initial dataset covers only 44 occupations and focuses on self-contained computer-based knowledge work, excluding manual labor and several context-dependent activities.
Abstract
from arXiv · showhide
We introduce GDPval, a benchmark evaluating AI model capabilities on real-world economically valuable tasks. GDPval covers the majority of U.S. Bureau of Labor Statistics Work Activities for 44 occupations across the top 9 sectors contributing to U.S. GDP (Gross Domestic Product). Tasks are constructed from the representative work of industry professionals with an average of 14 years of experience. We find that frontier model performance on GDPval is improving roughly linearly over time, and that the current best frontier models are approaching industry experts in deliverable quality. We analyze the potential for frontier models, when paired with human oversight, to perform GDPval tasks cheaper and faster than unaided experts. We also demonstrate that increased reasoning effort, increased task context, and increased scaffolding improves model performance on GDPval. Finally, we open-source a gold subset of 220 tasks and provide a public automated grading service at evals.openai.com to facilitate future research in understanding real-world model capabilities.
1 INTRODUCTION
GDPval is introduced to address limits in existing AI evaluations by measuring performance on realistic, economically valuable work across a broad range of occupations and formats.
- Benchmark motivation and scope: The benchmark responds to the gap between measuring AI adoption or GDP effects and directly evaluating performance on economically valuable work.Historical technological transitions suggest that economic effects can take years or decades to emerge after invention.
- Benchmark motivation and scope: GDPval evaluates AI models on real-world economically valuable tasks spanning 44 occupations across the top 9 sectors contributing to U.S. GDP.The full set contains at least 30 tasks per occupation, while the gold subset contains 5 tasks per occupation.
- Benchmark motivation and scope: Existing AI evaluations often emphasize reasoning difficulty, whereas GDPval emphasizes realism, representative occupational breadth, and economically relevant deliverables.The benchmark is intended to measure capabilities in work settings where time and cost also matter.
- Benchmark motivation and scope: GDPval tasks are based on actual industry work products and validated through multiple rounds of expert review.This design ties evaluation tasks to realistic work rather than only abstract reasoning difficulty.
- Benchmark motivation and scope: GDPval tasks require computer use and multimodal processing across formats including CAD files, images, video, audio, diagrams, presentations, spreadsheets, and conversations.Gold-subset tasks can require parsing up to 17 reference files, and full-set tasks up to 38.
2 TASK CREATION
GDPval constructs and validates realistic digital-work tasks by selecting economically important occupations, recruiting experienced professionals, and applying iterative quality control and expert grading.
- Occupation selection: GDPval covers 9 sectors and 44 occupations collectively earning $3T annually.The initial occupations were selected from sectors contributing over 5% to U.S. GDP and predominantly digital, high-wage occupations.
- Occupation selection: The occupation-selection process uses O*NET tasks and established economic task-content measures to identify predominantly digital work.Digital-task prevalence increases with non-routine cognitive content and decreases with routine and manual content.
- Expert task creation: Industry professionals with an average of 14 years of experience created realistic tasks based on their professional work.Experts were screened through interviews, background checks, training, and a quiz.
- Expert task creation: Each task pairs a request with a deliverable, maps the request to O*NET occupational tasks, and estimates dollar value from completion time and median hourly wages.Occupational experts also rated difficulty, representativeness, completion time, and overall quality against real-world standards.
- Quality control: All 1,320 full-set tasks underwent automated screening and multiple stages of human review, averaging five human reviews per task.Experts supplied comments and tasks were iteratively revised to improve quality and representativeness.
- Grading: Gold-subset grading uses blinded pairwise comparisons by relevant occupational experts, with an experimental automated grader achieving 66% agreement versus 71% human inter-rater agreement.The automated grader is faster and cheaper than expert grading.
3 EXPERIMENTS AND RESULTS
GDPval comparisons show frontier models approaching human expert deliverable quality, while performance improves with time, reasoning effort, prompting, and scaffolding. Human oversight may also reduce completion time and cost in the analyzed scenarios, although instruction-following and formatting errors remain common.
- Headline results: Claude Opus 4.1 was the best-performing evaluated model on the GDPval gold subset.The evaluation used blind pairwise comparisons by professional industry experts.
- Headline results: Model performance on the GDPval gold subset increased roughly linearly over time for OpenAI frontier models.
- Headline results: 47.6% of Claude Opus 4.1 deliverables were better than or as good as human deliverables on the GDPval gold subset.Claude excelled in aesthetics, while GPT-5 excelled in accuracy.
- Speed and cost comparison: AI assistance paired with expert oversight showed potential to save time and money relative to unaided experts in the analyzed scenarios.The setup involved repeated model sampling followed by human review and, if necessary, human completion.
- Model strengths and weaknesses: Instruction-following failures were the most common reason experts rejected Claude, Grok, and Gemini deliverables, while GPT-5 most often lost because of formatting errors.All models sometimes hallucinated data or miscalculated.
- Increasing reasoning effort and scaffolding: Additional reasoning effort improved performance, while prompting and scaffolding also increased GPT-5 performance.The experiments varied reasoning effort and used prompt-tuning and agent scaffolding.
- Increasing reasoning effort and scaffolding: Prompting eliminated black-square artifacts in GPT-5 PDFs and reduced egregious PowerPoint formatting errors from 86% to 64%.Multimodal deliverable inspection increased from 15% to 97%, and human preference win rates improved by 5 percentage points.
4 OPEN-SOURCING
The paper open-sources a 220-task GDPval gold subset with prompts and reference files and provides an experimental public automated grader.
- Open-sourcing: The 220-task gold subset includes open-sourced prompts and reference files.
- Open-sourcing: An experimental automated grader is publicly available at evals.openai.com, while human expert comparison remains the recommended grading method.The tasks were scrubbed of information that could identify their expert authors.
5 LIMITATIONS
GDPval’s initial evaluation is limited in dataset breadth, task realism, coverage of work types, and automated-grader capability. Its tasks focus on self-contained digital knowledge work and provide complete context in one-shot prompts.
- Dataset scope: The full GDPval set currently covers 44 occupations with 30 total tasks per occupation, making it a limited initial task sample.The authors state that they are expanding the dataset.
- Task scope: The initial version excludes manual and physical labor and tasks requiring extensive tacit knowledge, personally identifiable information, proprietary software, or interpersonal communication.
- Task realism: GDPval tasks are precisely specified and one-shot rather than interactive, unlike real-world tasks that may require discovering the relevant context.The paper reports that performance degrades with less context.
- Grading: The current automated grader has several limitations compared with human expert graders.The paper does not consider the automated grader a full substitute for industry expert grading.
- Grading: Constructing and running GDPval is expensive, particularly when using industry expert graders.
6 CONCLUSION
GDPval contributes a benchmark for economically valuable work, analyses of human and frontier-model deliverables, experiments on performance factors, an open gold subset, and an automated grader. The authors position it as a tool for tracking model progress and assessing AI’s social impacts.
- Contributions: GDPval measures real-world economically valuable tasks and compares human and frontier-model deliverables on quality, speed, and cost.
- Contributions: The paper tests how reasoning effort, prompting, scaffolding, and context affect benchmark results.
- Contributions: GDPval open-sources 220 gold-subset tasks with prompts and reference files and releases an automated grader at evals.openai.com.
- Conclusion: The authors hope GDPval will improve the science of tracking model progress and provide better data for assessing AI models’ social impacts.
A.1.1 AI DISCLOSURE
The paper discloses that AI tools supported parts of the research and engineering process. These uses included literature review, language editing, and routine coding assistance.
- AI models supported the literature review and revisions to the paper’s language.
- AI coding assistants were used in regular engineering workflows to help find and fix bugs.
- The disclosed AI use covered both research-writing tasks and software-engineering support.
A.1.2 SENSITIVE CONTENT AND POLITICAL CONTENT DISCLOSURE
Some GDPval tasks contain sensitive material, including sexual, alcoholic, vulgar-language, and political themes. The paper retains these tasks because they reflect themes addressed in several occupations and does not endorse their content.
- Some GDPval tasks include NSFW content involving sex, alcohol, vulgar language, and political content.
- The dataset retains sensitive tasks because they reflect real themes addressed in occupations such as film, literature, law, and politics.
- The paper states that retaining these tasks does not endorse the particular actions or views they contain.
A.1.3 THIRD-PARTY REFERENCES DISCLOSURE
GDPval includes limited third-party brand and trademark references for research and evaluation. The disclosure states that these references imply no affiliation or endorsement and clarifies the status of people and names appearing in the dataset.
- GDPval contains limited references to third-party brands and trademarks solely for research and evaluation.
- The paper states that third-party references do not imply affiliation or endorsement.
- Trademarks remain the property of their respective owners.
- Some dataset images and videos feature AI-generated individuals and real people who provided permission.
- Private individuals’ names and identifying references in GDPval are fictitious, with resemblance to actual persons or entities described as coincidental.
A.2.1 SPEED AND COST ANALYSIS, CONTINUED
GDPval’s speed and cost analysis compares model-assisted workflows with unaided expert completion while examining how performance varies across tasks, sectors, occupations, and deliverable formats.
- Speed and cost: 404 minutes and $361 are the average human completion time and cost for gold-subset tasks.Average expert review time was 109 minutes, costing $86; model completion time and cost depend on empirical API performance.
- Speed and cost: When review and redo work are included, the potential time and cost payoff from using models shrinks.The analysis also excludes human self-review, undesirable human deliverables, and catastrophic-mistake costs.
- Variation across tasks: Win rates vary across sectors and occupations, with strongest models approaching parity in some sectors and occupations but remaining low in others.Government, Retail Trade, and Wholesale Trade are examples of sectors where leading models approach parity.
- Variation across tasks: Performance differs by deliverable type: Claude leads most formats, while GPT-5 high leads pure text despite low overall win rates.The distinction reflects stronger visual and aesthetic performance for Claude and stronger accuracy on pure text for GPT-5 high.
- Variation across tasks: Win rates are highest for 0–2-hour tasks and decline steadily as completion time increases.Models therefore perform best on faster, less time-intensive tasks within the reported duration breakdown.
- Failure analysis: GPT-5 failures were most often acceptable but subpar, while roughly 29% were bad or catastrophic and roughly 3% were catastrophic.The failure analysis also reports that 23% of ratings marked the model better, approximately matching observed inter-rater agreement.