PaperlayerView exact revision on arXiv

Source-linked AI summary

PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning

Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego Mares, Pavit Singh, Michael Liu, Subodh Chawla, Pete Cline, Lucy Ogaz, Ernesto Hernandez, Zihao Wang, Pavi Bhatter, Marcos Ayestaran, Bing Liu, Yunzhong He

arXiv:2511.11562v1cs.CLcs.CY

TL;DR

Academic benchmarks provide limited evidence about model performance on open-ended, economically consequential professional work in Legal and Finance. PRBench offers expert-authored tasks with expert-curated rubrics and evaluates models across these domains. On its Hard subsets, top scores remain 0.39 for Finance and 0.37 for Legal, while analyses identify persistent reasoning and reliability gaps.

  • Problem

    Academic and open-ended preference evaluations provide limited, coarse evidence about model reasoning quality in high-stakes professional domains such as Legal and Finance.

    Source: §1 · p. 2

  • Method

    PRBench releases 1,100 expert-authored Finance and Legal tasks with expert-curated rubrics and evaluates models using those criteria.

    Source: §1 · pp. 2–3

  • Results

    The best Hard-subset scores are 0.39 for Finance and 0.37 for Legal, while models struggle with transparency, auditability, correctness, and domain-specific diligence.

    Source: §1 · p. 2, §4.1 · p. 7

  • Takeaways & Limitations

    PRBench provides fine-grained rubric-based evaluation of open-ended professional reasoning, and model gaps in opaque or incomplete reasoning limit trustworthiness in professional settings.

    Source: §7 · p. 11

Abstract

from arXiv · show

Frontier model progress is often measured by academic benchmarks, which offer a limited view of performance in real-world professional contexts. Existing evaluations often fail to assess open-ended, economically consequential tasks in high-stakes domains like Legal and Finance, where practical returns are paramount. To address this, we introduce Professional Reasoning Bench (PRBench), a realistic, open-ended, and difficult benchmark of real-world problems in Finance and Law. We open-source its 1,100 expert-authored tasks and 19,356 expert-curated criteria, making it, to our knowledge, the largest public, rubric-based benchmark for both legal and finance domains. We recruit 182 qualified professionals, holding JDs, CFAs, or 6+ years of experience, who contributed tasks inspired by their actual workflows. This process yields significant diversity, with tasks spanning 114 countries and 47 US jurisdictions. Our expert-curated rubrics are validated through a rigorous quality pipeline, including independent expert validation. Subsequent evaluation of 20 leading models reveals substantial room for improvement, with top scores of only 0.39 (Finance) and 0.37 (Legal) on our Hard subsets. We further catalog associated economic impacts of the prompts and analyze performance using human-annotated rubric categories. Our analysis shows that models with similar overall scores can diverge significantly on specific capabilities. Common failure modes include inaccurate judgments, a lack of process transparency and incomplete reasoning, highlighting critical gaps in their reliability for professional adoption.

1. Introduction

Academic reasoning benchmarks offer limited evidence about model performance in open-ended, high-stakes professional work. PRBench addresses this gap with expert-authored Finance and Legal tasks, curated rubrics, and analyses of model performance and prompt implications.

1. Introduction

  • Existing evaluations provide limited visibility into open-ended professional reasoning, despite the high stakes of Legal and Finance work.Preference-ranking benchmarks offer coarse, noisy, subjective signals that lack expert grounding and are difficult to interpret.Source: §1 · p. 2
  • PRBench introduces 1,100 expert-authored questions on reasoning-heavy, real-world Finance and Legal problems, paired with expert-curated, verified rubrics.Each rubric contains 10–30 descriptive criteria with importance weights.Source: §1 · p. 2
  • Models still struggle with correctness, process transparency, auditability, and domain-specific diligence, limiting their practical reliability.They may make inaccurate judgments or reach correct conclusions through incomplete or opaque reasoning.Source: §1 · p. 2
  • The open-source benchmark covers 25 Finance and Legal topics across 114 countries and dependencies and 47 US jurisdictions.The tasks are described as realistic and challenging.Source: §1 · pp. 2–3
  • An independent domain-expert study validates rubric reliability alongside the project’s internal quality checks.The stated aim is to ensure robustness of rubric design.Source: §1 · p. 3
  • The benchmark analysis annotates prompts’ economic implications and decision types to examine where models perform well or poorly.The evaluation also includes open-source and proprietary chat models, with substantial room for improvement reported.Source: §1 · p. 3

2. Overview of PRBench

PRBench comprises two expert-built benchmarks whose overall statistics are summarized in Table 1. Its topics, geographic coverage, and difficulty-based Hard subsets reflect the benchmarks’ scope and construction.

2. Overview of PRBench

  • 182 qualified experts created the conversations; about 30% are multi-turn, all user turns are in English, and assistant turns come from three open-source models.Experts curate a rubric for each conversation’s final model response; all 1,100 evaluation conversations are open-sourced, with a private heldout set retained for future contamination monitoring.Source: §2 · p. 3
  • Table 1 summarizes overall statistics for the two PRBench benchmarks.The caption identifies the table as dataset statistics, without specifying individual measures.Source: Table 1 · p. 3, §2 · p. 3
  • The benchmarks cover 13 Finance and 12 Legal topics, spanning over 114 global jurisdictions and 47 US jurisdictions.The 250 most difficult Legal and 300 most difficult Finance tasks form the respective Hard subsets, ranked by average scores across evaluated models.Source: §2 · p. 3

3. Data Collection

PRBench's data collection combines expert-authored real-world questions with jurisdiction-aware conversations and weighted rubrics across Finance and Legal. Rubrics follow explicit design criteria and undergo independent expert validation.

3. Data Collection

  • 182 qualified professionals contributed questions they or their clients would care about, requiring substantial analysis, interpretation, or creative thinking.Legal annotators held a JD or equivalent; Finance contributors had a Master’s, CFA, or at least six years of experience.Source: §3.1 · p. 3, §3 · p. 3

3.1 Building Conversations

  • About 30% of cases use multi-turn conversations of up to 10 turns, with prompts spanning 114 countries and 47 US jurisdictions.Experts may hint at relevant jurisdictions; a small set of prompts includes reference texts.Source: §3.1 · p. 4

3.2 Creating Rubrics

  • Rubrics assign nonzero integer weights from -10 to 10; positive criteria describe desired qualities, while negative criteria flag undesired response properties.Negative criteria can penalize discussion of an irrelevant legal provision.Source: §3.2 · p. 5
  • Criteria are designed to be correct, nonredundant, collectively exhaustive, atomic, objective, and self-contained.These standards aim to prevent duplicate penalties while covering the aspects of a complete response.Source: §3.2 · p. 5
  • Six severity levels, from Critically Important to Critically Detrimental, precede weight assignment and are intended to reduce noise and inconsistency.The procedure is described as encouraging hierarchical reasoning.Source: §3.2 · p. 5
  • Rubrics span 7 Finance and 8 Legal axes, including shared response qualities and domain-specific criteria.Shared categories include Practical Utility, Handling Uncertainty, Supplemental Insight, and Instruction Following.Source: §3.2 · p. 5
  • Independent experts marked final-rubric criteria agree or disagree, producing 93.9% agreement on rubric validity.A random subset also received manual review for overall quality and correctness.Source: §3.2 · p. 6

4. Evaluation

The evaluation finds low top scores on the Hard subset and variation across rubric categories and response lengths. An expert-agreement study supports the choice of judge model.

4.1 Results

  • 0.39 for Finance and 0.37 for Legal are the best Hard-subset scores; full-set leaders score 0.51 and 0.50, respectively.One of GPT-5 and GPT-5 Pro consistently leads, followed by Grok 4 Fast Reasoning; newer models improve consistently on the benchmark.Source: Figure 5 · p. 6, §4.1 · p. 7
  • Web search improves O3 and Grok performance, but all Hard-subset performances remain ≤0.4, with O3 leading.The benchmark questions are designed to be solvable through reasoning alone, and code interpreter adds no gains beyond web search.Source: §4.1 · p. 7
  • GPT-5 leads Handling Uncertainty, Practical Utility, Legal Accuracy, and Financial Accuracy; category leaders also include Gemini 2.5 Pro and Grok-4 Fast Reasoning.Gemini performs well on Instruction Following, Grok-4 Fast Reasoning leads Legal Risk & Ethical Disclosure, and Kimi K2 Thinking ranks among the top three in Supplemental Insight.Source: §4.1 · p. 7, Figure 6 · p. 7
  • Among similar-scoring Finance models, Kimi K2 Thinking produces substantially shorter responses; GPT-5 and O3 balance performance and efficiency.GPT OSS 120B, Gemini 2.5 Flash, and Claude Sonnet 4.5 produce responses nearly twice the length of the best responses.Source: §4.1 · pp. 7–8, Figure 7 · p. 8

4.2 Evaluating LLM Judge

  • LLM judges show agreement with experts comparable to expert–expert agreement, except Claude Haiku 4.5; o4-mini was chosen for lower querying costs.Agreement was measured against two experts on 101 tasks using Cohen’s κ and Macro F1.Source: §4.2 · p. 8, Table 3 · p. 8

5. Dissecting the Dataset: Prompts, Rubrics, and Beyond

The dataset analysis annotates the decisions and economic pathways represented in prompts, then uses hierarchical rubric clusters to examine model capability gaps.

5.1 Inside the Conversations

  • Most prompts represent high-stakes real-world decisions with tangible downstream economic impact, analyzed through Decision Type and Economic Pathway annotations.The annotations are released with the dataset, and their distributions are shown in Figure 8.Source: §5.1 · pp. 8–9, Figure 8 · p. 9

5.1.1 Is AI Ready for Assisting High-Stakes Decisions?

  • Decision-type labels are intended to support future analysis of which high-stakes decisions current models handle well or poorly.Examples include whether to litigate or arbitrate, how to allocate funds, and which market to launch in.Source: §5.1.1 · p. 9

5.1.2 Can AI Handle Economically Consequential Questions?

  • Economic Pathway annotations capture whether and how answers could produce positive or negative economic consequences.Prompt examples include capital allocation under interest-rate shocks, cross-border acquisitions, and stress tests.Source: §5.1.2 · p. 9

5.2 Inside the Rubrics

  • The rubric analysis hierarchically clusters similar capabilities into a five-level tree to move from aggregate scores to fine-grained capability gaps.The analysis is conducted independently for Finance and Legal, and underperforming clusters can guide further fine-tuning or analysis.Source: §5.2 · pp. 9–10, Figure 9 · p. 10
  • A Finance example moves from the Risk and Regulatory Disclosure category to increasingly specific international and domestic compliance clusters.The example refines to compliance and risk analysis under global tax anti-avoidance frameworks, illustrating selectable levels of granularity.Source: §5.2 · p. 10
  • In Legal tax-law strategy and compliance, GPT-5 scores 0.64 accuracy versus Grok-4’s 0.16.In the Finance international corporate tax cluster, Claude Opus 4.1 scores 0.34 while the top-performing model scores 0.76.Source: §5.2 · p. 10

6. Related Work

PRBench addresses limitations in existing professional-task evaluations with a public, large-scale benchmark and expert-curated rubrics. The paper situates rubric evaluation as a scalable approach for open-ended tasks and as a resource for further research.

6.1 Evaluating LLMs on Economically Valuable Tasks

  • Existing professional-task evaluations are often private, costly to judge, and small, limiting accessibility, scalability, and topic coverage.Table 4 compares PRBench with selected professional-domain benchmarks; its caption defines GT as ground-truth-based evaluation and Partial as a non-major open-ended QA subset.Source: §6.1 · p. 11, Table 4 · p. 11
  • PRBench offers 1,100 public tasks and 19,356 expert-curated criteria, with multi-turn interactions to build real legal and finance context.The work describes this resource as an order of magnitude larger than APEX, ProfBench, and BigLaw Bench.Source: §6.1 · p. 11

6.2 Rubric-Based Evaluation and Reward

  • Rubric-based evaluation uses self-contained, objective criteria for scalable, style-neutral evaluation of open-ended professional reasoning and complex task artifacts.The paper also notes rubric use in general-domain benchmarks, agent evaluations, and reward functions for reinforcement learning.Source: §6.2 · p. 11
  • PRBench contributes public expert-curated criteria with detailed legal and finance categories for research on professional reasoning and reward modeling.The paper positions this resource as an expansion of rubric-based evaluation research.Source: §6.2 · p. 11

7. Conclusion

PRBench is a public, expert-annotated benchmark for high-stakes professional reasoning in Finance and Law. Its evaluation finds progress alongside persistent gaps in transparency and domain-specific diligence.

7. Conclusion

  • PRBench combines over 1,100 expert-authored tasks and 19,000+ rubric criteria for interpretable evaluation of open-ended, economically consequential problems.The benchmark is intended to support transparent, reliable, and economically valuable AI systems for real-world decision-making.Source: §7 · p. 11
  • Models make inaccurate judgments or reach conclusions through incomplete or opaque reasoning, limiting trustworthiness in professional settings.The conclusion reports persistent gaps in process transparency and domain-specific diligence despite progress by proprietary and open-source models.Source: §7 · p. 11

A. Evaluations with Web Search & Code Interpreter

On the Hard subsets, web search generally helps Grok and O3 but hurts other evaluated models; adding code interpretation generally brings no further performance boost.

A. Evaluations with Web Search & Code Interpreter

  • Web search generally helps Grok and O3 on the Hard subsets but hurts the other evaluated models.The authors attribute the pattern primarily to over-reliance on external sources instead of a cohesive answer.Source: A. Evaluations with Web Search & Code… · p. 15
  • Adding code interpretation alongside web search provides no additional performance boost except for Grok 4 in Finance.Source: A. Evaluations with Web Search & Code… · p. 15
  • Tables 5 and 6 evaluate the Hard subset of 300 finance and 250 legal tasks with o4-mini as judge in a single run; Table 5 reports average clipped performance scores.Table 5 compares web search on versus off; Table 6 compares web search plus code interpreter on versus off.Source: Table 5 · p. 15, Table 6 · p. 15

B. Dataset Details

The appendix describes rubric weights and categories, points readers to their definitions, and identifies the evaluation judge template's relation to prior work.

B. Dataset Details

  • Negative rubric weights are reserved for penalizing undesired response properties, and Figure 10 displays criterion-weight distributions.About 30% of multi-turn conversations use intermediate assistant turns sampled from open-source models, with user turns written by human annotators.Source: B. Dataset Details · p. 15

B.1 Rubric Category Definitions

  • The contributor-facing rubric-category definitions are provided in Tables 7 and 8.Source: §B.1 · p. 15

C. Prompt Templates

D. Evaluation Details

The evaluation appendix specifies model settings, explains response and aggregate scoring plus min-normalization, reports expert rubric validation, and outlines decision and economic-pathway categories.

D. Evaluation Details

  • Models use high reasoning mode except Claude Sonnet 4.5, whose thinking budget is 32,768; Gemini 2.5 Pro and Flash use thinking_budget=-1.Figure 10 shows rubric-weight distributions, with weights at both spectrum ends more frequent in Legal than Finance; Tables 7 and 8 give rubric-category definitions.Source: D. Evaluation Details · p. 15, Table 7 · p. 16, Table 8 · p. 17

D.1 PRBench Scores

  • The LLM judge assigns binary criterion grades, which feed weighted response scores that are averaged across prompts for each model.Each prompt has rubrics with nonzero weights from -10 to 10, and at least one positive-weight rubric keeps the denominator above zero.Source: §D.1 · p. 18

D.2 Min-Normalized Scoring

  • Min-normalization adjusts each response score by its prompt's minimum possible score before averaging, improving comparisons across prompts with differing rubric weights and phrasing.The paper uses normalized scores for rubric-category results in Figure 6; its rubric phrasing convention further mitigates sensitivity to positive-versus-negative wording.Source: §D.2 · p. 19

E. Additional Details for Rubric Validation Assessment

  • Independent subject-matter experts agreed that rubric criteria were justified in 93.9% of validation assessments.Reviewers who were not task authors marked each criterion agree or disagree based on construction, accuracy, and task relevance, and supplied written justifications.Source: E. Additional Details for Rubric… · p. 19

F. Definitions for Decision Type and Economic Pathway

  • The decision-type scheme includes informational requests, transactions, operations, planning, compliance, and an “Other” category, while economic-pathway tables are provided for Legal and Finance.The listed definitions cover informational background, transaction choices, operational steps, and compliance with standards.Source: Table 10 · p. 21, F. Definitions for Decision Type and… · p. 22

G. Sample Tasks

The appendix identifies one sample task for Finance and one for Legal.

G. Sample Tasks

Loading 2511.11562v1…