PaperlayerView exact revision on arXiv

Source-linked AI summary

ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge

Zhilin Wang, Jaehun Jung, Ximing Lu, Shizhe Diao, Ellie Evans, Jiaqi Zeng, Pavlo Molchanov, Yejin Choi, Jan Kautz, Yi Dong

arXiv:2510.18941v2cs.CLcs.AIcs.LG

TL;DR

Many valuable professional tasks require evaluating open-ended responses, but verification-based assessments are concentrated in tasks with easily checked answers. ProfBench provides expert-curated rubrics across professional domains and methods to reduce LLM-judge bias and evaluation cost. GPT-5-High achieves only 65.9% performance on the benchmark.

  • Problem

    Verification-based evaluation is concentrated in tasks with easily checked answers, while open-ended professional work needs assessment beyond short answers and unique-answer tasks.

    Source: §1 · pp. 1–2

  • Method

    ProfBench uses expert-curated rubrics across four professional domains and proposes methods to reduce LLM-judge bias and evaluation cost.

    Source: §1 · pp. 2–3

  • Results

    GPT-5-High achieves only 65.9% performance on ProfBench, which includes more than 7,000 response-criterion pairs.

    Source: §1 · p. 3

  • Takeaways & Limitations

    ProfBench provides a rubric-based benchmark for open-ended professional tasks and is designed to be fairer and more accessible through reduced judge bias and evaluation costs.

    Source: §7 · p. 9

    The authors report that DeepResearch-Bench RACE uses synthetic criteria not verified by expert humans, with its criteria and reference answer generated by Gemini-2.5-Pro.

    Source: §1 · p. 2

Abstract

from arXiv · show

Evaluating progress in large language models (LLMs) is often constrained by the challenge of verifying responses, limiting assessments to tasks like mathematics, programming, and short-form question-answering. However, many real-world applications require evaluating LLMs in processing professional documents, synthesizing information, and generating comprehensive reports in response to user queries. We introduce ProfBench: a set of over 7000 response-criterion pairs as evaluated by human-experts with professional knowledge across Physics PhD, Chemistry PhD, Finance MBA and Consulting MBA. We build robust and affordable LLM-Judges to evaluate ProfBench rubrics, by mitigating self-enhancement bias and reducing the cost of evaluation by 2-3 orders of magnitude, to make it fair and accessible to the broader community. Our findings reveal that ProfBench poses significant challenges even for state-of-the-art LLMs, with top-performing models like GPT-5-high achieving only 65.9% overall performance. Furthermore, we identify notable performance disparities between proprietary and open-weight models and provide insights into the role that extended thinking plays in addressing complex, professional-domain tasks. Data: https://huggingface.co/datasets/nvidia/ProfBench and Code: https://github.com/NVlabs/ProfBench and Leaderboard: https://huggingface.co/spaces/nvidia/ProfBench

1 INTRODUCTION

Verification-based evaluation is concentrated in tasks with easily checked answers, while valuable professional work often requires open-ended responses. ProfBench addresses this gap with expert-curated rubrics across professional domains and evaluates models and LLM judges, including methods to reduce judge bias and cost.

1 INTRODUCTION

  • Rubrics can decompose open-ended, real-world tasks with many valid responses into criteria that a good response must fulfill.The introduction contrasts these tasks with math, programming, and short-answer evaluations where correctness is easier to verify.Source: §1 · pp. 1–2
  • Existing publicly available rubric benchmarks did not cover diverse professional domains beyond health and machine learning, and DeepResearch-Bench RACE used synthetic, unverified criteria.The authors also note that its criteria and reference answer were generated by Gemini-2.5-Pro, which received more than 48.5 out of 50 on each of four axes.Source: §1 · p. 2
  • ProfBench introduces expert-curated rubrics across Chemistry, Physics, Consulting, and Finance for professional-grade LLM tasks.The experts have PhDs in Chemistry or Physics, MBAs in Consulting or Finance, or equivalent experience and incumbent professional roles.Source: §1 · p. 2
  • GPT-5-High reaches only 65.9% performance on ProfBench, which includes more than 7,000 response-criterion pairs and requires PhD- or MBA-level knowledge.The benchmark covers scientific research and business tasks.Source: §1 · p. 3
  • The authors evaluate more than 40 models as response generators and LLM judges, and propose judge-bias and cost-reduction methods.Their reported setup can run with no more than 1% bias across three models from different providers for $12 using o3, 2–3 orders of magnitude cheaper than existing rubric-based evaluation.Source: §1 · p. 3

2 PROFBENCH OVERVIEW

ProfBench contains 7,347 human-written response-criterion pairs across 80 tasks, equally divided among four professional domains. The overview compares its scale with other benchmarks and notes the resource demands of expert rubric annotation.

2 PROFBENCH OVERVIEW

  • ProfBench contains 7,347 human-written response-criterion pairs across 80 distinct tasks, equally divided among four domains: Chemistry PhD, Physics PhD, Finance MBA, and Consulting MBA.Source: §2 · p. 3
  • PaperBench contains 8,316 response-criterion pairs across 20 tasks, while HealthBench contains 8,053 pairs across hundreds of tasks.DeepResearch-Bench has 100 tasks with one reference response each and no human-written rubric.Source: §2 · p. 3
  • Expert-rubric benchmarks tend to be small because each task takes time to annotate and recruiting professionals such as industry-experienced Physics PhDs is difficult.DeepResearch-Bench averages 2.3 English tasks per domain across its 22 domains.Source: §2 · p. 3

3 DATA COLLECTION

Data collection combines professional annotators, multi-part report tasks, reviewed rubrics, and human scoring of model responses. The process sets expertise and quality checks while restricting annotation to English text and publicly findable documents.

3 DATA COLLECTION

  • 38 annotators from 8 countries contributed across 80 tasks, with 44.7% holding PhDs and 18.4% holding MBAs.Annotators had an average of 5.24 years of experience after their highest degree and spent about 10 to 20 hours on each task.Source: §3 · p. 3
  • Prompts were designed to challenge frontier models available in July 2025 and often combine related subquestions into multi-page reports.Prompts and expected generations had to be in English and text-only; annotators could use internet-searchable documents but not proprietary documents.Source: §3 · p. 4
  • 41.4% of annotator-written criteria were marked as needing improvement during review.Each task aimed for 15 to 60 independently gradable criteria that collectively capture response quality; reviewers provided feedback and recommendations on each criterion.Source: §3 · p. 4
  • Annotators scored responses from o3, Grok4, and DeepSeek R1-0528 against each criterion with Yes or No and a brief justification.The models were selected as those believed to perform best on the collected tasks at the start of annotation in July 2025.Source: §3 · p. 4

4 BENCHMARKING MODELS AS LLM-JUDGES

This section defines criterion-level LLM judging and evaluates judges by human agreement, bias, and cost. It compares judge models and selects a cost-conscious judge for the benchmark.

4.1 EVALUATION

  • Judges classify each response-criterion pair as fulfilled or not, with evaluation based on human-aligned Macro-F1 and a bias-adjusted Overall score.Bias-Index is the range between the maximum and minimum model-specific bias; Overall is Macro-F1 minus Bias-Index.Source: §4 · p. 4, §4.1 · p. 4
  • The cost estimate multiplies input and output token counts by public prices, excludes discounts, and may fall to one-tenth through caching alone.Reasoning judges may generate up to 32,000 tokens, while non-reasoning judges generate one Yes-or-No token; the prompt was selected in preliminary GPT-4.1 experiments.Source: §4.1 · pp. 4–5
  • Fleiss’ κ = 0.912: two expert annotators showed excellent agreement when re-annotating 1,127 response-criterion pairs.Source: §4.1 · p. 5

4.2 RESULTS

  • 75.4%: GPT-4.1 leads non-thinking judges, while Gemini-2.5-Pro leads reasoning judges at 78.2%; open-weight alternatives approach these scores at much lower cost.Kimi-K2-0711 is 0.2% behind GPT-4.1, and GPT-OSS-120B-low is 1.5% behind Gemini-2.5-Pro.Source: §4.2 · p. 5
  • +20.8%: GPT-4.1-mini improves over nano, but GPT-4.1 adds only +0.5%; Llama-3.3-70B gains +3.4% over Llama-3.1-70B, versus +0.9% from scaling to 405B.The results indicate diminishing returns from size alone and differences associated with model-family improvements.Source: §4.2 · p. 5
  • 0.7% to 4.8%: enabling thinking improves matched-model scores, while GPT-5 minimal-to-low reasoning gains range from 4.4% to 18.6%.Higher reasoning effort is not consistently better; the passage suggests added thinking may improve human alignment while increasing model-specific bias.Source: §4.2 · p. 5
  • 78.2% Overall: GPT-OSS-120B matches Gemini-2.5-Pro while costing 1.68% as much, and ProfBench-Judge costs $0.70 versus $1320 for PaperBench JudgeEval.The selected judge uses high reasoning effort for Physics, Chemistry, and Style criteria, and low effort for others.Source: Table 2 · p. 6, §4.2 · p. 6

5 BENCHMARKING MODELS AS REPORT-GENERATORS

This section evaluates report-generating models on tasks grounded in documents and scored with expert criteria. GPT-5 leads, but performance varies across domains and model settings, while task-specific sampling can reduce evaluation cost.

5 BENCHMARKING MODELS AS REPORT-GENERATORS

  • Reports answer prompts using grounding documents, and a selected GPT-OSS-120B judge scores criteria with importance weights from 1 to 4.The schema’s predicted scores differed from human-annotated performance by 0.7% to 1.3% across three models; generation was repeated 16 times to estimate performance variance.Source: §5 · pp. 6–7

5.1 RESULTS

  • 65.9%: GPT-5 is the top-performing report generator; Physics is lowest at 49.3%, followed by Finance at 63.8%, Chemistry at 70.6%, and Consulting at 80.0%.The passage compares ProfBench with other benchmarks and reports that GPT-5 scores 67.2% on HealthBench.Source: Table 3 · p. 7, §5.1 · p. 7
  • 15.0%: the closed-source/open-weight performance gap is largest in Finance, compared with less than 1% in Physics and 9.2% in Chemistry and 9.6% in Consulting.The leading proprietary models are GPT-5 at 65.9%, o3 at 61.4%, and Gemini 2.5 Pro at 60.3%; GPT-OSS-120B leads open-weight models at 54.9%.Source: §5.1 · p. 8
  • 10.2%: GPT-5-mini improves over GPT-5-nano, whereas GPT-5 adds 5.6% over GPT-5-mini, illustrating diminishing returns from size alone.The passage notes similar patterns in Gemini, Llama 4, and Qwen3 families.Source: §5.1 · p. 8
  • 0.3% to 2.3%: enabling thinking slightly improves overall scores for matched models, while increasing GPT-5 reasoning effort from minimal to high improves performance by 4.8%.In contrast, the Qwen3 30B thinking model scores 44.6%, while its instruction-following counterpart scores 49.3%.Source: §5.1 · p. 8
  • 3.0%: GPT-5’s low-verbosity score drop versus medium verbosity accompanies a 30.7% shorter response; high verbosity adds 1.1% in score with 32.4% more characters.The passage says response-length effects are typically within the standard deviation of the verbosity settings.Source: §5.1 · p. 8
  • 4 responses per task: variance-based integer-programming allocation cuts inference cost to 25% of the original without compromising estimation robustness.The allocation gives more samples to high-variance tasks and fewer to low-variance tasks; Figure 3 encodes score spread and optimal sample counts.Source: §5.1 · pp. 8–9, Figure 3 · p. 9

6 ABLATION: HOW IMPORTANT ARE GROUNDING DOCUMENTS?

The ablations test performance without task grounding documents, with and without web search. Removing the documents reduces performance, while search recovers some of the loss but uses substantially more prompt tokens.

6 ABLATION: HOW IMPORTANT ARE GROUNDING DOCUMENTS?

  • The ablations compare prompting without task documents against prompting without them but allowing web search to retrieve relevant documents.The task documents have publicly accessible URLs.Source: §6 · p. 9
  • 9.4 to 11.9% performance reduction follows removing grounding documents with o3 and o4-mini, with a larger 17.1 to 18.7% reduction in information extraction / recall.The authors say this suggests explicit reference to task documents is crucial for generation quality.Source: §6 · p. 9
  • 4.1 to 7.0% of performance is recovered by adding web search after removing the grounding documents.Search uses 0.12 to 0.24 million input tokens per task, or 4 to 12x the original grounding-document tokens.Source: §6 · p. 9

7 CONCLUSION

ProfBench uses expert-curated rubrics across four professional domains to evaluate LLMs on open-ended tasks with real-world value. It is designed to reduce LLM-judge bias and evaluation costs.

7 CONCLUSION

  • ProfBench is the first benchmark with expert-curated rubrics across Physics PhD, Chemistry PhD, Finance MBA, and Consulting MBA domains.The benchmark is described as fair and accessible, with reduced LLM-judge bias and substantially lowered evaluation costs.Source: §7 · p. 9
  • ProfBench moves beyond exam-style short-answer tasks to systematic evaluation of open-ended problems with real-world value.Source: §7 · p. 9
  • The benchmark targets professional-grade LLM evaluation across scientific research and business domains.The conclusion names Physics PhD, Chemistry PhD, Finance MBA, and Consulting MBA.Source: §7 · p. 9

ETHICS STATEMENT

Data collection was performed by a vendor following internal ethical and legal review, with qualified personnel working under fair conditions.

ETHICS STATEMENT

  • The vendor completed internal ethical and legal reviews before beginning data collection.Source: ETHICS STATEMENT · p. 10
  • Personnel were qualified for their roles and compensated at or above locally applicable standards.Source: ETHICS STATEMENT · p. 10
  • Annotators agreed not to infringe third-party intellectual-property or proprietary rights, with indemnification provisions for claims.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper points readers to separate sections for data collection and evaluation details. Its examples illustrate professional-domain task formats, prompt-template comparisons, and domain-specific annotator qualifications.

REPRODUCIBILITY STATEMENT

A EXAMPLES

  • Chemistry and physics examples ask for multi-step domain calculations, including acid-base titration and an axion-model anomaly ratio.The examples include rubrics for specific numerical answers and calculation steps.Source: A EXAMPLES · p. 17
  • Consulting examples specify response formats, calculation requirements, and task limits, including a Hong Kong education-market analysis.The finance example requests an investment-memo analysis of IFFIm, while chemistry and physics examples pose technical calculation problems.Source: A EXAMPLES · p. 18
  • The finance example asks for a detailed IFFIm investment memo covering its financing mechanism, issuances, risks, and potential as a blueprint.Source: A EXAMPLES · p. 19

B PROMPT TEMPLATES

  • The main judge prompt uses a Yes/No response, while alternatives use True/False or 1/0; elaborate prompts did not improve results and increased token expenses.The main template performed better than alternative 1 by 2.2% and alternative 2 by 4.5%.Source: B PROMPT TEMPLATES · pp. 19–20

C ANNOTATOR RECRUITMENT

  • Consulting annotators generally needed two years at a specified major firm or four years at another consulting firm; finance annotators generally needed two years at a selected bank.Physics and chemistry annotators generally needed a relevant PhD from a Global Top 100 university, with interviews assessing field knowledge and communication.Source: C ANNOTATOR RECRUITMENT · p. 20

D FURTHER DESCRIPTIVE STATISTICS

ProfBench analyses describe a four-domain, expert-rubric benchmark and examine model performance, judge reliability, and variance-reducing sample allocation. The allocation analysis finds lower performance variance with optimal allocation, while the descriptive results show substantial differences across models and domains.

D FURTHER DESCRIPTIVE STATISTICS

  • 80 prompts are divided equally across four domains, and source documents are truncated to at most 20 pages per PDF to fit current models’ context limits.The authors state this truncation makes tasks substantially easier, but is necessary for benchmarking many current models.Source: D FURTHER DESCRIPTIVE STATISTICS · pp. 20–22
  • 62.9% of rubric criteria assess reasoning, 34.1% extraction, and 3.0% style; reasoning chiefly checks logical validity, while style chiefly checks formatting.The taxonomy subcategories were proposed with Qwen3-235B-A22B-Instruct-2507 and vetted by the research team.Source: D FURTHER DESCRIPTIVE STATISTICS · p. 21, Table 5 · p. 21
  • 51.6% of criteria are fulfilled by o3, 47.4% by Grok4, and 45.2% by R1-0528; average fulfillment is 39.1% in Finance and 40.3% in Physics.Consulting criteria are fulfilled 56.5% and Chemistry criteria 59.4% of the time.Source: D FURTHER DESCRIPTIVE STATISTICS · p. 21

E INFERENCE SETUP

  • 78.2% is reported for both Gemini-2.5-Pro and gpt-oss-120b judges, whose bias-index values are below 0.5% across three reference models.For the official gpt-oss-120B judge, relative bias is below 0.5% and absolute bias across three models is below 1.0%.Source: E INFERENCE SETUP · p. 23
  • A 10,000-iteration permutation test for gpt-oss-120b yields a two-sided p-value greater than 0.56 for the observed bias index.The test shuffles response model labels while fixing judge predictions and human labels.Source: E INFERENCE SETUP · p. 23

F OPTIMAL PERFORMANCE ON PROFBENCH AT EACH PRICE-POINT

  • OpenAI models occupy the Pareto frontier at each price point, while Gemini-2.5 and Qwen3-Instruct-2507 are close to it.The passage attributes OpenAI’s position as likely related to first-mover advantages in rubric-style data.Source: Figure 4 · p. 24

G FORMULATING OPTIMAL ALLOCATION PROBLEM

  • Task-level performance variance motivates allocating more responses to high-variance tasks, and the allocation objective is solved by dynamic programming.The variance estimate assumes i.i.d. sampling and independent tasks; task variances are estimated from 16 rollouts per prompt across four representative models.Source: G FORMULATING OPTIMAL ALLOCATION PROBLEM · p. 24
  • 50% of the naive point estimate’s standard deviation is achieved with optimal allocation at B = 160 (average n_i = 4) and N = 40.Optimal allocation also consistently reduces variance relative to the alternative methods across budget scales.Source: G FORMULATING OPTIMAL ALLOCATION PROBLEM · p. 25

H FUTURE DIRECTIONS

The authors outline ways to extend ProfBench’s difficulty and scope while noting constraints on tool use, modality, domains, and objective creativity assessment. The review process iteratively refines prompts and rubrics, with reviewer support for response annotation.

H FUTURE DIRECTIONS

  • Search tools are not evaluated further because implementations differ across providers, making it difficult to separate tool capability from model capability or tool integration.Code-interpreter use also complicates fair comparisons because only OpenAI offered it through a publicly accessible API at the time of the experiments.Source: H FUTURE DIRECTIONS · p. 25
  • ProfBench is text-only by design, so its tasks exclude image and audio inputs to support popular text-only open-weight models.The authors identify multimodal evaluations as a possible direction as more open-weight models gain those capabilities.Source: H FUTURE DIRECTIONS · p. 25
  • The benchmark covers four domains because expert annotators were constrained by recruitment feasibility, research timeframe, and budget.The authors report no near-term plans to expand newly annotated domains and recommend using ProfBench alongside other benchmarks where appropriate.Source: H FUTURE DIRECTIONS · p. 26
  • Originality and creativity are difficult to measure objectively, so prompts are written not to require creative answers.The authors suggest creativity assessment may become a next step as models improve on objective tasks.Source: H FUTURE DIRECTIONS · p. 26
  • Current tasks use expert-truncated grounding documents of at most 20 pages per file, an easier setup that can be replaced with original documents as models improve.The original documents can exceed 800 pages, offering a possible way to test long-context processing and avoid saturation.Source: H FUTURE DIRECTIONS · p. 26

I REVIEW PROCESS

  • Prompt and rubric review iterates through reviewer feedback, including requests for clearer task specifications and self-contained criteria.The Physics example includes edits to specify the holonomy basis and acceptable answer form, then revisions to criteria that relied on earlier definitions.Source: I REVIEW PROCESS · pp. 26–27, §4 · p. 27

12. Reviewer B approves response annotations

  • One annotator develops the prompt and rubrics and scores model responses, with a reviewer supporting the process.The reviewer may vary across stages if the original reviewer is unavailable; the review checks binary gradability, non-overlap, and coverage of important response qualities.Source: §12 · p. 27

J SENSITIVITY AND QUALITATIVE ANALYSES OF LLM JUDGES

The sensitivity analyses find low provider bias among leading judges, while qualitative review identifies recurring judgment errors. The authors caution that these errors may affect absolute scores more than relative comparisons.

Sensitivity Analysis

  • Top judges’ max–min differences are less than 1%, suggesting no significant preference for a single provider among the three reference systems.Obtaining expert judgments for many additional models would require costly, time-consuming re-annotation.Source: J SENSITIVITY AND QUALITATIVE ANALYSES… · pp. 27–28
  • gpt-oss-120b achieves 0.22% in standard deviation of biases and 0.15% in fairness gap; top judges show consistently small biases under these metrics.Table 6 compares standard deviation of biases with provider-level fairness gap.Source: J SENSITIVITY AND QUALITATIVE ANALYSES… · p. 28, Table 6 · p. 28

Performance across categories

  • Models generally perform better on Extraction, then Reasoning, and worst on Style, where criteria can require checking nuanced requirements across a response.Examples include format requirements and verifying that all concentration values meet a requested precision.Source: J SENSITIVITY AND QUALITATIVE ANALYSES… · p. 28

Qualitative Analysis

  • Qualitative review identifies judges missing equivalent statements, mistaking implied statements for explicit ones, and failing to integrate information from different response parts.These are reported as recurring shortcomings of LLM judges.Source: J SENSITIVITY AND QUALITATIVE ANALYSES… · pp. 28–29
  • LLM-judge errors tend to be systematic rather than random and may affect absolute scores more than relative comparisons; judge-human gaps were 0.7 to 1.3% across three models.The authors describe LLM judges as imperfect oracles.Source: J SENSITIVITY AND QUALITATIVE ANALYSES… · p. 29

K REPORT-GENERATION PERFORMANCE ACROSS VARIOUS DOMAINS

The authors test whether differences in domain difficulty undermine overall model comparisons. Their analyses find stable cross-domain performance hierarchies and strong correlations between domain and aggregate scores.

Design philosophy

  • ProfBench preserves the authentic difficulty of professional work rather than equalizing difficulty through task selection, rubric pruning, or post-hoc reweighting.The authors argue forced equalization could discard diagnostically valuable tasks, relax rubrics, or introduce arbitrary weighting.Source: K REPORT-GENERATION PERFORMANCE ACROSS… · p. 30

Aggregation validity

  • The authors made no specific data intervention beyond matching task counts across domains and report domainwise and reasoning-type performance alongside aggregate scores.They note that overall scores inevitably distill subcategory-level information.Source: K REPORT-GENERATION PERFORMANCE ACROSS… · p. 30
  • To test aggregation, the authors normalized each domain’s scores by its best model and examined rank correlation between domain and aggregate performance.They also compared the variability of normalized scores across the four domains.Source: K REPORT-GENERATION PERFORMANCE ACROSS… · p. 30
  • Correlations between domain performance and overall scores are close to or beyond 0.8 in all four domains, while performance hierarchies and gaps remain stable across domains.The authors conclude that domain-based categorization and aggregation are diagnostically valid and do not systematically penalize a subset of domains.Source: K REPORT-GENERATION PERFORMANCE ACROSS… · p. 30, Table 7 · p. 30

Related papers

Loading 2510.18941v2…