Source-linked AI summary
ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
Manasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi, Clinton Wang, Ankit Aich, Huy Nghiem, Tahseen Rabbani, Ye Htet, Brian Jang, Sumana Basu, Aishwarya Balwani, Denis Peskoff, Marcos Ayestaran, Sean M. Hendryx, Brad Kenstler, Bing Liu
TL;DR
Deep Research evaluation must capture open-ended, multi-step, cross-document answers rather than short factual responses, while avoiding weak oversight and narrow task coverage. ResearchRubrics builds a human-authored benchmark and evaluates rubric adherence with human and model-based protocols. Leading agents achieve under 68% compliance, with failures concentrated in implicit reasoning and multi-document synthesis.
Problem
Deep Research is difficult to evaluate because answers are long, diverse, open-ended, and require multi-document synthesis, while existing benchmarks can rely on automated rubrics or narrow scopes.
Method
ResearchRubrics pairs realistic diverse prompts with 2,593 human-written, fine-grained rubrics, complexity axes, mandatory and optional criteria, and ternary LLM-as-judge evaluation.
Results
Leading Deep Research agents achieve around 67% rubric compliance and often fail on implicit reasoning, multi-document synthesis, and well-justified citation-supported answers.
Takeaways & Limitations
Fine-grained human-aligned rubrics expose capability gaps that aggregate metrics miss and support deployment decisions based on sufficiency as well as excellence.
Takeaways & Limitations
Response length correlates with rubric scores, and controlled experiments are needed to distinguish legitimate information density from residual verbosity bias.
Abstract
from arXiv · showhide
Deep Research (DR) is an emerging agent application that leverages large language models (LLMs) to address open-ended queries. It requires the integration of several capabilities, including multi-step reasoning, cross-document synthesis, and the generation of evidence-backed, long-form answers. Evaluating DR remains challenging because responses are lengthy and diverse, admit many valid solutions, and often depend on dynamic information sources. We introduce ResearchRubrics, a standardized benchmark for DR built with over 2,800+ hours of human labor that pairs realistic, domain-diverse prompts with 2,500+ expert-written, fine-grained rubrics to assess factual grounding, reasoning soundness, and clarity. We also propose a new complexity framework for categorizing DR tasks along three axes: conceptual breadth, logical nesting, and exploration. In addition, we develop human and model-based evaluation protocols that measure rubric adherence for DR agents. We evaluate several state-of-the-art DR systems and find that even leading agents like Gemini's DR and OpenAI's DR achieve under 68% average compliance with our rubrics, primarily due to missed implicit context and inadequate reasoning about retrieved information. Our results highlight the need for robust, scalable assessment of deep research capabilities, to which end we release ResearchRubrics(including all prompts, rubrics, and evaluation code) to facilitate progress toward well-justified research assistants.
1. Introduction
Deep Research agents answer open-ended, multi-document queries, but existing benchmarks inadequately assess their long-form synthesis and reasoning. ResearchRubrics addresses this gap with a human-crafted, complexity-aware benchmark and fine-grained rubric evaluation.
- Deep Research tasks require multi-step web exploration, targeted retrieval, and synthesis for open-ended queries.
- Existing benchmarks often assess short factual answers, use LLM-generated rubrics, or cover narrow domains, limiting evaluation of realistic deep research.
- ResearchRubrics organizes deep research queries by conceptual breadth, logical nesting depth, and exploration level.
- ResearchRubrics pairs diverse prompts across nine domains with 2,593 expert-written rubric criteria covering grounding, reasoning, completeness, relevance, and clarity.
- The benchmark uses outcome-based rubrics, separates mandatory from optional criteria, and includes negative criteria for extraneous or incorrect content.
- Ternary grading assigns partial credit, while ablation studies examine how rubric design affects agreement between LLM judges and human evaluators.
2. Related Work
Existing deep research benchmarks commonly rely on static datasets, answer keys, automated evaluation, or narrow academic tasks. ResearchRubrics instead combines realistic academic and everyday queries with expert-written rubrics and scalable LLM judging.
- Earlier benchmarks derive tasks from static corpora or use expert-curated questions.
- Derived benchmarks span academic retrieval, web retrieval, static-data queries, and specialized academic synthesis, often using automated metrics or published surveys as references.
- Expert-curated benchmarks include short-answer advanced-domain questions and long-form PhD-level problems, but reported weaknesses include narrow ambiguity coverage and automated rubric or reference-report generation.
- ResearchRubrics addresses these limitations with realistic academic and everyday queries, expert-written rubrics, and LLM judges that assess grounding, synthesis, reasoning, clarity, and citation usage.
3. Overview of RESEARCHRUBRICS
RESEARCHRUBRICS is a human-built benchmark of realistic, diverse research prompts paired with fine-grained rubrics and annotated task-complexity dimensions. It evaluates responses through weighted criteria and model-based judgments, while supporting analysis of rubric failures and grader consistency.
- Benchmark construction: 101 prompts span diverse research topics, each paired with 20–43 human-written criteria, for 2,593 unique rubric items.Experts iteratively refined every prompt and criterion without seeding or generating criteria with LLMs.
- Benchmark construction: Three experts propose, review, iterate, and independently finalize prompts and rubrics, while contributors work only in familiar domains.Prompt ideas come from user forums, Q&A sites, and brainstorming sessions to emulate realistic research questions.
- Task domains: The benchmark covers nine broad categories with a fairly even distribution, combining specialized topics such as AI/ML and historical analysis with everyday consumer research.Other categories add targeted challenges including creative synthesis and real-time news retrieval.
- Prompt complexity dimensions: Tasks receive orthogonal labels for Conceptual Breadth, Logical Nesting Depth, and Exploration, enabling balanced coverage and performance analysis by task complexity.The labels support filtering for focused experiments, such as high-depth reasoning tasks.
- Rubric design: Responses are judged against tailored criteria spanning six evaluation axes, with mandatory and optional requirements and positive or negative numerical weights.Mandatory criteria define sufficiency, optional criteria capture desirable qualities, and negative criteria penalize failures such as inaccuracies or irrelevance.
- Evaluation methodology: An LLM grader assigns Satisfied, Partially Satisfied, or Not Satisfied verdicts to every criterion, and normalized weighted sums produce task scores.The benchmark also compares model-based and expert judgments using class-level Macro F1 scores and analyzes failure rates by rubric category.
4. Experimental Results and Analysis
Across evaluated systems, rubric compliance remains below 70%, with failures concentrated in implicit reasoning, synthesis, and deeper multi-step tasks. The analyses also expose trade-offs in citation coverage, response length, and rubric design.
- 45-50% of failures came jointly from implicit reasoning and synthesis, despite below-20% failure rates for explicit retrieval and communication quality.
- Performance declined monotonically with greater logical nesting, while conceptual breadth was associated with a less steep difficulty increase.
- Gemini DR produced 111 citations at 81% accuracy, whereas Perplexity achieved 90% accuracy with 31 citations, exposing a breadth–accuracy trade-off.
- Binary grading achieved 0.72–0.76 Macro F1 and increased human agreement by approximately 20 percentage points over ternary grading.
- Concrete examples improved rubric alignment by 3-4% under binary grading, whereas automated rubric augmentation reduced alignment by 15-20%.
5. Conclusion and Future Work
RESEARCHRUBRICS provides fine-grained, human-aligned evaluation across 101 diverse research challenges and identifies substantial capability gaps in current deep research agents. The best agents achieve only around 67% rubric compliance, with recurring failures in cross-document integration and citation-supported justification.
- RESEARCHRUBRICS evaluates 101 diverse research challenges using expert-written criteria covering factual recall, reasoning soundness, source usage, completeness, and clarity.Its multi-dimensional assessment is intended to expose capability gaps that aggregate metrics can miss.
- Around 67% rubric compliance is achieved by today’s best agents, which often fail to integrate information across documents or provide well-justified answers with proper citations.
A. Extended Related Work
Prior deep research benchmarks span static-corpus retrieval, expert-authored questions, long-form research problems, and dynamic literature synthesis, but each leaves important evaluation gaps. RESEARCHRUBRICS addresses these gaps with realistic academic and everyday queries paired with fine-grained expert rubrics.
- Early benchmarks assess navigation and retrieval from static academic or web corpora, but static data limits coverage of open-ended synthesis and risks answer leakage.Static benchmarks also cannot adapt to newly emerging information.
- Humanity’s Last Exam tests expert knowledge with short-answer questions, while DeepResearch Bench and ExpertLongBench evaluate longer expert tasks using reference-based metrics, citation counts, or domain checklists.DeepResearch Bench reported that no model exceeded roughly 30% on its overall metrics, but its scoring leans heavily on reference overlap and simple citation counts.
- DeepScholar-Bench evaluates dynamic literature synthesis through up-to-date arXiv retrieval, emphasizing knowledge synthesis, retrieval quality, and source verifiability.
- AI co-scientist systems and benchmarks such as SPOT increase the need to assess whether generated reasoning and conclusions withstand expert scrutiny, not merely whether facts are retrieved.
- RESEARCHRUBRICS offers a middle ground by pairing broad academic and everyday research queries with expert-crafted rubrics covering factual grounding, cross-source synthesis, reasoning validity, and clarity.
B. Extended Results
The appendix extends the paper’s quantitative analysis to benchmark composition, task complexity, error structure, and the relationship between output length and rubric compliance.
- The appendix expands quantitative analysis of composition, complexity, and error structure, and examines how output length relates to rubric compliance.
B.1 Benchmark Composition and Rubric Coverage
ResearchRubrics is designed to measure holistic research ability through multi-axis tasks and fine-grained criteria, while domain-specific axis mixes capture differing expectations for synthesis, explicitness, and citation behavior.
- Tasks touch a mean of 4.74 rubric axes, reflecting the benchmark’s goal of measuring holistic research ability rather than single-skill performance.
- Each task contains 20–43 rubric criteria, with a mean of approximately 26 criteria per task.
- Rubric-axis proportions vary by domain, indicating that domains differ in their expected mix of explicitness, synthesis, and citation behaviors.
- Most tasks require 4 to 5 distinct quality dimensions simultaneously, encouraging balanced capabilities rather than single-axis optimization.
B.2 Performance Stratified by Complexity Dimension
Rubric compliance declines as logical nesting increases, while conceptual breadth is associated with a milder difficulty increase. Gemini DR leads across complexity tiers, followed by ChatGPT DR and Perplexity DR.
- Gemini DR achieves roughly 65–70% average rubric compliance across most complexity tiers, leading ChatGPT DR and Perplexity DR.ChatGPT DR reaches around 60–65%, while Perplexity DR remains near 50%, across binary and ternary grading.
- Performance degrades monotonically with increased logical nesting depth, with sharp drops on multi-step analytical or evaluative problems.The decline is particularly pronounced for models relying on retrieval-centric architectures.
- Conceptual breadth correlates with difficulty less steeply than logical nesting, while systems handle multi-domain synthesis better than extended inferential chaining.
B.4 Misclassification Failures in Human-LLM Judge Alignment during Auto-Evaluation
Figure 15 examines disagreements between LLM-as-a-judge and human evaluators across rubric categories and importance levels. Figure 14 shows failure-rate distributions across the benchmark’s three complexity axes within each domain.
- Grading mismatches are compared across rubric categories, mandatory-versus-optional importance, and normalized category mismatch rates.The comparison covers both binary and ternary classification settings.
- Failure rates are distributed across conceptual breadth, logical nesting, and exploration axes within each domain.
C. Prompt and Response Length Analysis
Response-length analysis compares three Deep Research agents and relates verbosity to rubric compliance. Longer outputs generally score higher, while model response lengths differ substantially.
- C.2 Response Length and Compliance: Figures 18a–18d compare response length in tokens and words with overall rubric compliance for Gemini DR, ChatGPT DR, and Perplexity DR.
- C.2 Response Length and Compliance: Gemini produces 7,500–7,600-word responses on average, ChatGPT 6,300–6,400, and Perplexity approximately 1,800 words.High standard deviations above 2,000–3,000 words indicate substantial prompt-dependent variation.
- C.2 Response Length and Compliance: Figure 15 organizes binary and ternary judge–human mismatches by rubric category and importance, including mandatory and optional criteria.
- C.2 Response Length and Compliance: The ternary mismatch views include category distributions, importance proportions, and rubric-type labels for mandatory and optional criteria.
- C.2 Response Length and Compliance: Moderate positive correlations of r ≈0.20 −0.28 for Gemini and ChatGPT indicate that longer responses generally achieve higher scores.Perplexity DR, which produces the shortest outputs, has the lowest correlations.
- C.2 Response Length and Compliance: Criterion-based scoring partly reflects informational density, because longer reports cover more rubric criteria rather than simply receiving stylistic preference.
C.1 Prompt Word Count Analysis
ResearchRubrics analyzes prompt length as a proxy for task complexity and documents the prompts and templates used for reproducible evaluation and rubric preprocessing. Longer prompts align with greater complexity.
- C.1 Prompt Word Count Analysis: Prompt lengths average 87.6 ± 58.6 words, with a median of 68 and a range of 13–315 words.Most prompts are below 100 words, but a long right tail extends beyond 200 words.
- C.1 Prompt Word Count Analysis: General Consumer Research, Technical Documentation, and Business Planning & Research have the longest average prompts, often exceeding 100 words.AI & ML, Current Events, and Other tend to use more concise prompts.
- C.1 Prompt Word Count Analysis: Higher conceptual breadth, deeper logical nesting, and greater exploration are systematically associated with longer prompts, often doubling average length.
- E. Prompts: Two prompt templates support ablation experiments: one for example removal and one for rubric augmentation.Both templates are shown for reproducibility.
- E. Prompts: The benchmark includes prompt domains and word-count statistics across domains and complexity dimensions.
- E. Prompts: The LLM-as-a-judge prompt evaluates each rubric criterion using Not Satisfied, Partially Satisfied, and Satisfied scores of 0.0, 0.5, and 1.0.
- E. Prompts: The example-removal template deletes only example portions while preserving all other rubric text, formatting, punctuation, and structure exactly.