ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents

Introduction

• Existing QA benchmarks [21] [29] [13] miss long-form synthesis, while some DR benchmarks raise circularity concerns [8] [7] or cover narrow technical tasks [15] [20] [24].The introduction motivates broader evaluation because real-world queries span topics such as business reports and consumer questions.Source: §1 · pp. 1–2
• The framework characterizes deep-research queries by conceptual breadth, logical nesting depth, and exploration or ambiguity, exposing complexity that simpler QA benchmarks do not capture.These axes describe topic diversity, reasoning steps such as sub-questions and conditionals, and goal underspecification.Source: §1 · p. 2
• OpenAI’s DeepResearch, Gemini Deep Research, and Perplexity’s Deep Research all score below 68% average rubric compliance, exposing weaknesses in synthesis and rigorous justification.The paper identifies missed implicit context and inadequate reasoning about retrieved information as areas for improvement.Source: §1 · p. 2
• ResearchRubrics pairs prompts across nine domains with 2,593 human-written and reviewed criteria covering grounding, reasoning, completeness, relevance, clarity, and penalties for extraneous or incorrect content.The benchmark is designed to reflect real-world use cases and domain-specific requirements.Source: §1 · p. 2
• Outcome-based, fine-grained rubrics assess long-form answers against expert judgments and distinguish mandatory sufficiency criteria from optional criteria.The benchmark also includes negative rubrics to penalize extraneous or incorrect content.Source: §1 · p. 2
• The authors propose ternary grading to assign partial credit to rubric criteria and examine its suitability for automated evaluation.Their evaluation protocols compare binary and ternary grading for each criterion.Source: §1 · p. 2
• Rubric-design recommendations, validated through ablations, improve agreement between LLM-as-a-judge evaluations and human evaluators.The paper also experiments with how much detail rubrics should contain.Source: §1 · pp. 2–3

Related Work

• Earlier benchmarks mainly derive or construct tasks from static corpora or rely on expert-curated questions, while some newer work uses dynamic real-world scenarios.AcademicBrowse [30] and BrowseComp [26] assess retrieval, ResearchBench [16] derives complex queries from static data, and DeepScholar-Bench [20], ReportBench [15], and DeepResearch Arena [24] illustrate dynamic-task approaches with specialization, replication, or automatic-rubric limitations.Source: §2 · p. 3
• Expert-curated benchmarks include short-answer questions or long-form reports, but may not directly test open-ended analysis or rely on weak rubric and reference-based evaluation.Humanity’s Last Exam [21] targets advanced-domain short answers; DeepResearch Bench [8] uses LLM-generated specialized-domain rubrics, reference reports, and overlap metrics.Source: §2 · p. 3
• Newer expert-led benchmarks ease evaluation by narrowing domains or prompt scope; ExpertLongBench [22] uses existing references with CLEAR, limiting prompts to highly academic or professional queries.Source: §2 · p. 3
• RESEARCHRUBRICS pairs realistic academic and everyday research queries with human-written rubrics for grounding, synthesis, reasoning, clarity, and citation use, evaluated by scalable LLM judges.It complements ExpertLongBench while emphasizing domain diversity, rubric quality, and deep-research-specific tasks.Source: §2 · p. 4

3. Overview of RESEARCHRUBRICS

• The benchmark contains 101 single-turn prompts, each paired with 20–43 expert-written criteria, for 2,593 unique rubric items.Experts iteratively refined every prompt and criterion; none were seeded or generated by LLMs.Source: §3 · p. 4

3.1 Data Collection and Task Domains

• A three-expert pipeline proposes, iteratively reviews, and independently finalizes each prompt and rubric, with participants working in familiar domains.Experts are defined by strong STEM backgrounds and task-design and evaluation skills, rather than specialization in each prompt’s domain.Source: §3.1 · pp. 4–5
• Prompts adapted from forums, Q&A sites, and brainstorming span nine categories and vary in breadth and depth; AI/ML and historical analysis are the largest categories.The distribution is fairly even, with additional categories covering consumer research, creative synthesis, and real-time news retrieval.Source: §3.1 · p. 5

3.2 Prompt Complexity Dimensions

• Tasks receive Breadth, Depth, and Ambiguity labels, operationalizing conceptual breadth, logical nesting depth, and exploration for performance analysis and task filtering.The dimensions capture differences such as integrating many sources versus following long reasoning chains.Source: §3.2 · p. 5

3.3 Rubric Design

• Tailored rubrics separate mandatory sufficiency requirements from optional qualities, with criterion weights from −5 to 5 rewarding valued attributes and penalizing common failures.Weights of ±4 or ±5 indicate mandatory criteria; [−3, 3] indicates optional criteria, calibrated to a six-level human preference scale.Source: §3.3 · pp. 5–6

3.4 Evaluation Methodology

• A model-based judge assigns Satisfied, Partially Satisfied, or Not Satisfied verdicts per criterion, and the task score is the weighted sum normalized by total positive weight.Negative criteria apply their negative weights when met; partial satisfaction contributes 0.5 and non-satisfaction 0 to the criterion indicator.Source: §3.4 · p. 7
• Failure breakdowns count only Not Satisfied verdicts and average category rates across tasks containing that category, so the ratios need not sum to 1.Macro F1 across the three verdict classes compares model graders with expert judgments; grader-alignment factors are further examined through ablations.Source: §3.4 · p. 7

4.1 Experimental Setup

• The study benchmarks OpenAI, Gemini, and Perplexity Deep Research across six dimensions using binary and ternary grading, with three LLM judges compared against nine expert annotators.The human ground truth covers 303 responses; binary grading treats partially satisfied criteria as not satisfied.Source: §4.1 · p. 8

4.2 Main Results

• 67.7% ternary and 61.5% binary compliance for Gemini DR were the highest scores, and no evaluated system exceeded 70%.The comparison uses ternary and binary grading of overall rubric compliance.Source: Table 5 · p. 8, §4.2 · p. 8
• 45–50% of failures involve implicit reasoning and synthesis, while optional criteria account for most implicit-reasoning failures and mandatory criteria drive failures in explicit requirements and synthesis.Agents have failure rates below 20% for explicit factual retrieval and communication quality; the mandatory/optional distinction also contextualizes HealthBench’s 33% average-to-minimum performance degradation.Source: §4.2 · p. 8, Figure 5 · p. 9, Figure 8 · p. 10
• Performance falls with deeper logical nesting: tasks exceeding four sequential inference steps or 35 minutes of human-equivalent time show universal collapse, while domain results range from 76% consulting coverage to below 40% technical nugget coverage.Gemini leads across most complexity tiers at roughly 70% average compliance, with ChatGPT next and Perplexity slightly behind; depth degrades performance more than breadth, and organizational-structure scores reach 85%.Source: §4.2 · pp. 8–9, Figure 6 · p. 9, §4.5 · p. 12
• r ≈0.24–0.28 for Gemini and ChatGPT links response length with higher scores, while criterion-level analysis says this reflects information density, possible verbosity bias, or both.Reports are 10–100 times longer than standard LLM responses; the analysis says distinguishing the explanations requires varying length while holding information content constant.Source: §4.2 · p. 9, Figure 7 · p. 10, §4.5 · p. 12

4.5 Discussion: Systematic Patterns and Their Implications

• 111 citations at 81% accuracy for Gemini and 31 citations at 90% accuracy for Perplexity illustrate a breadth–accuracy trade-off that the paper attributes to unresolved architectural constraints.The discussion characterizes the systems as unable to balance comprehensive coverage and precision, and relates persistent failures to multi-hop reasoning limits and reasoning-action mismatch.Source: §4.2 · p. 10, §4.5 · p. 12

4.3 Human-LLM Judge Alignment for Auto-Evaluation

• 0.72–0.76 Macro F1 agreement is achieved with binary grading, about 20 percentage points above ternary grading; Gemini-2.5-Pro reaches 0.76, still 12–17 points below best human agreement.The study reports that this level of consistency supports automated evaluation of the benchmark’s 2,593 criteria.Source: §4.3 · p. 10, Table 6 · p. 11

4.4 Rubric Design Impact

• Concrete rubric examples improve human–model alignment by 3–4% under binary grading and 2–3% under ternary grading, while LLM-based augmentation reduces alignment by 15–20%.The paper attributes the augmentation decline to semantic drift and emphasis distortion, finding concise expert-authored rubrics with targeted examples outperform machine-generated elaboration.Source: §4.4 · p. 11, Table 7 · p. 11

Conclusion and Future Work

• ResearchRubrics evaluates 101 diverse research challenges using expert-written criteria that assess factual recall, completeness, reasoning soundness, source usage, and clarity.The benchmark offers a multidimensional assessment of deep research agents rather than focusing on factual recall alone.Source: §5 · p. 12
• Today’s best agents achieve only around 67% compliance, with recurring failures in implicit reasoning, multi-document synthesis, and sustained sequential reasoning pointing to a need for architectural innovation.Agents often struggle to integrate information across documents and provide well-justified answers with proper citations.Source: §5 · p. 12
• Rubric granularity reveals capability gaps hidden by aggregate metrics, while mandatory and optional criteria place agents on a sufficiency–excellence continuum to inform deployment decisions.This distinction emphasizes minimum viable performance rather than average scores.Source: §5 · p. 12

A. Extended Related Work

• Surveys by Huang et al. [12] and Xu and Peng [27], co-scientist systems [10], and SPOT [23] underscore the need to evaluate research reasoning and conclusions against expert scrutiny.Co-scientist systems generate and refine hypotheses or experimental plans, extending beyond information retrieval; SPOT checks AI-generated scientific papers for logical errors or inconsistencies.Source: A. Extended Related Work · pp. 15–16
• Static-corpus benchmarks AcademicBrowse [30], BrowseComp [26], and ResearchBench [16] test retrieval but constrain scope to answer keys and risk leakage or staleness.AcademicBrowse centers on academic-paper queries, while BrowseComp contains over 1,200 multi-hop web questions; static data can leave answers in training data or miss new information.Source: A. Extended Related Work · p. 15
• HLE [21] emphasizes expert short-answer questions, while DeepResearch Bench [8] and ExpertLongBench [22] assess long-form tasks with reference-dependent scoring.DeepResearch Bench combines reference-based metrics and citation counts; ExpertLongBench's CLEAR framework compares structured checklists from model outputs and gold references, requiring high-quality reference outputs.Source: A. Extended Related Work · p. 15
• Dynamic benchmarks extend evaluation to current literature and exploratory prompts, but retain limits in task scope, reference dependence, or automatically generated rubrics.DeepScholar-Bench [20] specializes in academic writing and uses automated metrics; ReportBench [15] favors overlap with survey content; DeepResearch Arena [24] draws over 10,000 seminar-derived tasks across 12 disciplines but may miss domain nuances through automatic rubric generation.Source: A. Extended Related Work · p. 15
• RESEARCHRUBRICS pairs realistic academic and everyday queries with expert-written rubrics for grounding, synthesis, reasoning, clarity, and citations, scored by LLM judges.The authors position it between specialized expert-task benchmarks and massive automatically generated suites, using manually quality-checked criteria rather than simplistic overlap measures.Source: A. Extended Related Work · p. 16

• The appendix analyzes composition, complexity, and error structure, and clarifies how output length relates to rubric compliance.Source: B. Extended Results · p. 16

B.1 Benchmark Composition and Rubric Coverage

• 4.74 axes per task on average, with most tasks covering 4 to 5 dimensions simultaneously.This multi-axis coverage reflects the goal of measuring holistic research ability rather than single-skill performance.Source: §B.1 · p. 16, Figure 9 · p. 16
• 20–43 criteria per task, with a mean of approximately 26.Source: §B.1 · p. 16, Figure 10 · p. 17
• Domains differ in their expected mix of explicitness, synthesis, and citation behaviors.Figure 11 decomposes the proportions of the six rubric axes by domain.Source: §B.1 · p. 16, Figure 11 · p. 17

B.2 Performance Stratified by Complexity Dimension

• Performance degrades monotonically as logical nesting increases, with sharp drops on multi-step analytical or evaluative problems, especially for retrieval-centric models.Shallow single-hop or two-step queries are handled well.Source: §B.2 · p. 18
• Conceptual breadth also correlates with difficulty, but less steeply than nesting; systems handle multi-domain synthesis better than extended inferential chaining.Source: §B.2 · p. 18
• Gemini DR achieves roughly 65–70% average rubric compliance across most tiers, ahead of ChatGPT DR at around 60–65% and Perplexity DR near 50%.The comparison holds under both binary and ternary grading schemes.Source: §B.2 · pp. 17–18, Figure 12 · p. 18, Figure 13 · p. 18

B.3 Domain-wise Failure Structure

• Figure 14 presents failure rates across the complexity axes within each domain, but the supplied passage does not describe the domain-specific distributions.Source: §B.3 · p. 18

B.4 Misclassification Failures in Human-LLM Judge Alignment during Auto-Evaluation

• Implicit Criteria accounted for most misclassifications, while category-normalized disagreement rates were slightly higher for References & Citation Quality and Synthesis of Information.The analysis compared mismatch patterns across binary and ternary classification settings; the higher normalized rates suggest difficulty judging adequate mentions of references or arguments.Source: §B.4 · p. 18
• Mandatory criteria had a lower proportion of mismatches, indicating closer alignment between the model and human raters on mandatory response aspects.Source: §B.4 · p. 18

C.2 Response Length and Compliance

• Gemini averaged 7,500–7,600 words, ChatGPT 6,300–6,400, and Perplexity about 1,800 in Markdown responses.The differences were consistent across words and tokens, and between text and rendered formats.Source: §C.2 · p. 19
• Response lengths had high variance, with standard deviations above 2,000–3,000 words, reflecting substantial prompt-dependent variation in verbosity.Source: §C.2 · p. 19
• Across 101 tasks, prompt lengths were right-skewed, with a mean of 87.6 words and a median of 68 words.Source: Figure 16 · p. 21
• Longer prompts were consistently associated with higher levels of conceptual breadth, logical nesting, and exploration.Source: Figure 17 · p. 21
• Gemini and ChatGPT response lengths had moderate positive correlations with rubric compliance (r ≈0.20 −0.28), while Perplexity showed the lowest correlations.The analysis examined response length in tokens and words against overall rubric compliance across the three agents.Source: §C.2 · p. 21
• The length–quality association may reflect broader rubric coverage rather than evaluator preference for verbosity, while criterion-based scoring means it partly captures genuine informational density.Source: §C.2 · p. 21

C.1 Prompt Word Count Analysis

• Prompts averaged 87.6 ± 58.6 words (median 68; range 13–315), with most below 100 words and a long right tail.Source: §C.1 · p. 19
• General Consumer Research, Technical Documentation, and Business Planning & Research have the longest average prompts, often exceeding 100 words; AI & ML, Current Events, and Other are more concise.Source: §C.1 · p. 19, Table 8 · p. 22
• Prompts with greater conceptual breadth, deeper logical nesting, and more exploration are systematically longer, often doubling in average length versus simpler tasks.The analysis treats prompt length as a useful proxy for complexity because longer prompts tend to include more context, sub-questions, and open-ended reasoning requirements.Source: §C.1 · p. 19

E. Prompts

• Appendix E provides the prompt used for grading with the LLM-as-a-judge framework.Source: E. Prompts · p. 21, Figure 20 · p. 26
• The ablation experiments use separate prompt templates for example removal and rubric augmentation, both shown for reproducibility.Source: E. Prompts · pp. 21–22, Figure 19 · p. 25, Figure 21 · p. 27
• Appendix E also includes per-model word and token statistics, prompt-domain information, and a comparison of length versus score across token and word counts in binary and ternary settings.Source: Table 9 · p. 22, Table 10 · p. 23, Figure 18 · p. 24