PaperlayerView exact revision on arXiv

Source-linked AI summary

Legal Research Bench: Measuring End-to-End Reliability in Long-Horizon Legal Research Agents

Katrina Drozdov, Oliver Chen, Langston Nashold, Rayan Krishnan

arXiv:2610.00609v1cs.AIcs.CLcs.CY

TL;DR

Legal research agents need to produce complete, authority-grounded answers, but existing legal benchmarks often measure average correctness or partial credit. LRB evaluates expert-authored research questions with all-pass grading and source verification; the strongest tested model reaches only 42.9% all-pass, with results varying by task setting.

  • Problem

    Legal research requires grounded answers supported by valid authority, while most existing legal benchmarks report average correctness or partial credit rather than complete research reliability.

    Source: §1 · p. 1

  • Method

    LRB evaluates tool-using agents on 413 expert-authored open-ended U.S. legal research questions using all-pass rubrics, source verification, and an LLM judge validated against attorneys.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    The strongest tested model reaches 42.9% all-pass, failures vary across practice areas and reconciliation questions are especially difficult, and additional turns and tool calls do not reliably predict accuracy.

    Source: §7 · pp. 8–9

  • Takeaways & Limitations

    Legal research agents should be judged by whether they consistently produce work lawyers can safely rely on, not merely plausible research.

    Source: §7 · p. 9

    LRB covers U.S., English-language research with objective answers, excluding legal work without a single correct answer, such as strategy, client counseling, and negotiation.

    Source: §6 · p. 7

Abstract

from arXiv · show

Legal research is a core and time-consuming legal workflow. Lawyers must identify controlling authority, verify that it remains valid, reconcile statutes and cases, and synthesize a grounded answer. Language model agents are a natural fit for this retrieval-intensive workflow, and automating even part of it would be valuable. But that value depends on reliability: a single missing authority, stale citation, or wrong legal conclusion can make an otherwise plausible answer unusable. We introduce \textbf{Legal Research Bench} (LRB), a benchmark of 413 open-ended U.S. legal research questions written by experts, each paired with a gold answer, supporting authorities, and a binary grading rubric. We evaluate thirteen frontier models in a harness with web search, case-law search, page parsing, and retrieval tools. We score agent responses through all-pass grading with source verification, where a response is correct only if every required criterion is satisfied and its cited authorities verify. We also validate the LLM judge against expert attorneys ensuring that benchmark scores track attorney judgment. Agents remain far from reliable: among the models we tested, the strongest, Claude Opus 4.8, is fully correct on 42.9\% of questions. Performance also varies substantially by task setting: all-pass rates differ across areas of law and are lower on questions requiring reconciliation of conflicting authorities. Across models, more turns, tool calls, and inference cost do not predict higher accuracy.

1 Introduction

Legal Research Bench addresses the gap between partial-credit legal benchmarks and the need for complete answers supported by valid authority. It evaluates tool-using agents with all-pass grading and finds reliability remains limited.

1 Introduction

  • The strongest model reaches 42.9% all-pass, while reconciliation questions are especially difficult and greater agentic effort does not reliably predict higher accuracy.Failures also vary across areas of law.Source: §1 · p. 2
  • LRB introduces 413 expert-authored open-ended U.S. legal research questions with gold answers, verified authorities, and binary rubrics.Responses count as correct only when every required criterion is met and cited authorities verify.Source: §1 · pp. 1–2
  • The benchmark evaluates tool-using agents that search the web and case law, parse and fetch pages, recall gathered content, and submit final answers.Figure 1 depicts the evaluation setup, with runs capped at three hours.Source: §1 · p. 2, Figure 1 · p. 2
  • The study validates its LLM judge against expert attorneys before using it to score outputs.Source: §1 · p. 2

2 Related work

Prior legal benchmarks generally target static tasks or partial-credit outcomes, while long-horizon agent benchmarks emphasize tool-grounded behavior. LRB measures open-ended legal research outcomes and validates its judge against expert attorneys.

Legal benchmarks

  • Most legal NLP benchmarks assess static tasks, while newer agent benchmarks differ in domain or use bounded matter-specific documents rather than open-ended U.S. legal authority.Harvey’s Legal Agent Benchmark also uses all-pass scoring, but its task setting differs from LRB’s open-ended research.Source: §2 · pp. 2–3

Long-horizon agents

  • Long-horizon benchmarks favor outcome-level assessment because different trajectories can reach the same answer, so LRB scores submitted answers rather than paths.Source: §2 · p. 3

Judges and measurement validity

  • Model-judge scores require validation against expert labels, and LRB validates its all-pass judge against expert attorneys before scoring.Related measurement work emphasizes explicit constructs and procedures.Source: §2 · p. 3

3 Legal Research Bench

LRB pairs attorney-authored, objectively answerable legal research questions with curated sources and binary criteria for all-pass grading. Its dataset spans multiple legal areas and question types, and its primary metric requires every criterion and source check to pass.

3.1 Data collection

  • Practicing U.S. attorneys authored and reviewed questions with gold answers, curated sources, and binary rubrics, using novel fact patterns and dated answers when authority could change.Fifteen attorneys averaging 15 years of experience contributed; building each item required about four attorney-hours.Source: §3.1 · p. 3

3.2 Dataset

  • The 413-question corpus spans eight areas of law and ten question types, with overlapping reasoning categories and difficulty attributes including reconciliation and temporal validity.Questions are multi-label, and the corpus includes public, validation, and held-out test partitions; Table 1 summarizes its composition.Source: §3.2 · pp. 3–4, Table 1 · p. 4

3.3 Rubric and metrics

  • All-pass is the primary metric: every attorney-defined necessary criterion must be satisfied, and an invalid cited URL fails the source check.Weighted pass is reported only as a secondary partial-credit measure.Source: §3.3 · p. 4

4 Experimental setup

Thirteen frontier models are evaluated as tool-using agents under a three-hour per-question budget. GPT-5.4 is selected as judge after calibration against expert attorneys, with task-level agreement close to the human baseline.

4.1 Agents

  • Agents use web and case-law search, page parsing, document retrieval, and final-answer tools under a three-hour wall-clock cap per question.The cap is far from binding in practice: 95% of runs finish within roughly 50 minutes.Source: §4.1 · p. 4

4.2 Judge calibration

  • Table 2 compares judge agreement, κ, false-positive and false-negative rates, and pass rates against attorneys; GPT-5.4’s all-pass agreement with experts is 86.7%, versus the 87.9% human baseline.GPT-5.4 is selected for its balanced errors and pass rate close to the attorney majority.Source: Table 2 · p. 5, §4.2 · p. 5
  • All three candidate judges meet or exceed the human item-level agreement baseline, with κ ranging from 0.708 to 0.726.The calibration study covered 341 rubric items from 45 responses to 15 questions; attorney agreement with the majority was 83.4% (κ = 0.644).Source: §4.2 · p. 5

5 Results

Across the full corpus, even the strongest model is fully correct on fewer than half of questions, and partial credit often masks substantive omissions. Reliability varies by area and reconciliation demands; more effort does not reliably predict accuracy, while removing tools sharply reduces performance.

5.1 Reliability remains limited

  • 42.9% is Claude Opus 4.8’s all-pass rate, the highest across models; the weakest model reaches 12.6%, while weighted pass ranges from 68% to 86%.The gap between weighted and all-pass reflects answers missing at least one required element.Source: §5.1 · p. 5

5.2 Partial correctness hides substantive omissions

  • 29.5% of questions have at least one missed high-value criterion, and 4.3% of runs fail source verification despite satisfying every rubric criterion.High-value misses account for roughly 41% of all failures; other omissions often involve required statutes, rules, authority, or legal interpretation.Source: §5.2 · p. 5

5.3 Reliability varies systematically

  • 44.2% in Health versus 14.9% in Family marks the area range; reconciliation questions pass at 20.4% versus 29.9% without that attribute, a 9.4-point gap.The reconciliation penalty holds for every model and persists when controlling for rubric length and source count; temporal-validity questions are near the benchmark average at 28.7%.Source: §5.3 · p. 6, Figure 2 · p. 7, Figure 3 · p. 7

5.4 More turns, tool calls, and cost do not predict reliability

  • More turns, tool calls, and inference cost do not reliably predict correctness; Claude Opus 4.8 leads with 11.7 turns, while Kimi K2.6 averages nearly 100 turns and 112 tool calls but reaches 15.5% all-pass.GPT-5.5 costs $7.34 per question yet trails Opus, which scores higher at roughly 40% of the cost.Source: §5.4 · p. 6

5.5 Tool access is necessary but not sufficient

  • 22.5 points is the average all-pass drop without tools, and every tool-using model outperforms every no-tool model.Claude Opus 4.8 falls 34.1 points, while greater tool use among tool-enabled agents does not predict higher reliability.Source: §5.5 · p. 6

6 Limitations and future work

The benchmark is limited to objective-answer U.S. English research questions and a fixed snapshot of models, tools, and authority. Its scores use model-mediated grading and single runs, and the authors call for broader settings and evaluation of uncertainty and human review.

6 Limitations and future work

  • LRB covers U.S., English-language research and a fixed snapshot; its objective-answer design excludes work without one correct answer, including strategy, counseling, negotiation, and drafting choices.The reported scores also depend on model-mediated grading rather than full expert adjudication of every response.Source: §6 · pp. 7–8
  • Each model was run once per question, so reported rates capture question-sampling uncertainty but not run-to-run variance.Future work should extend jurisdictions, languages, legal tasks, tool environments, and human-in-the-loop workflows, and evaluate uncertainty, clarification, and deferral to human review.Source: §6 · p. 8

7 Conclusion

Legal Research Bench measures end-to-end reliability through all-pass grading and attorney-validated judging, and the strongest tested model reaches only 42.9% all-pass. Failures vary systematically, especially for reconciliation tasks, while greater turn and tool-call counts do not reliably predict accuracy.

7 Conclusion

  • 42.9% is the strongest tested model’s all-pass rate, despite substantial partial credit across frontier tool-using models.LRB pairs all-pass grading of final answers with judge validation against expert attorneys.Source: §7 · p. 8
  • All-pass rates vary across practice areas, and reconciliation tasks are especially difficult.The conclusion characterizes these failures as systematic rather than incidental.Source: §7 · pp. 8–9
  • Higher turn counts and tool-call counts do not reliably predict higher accuracy; the paper argues agents should be judged by whether lawyers can safely rely on their work.The proposed standard is consistent production of dependable research, not merely plausible answers.Source: §7 · p. 9

Ethics Statement

The ethics statement warns that legal research errors can affect rights and that fluent answers may conceal a wrong central conclusion. It also describes the benchmark’s safeguards and judge-calibration checks.

Ethics Statement

  • Legal research errors can affect liberty, livelihood, and legal rights; the benchmark is not deployment certification, and outputs require licensed-attorney review.The authors specifically warn that fluent, well-structured answers can appear authoritative despite a wrong central legal conclusion.Source: Ethics Statement · p. 9
  • The questions are original hypotheticals without client, personally identifiable, privileged, or confidential information; remaining items are held out to limit contamination.Attorneys who contributed to the benchmark were compensated and consented to use of their work.Source: Ethics Statement · p. 9

A Sample question

  • P-005 asks whether a lessee may stop payments after leased trailers fail and the manufacturer enters bankruptcy; its rubric has 12 weighted criteria.The lease disclaims lessor warranties and directs equipment claims to the manufacturer.Source: A Sample question · p. 11

B Judge calibration: additional analyses

  • GPT-5.4 is selected despite slightly lower κ than Gemini 3.1 Pro Preview because its error balance and pass rate are closer to expert judgment.Its pass rate is 66.3% versus the expert-majority 67.7%; mildly stricter judgments make reported all-pass rates slightly conservative.Source: B Judge calibration: additional analyses · p. 11
  • After excluding four near-uniform questions, GPT-5.4’s κ falls from 0.715 to 0.705; every judge’s κ drops by less than one point, and no judge is most lenient toward its own provider’s models.The harder-item analysis retains 308 comparisons.Source: B Judge calibration: additional analyses · p. 11

C Fine-grained question types and adjusted category effects

Reconciliation remains associated with lower all-pass after adjustment for rubric length, source count, and area of law. Broad reasoning-category rates vary little across models, while paired comparisons support leaderboard tiers rather than a strict ordering.

C Fine-grained question types and adjusted category effects

  • Pooled category- and area-level rates use question-level cluster-bootstrap 95% confidence intervals, carrying all thirteen model outcomes with each resampled question.All-pass has an intraclass correlation of 0.40 within questions and a design effect of 5.8 at thirteen models.Source: C Fine-grained question types and… · p. 11
  • The ten question types map to three overlapping reasoning categories and two difficulty attributes; questions can carry multiple labels, so category counts are descriptive marginals.Table 5 reports pooled all-pass rates and question counts for the types.Source: C Fine-grained question types and… · p. 11, Table 5 · p. 12
  • Figure 5 shows little cross-model variation among the three primary reasoning categories, which sit near the 27.2% overall rate; Reconciliation is the clearest difficulty signal.The figure also breaks out the two difficulty attributes by model.Source: Figure 5 · p. 12, C Fine-grained question types and… · p. 12
  • 0.60 to 0.68: adjusting for rubric length and source count weakens the reconciliation odds ratio, but it remains below one (coefficient −0.380; 95% CI [−0.73, −0.04]).Both controls predict all-pass, yet neither explains the penalty; reconciliation questions average 9.0 criteria versus 9.5 for others.Source: C Fine-grained question types and… · pp. 12–13, Table 6 · p. 13

D Paired model comparisons

  • Only two of twelve adjacent leaderboard pairs differ significantly at the 5% level, supporting performance tiers rather than a strict model ordering.The comparisons use paired per-question outcomes across the full 413-question corpus.Source: Table 7 · p. 13, D Paired model comparisons · p. 13

E Additional results

Results across areas and tool-use analyses show that neither one model nor greater search volume consistently accounts for stronger performance. Central legal errors remain a substantial share of failures.

E Additional results

  • No single model leads across all areas of law; Figure 6 reports the best single-model all-pass rate for each area.The figure summarizes the area-by-model heatmap.Source: E Additional results · p. 13, Figure 6 · p. 14
  • Web search averages about 27 calls per session versus about 11 for case-law search, but call volume does not track accuracy across models.Claude Opus 4.8 balances the two search types at about 8–9 calls each; Gemini 3.1 Pro Preview makes the fewest calls yet ranks tenth.Source: E Additional results · p. 13, Figure 7 · p. 14
  • 28.7 calls per question: Claude Opus 4.8 leads the benchmark, while Kimi K2.6 makes 112.3 calls and GPT-5.4-mini 94.4.Figure 8 plots all-pass against mean tool calls per question; call volume does not track correctness.Source: E Additional results · p. 14
  • Roughly 41% of failures across the corpus are central-wrong errors, where a model misses a high-value [+3] item.The failure decomposition distinguishes these from lower-tier-only failures on [+1]/[+2] items.Source: E Additional results · p. 14

F Answer length and verbosity

Agent answers are generally much longer than expert references, and length’s positive association with all-pass does not establish that verbosity itself helps. A single-question trajectory comparison likewise shows that more tool calls do not guarantee a grounded, complete answer.

F Answer length and verbosity

  • 536 words is the gold-reference mean; Claude Sonnet 4.6 averages roughly 2,320 words, while GPT-5.4-mini averages about 350.Only GPT-5.4-mini falls below the reference length.Source: F Answer length and verbosity · p. 14
  • Spearman ρ ≈0.7 links answer length with all-pass, but GPT-5.5 has the second-highest all-pass with one of the shortest answers, about 990 words.The authors suggest the association tracks underlying capability rather than verbosity itself; extra length appears largely structural and expository.Source: F Answer length and verbosity · p. 15, Figure 10 · p. 16

G Agent trajectories

  • The P-005 trajectories illustrate distinct failure modes: incomplete but sourced, complete but ungrounded, and failure to reach primary authority.Claude Opus 4.8 uses 38 calls, Kimi K2.6 uses 174, and Grok 4.3 relies on web search alone.Source: G Agent trajectories · p. 16
  • On P-005, trajectory length ranges from 19 to 174 calls, yet the five full-score models cluster between 37 and 45 calls.Claude Opus 4.8 cites eight verified authorities but misses a criterion; Kimi K2.6 meets every criterion but fails source verification, while Grok 4.3 never searches CourtListener.Source: G Agent trajectories · p. 16, Figure 11 · p. 17
Loading 2610.00609v1…