PaperlayerView exact revision on arXiv

Source-linked AI summary

A Reasoning-Focused Legal Retrieval Benchmark

Lucia Zheng, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D. Manning, Peter Henderson, Daniel E. Ho

arXiv:2505.03970v1cs.CL

TL;DR

Realistic legal RAG benchmarks must test both retrieval and downstream legal question answering, but existing benchmarks leave important gaps. This paper introduces Bar Exam QA and Housing Statute QA using lawyer-like annotation processes and finds that reasoning-focused retrieval is more effective than simple lexical retrieval. The benchmarks remain limited in domain coverage and answer format, so they do not represent the full complexity of real-world legal tasks.

  • Problem

    Existing legal RAG benchmarks do not adequately capture realistic retrieval challenges together with downstream legal question answering.

  • Method

    The paper constructs Bar Exam QA and Housing Statute QA from substantive legal questions, supporting passages, and human annotations modeled on legal research.

  • Results

    Generative query expansion with reasoning roll-outs improves retrieval performance, while lexically focused retrievers struggle on these benchmarks.

  • Takeaways & Limitations

    Legal retrieval systems may need reasoning capabilities, such as query expansion or stronger embeddings, rather than relying only on simple retrievers.

  • Takeaways & Limitations

    The datasets are restricted in subject-matter domains and multiple-choice answer forms and may not represent the full natural distribution of legal questions.

Abstract

from arXiv · show

As the legal community increasingly examines the use of large language models (LLMs) for various legal applications, legal AI developers have turned to retrieval-augmented LLMs ("RAG" systems) to improve system performance and robustness. An obstacle to the development of specialized RAG systems is the lack of realistic legal RAG benchmarks which capture the complexity of both legal retrieval and downstream legal question-answering. To address this, we introduce two novel legal RAG benchmarks: Bar Exam QA and Housing Statute QA. Our tasks correspond to real-world legal research tasks, and were produced through annotation processes which resemble legal research. We describe the construction of these benchmarks and the performance of existing retriever pipelines. Our results suggest that legal RAG remains a challenging application, thus motivating future research.

1 Introduction

Legal RAG lacks realistic benchmarks that test both difficult legal retrieval and downstream question answering. The paper introduces two human-annotated benchmarks and finds that low lexical overlap challenges common retrievers, while reasoning-based query expansion improves performance.

  • Benchmark gap: Legal RAG benchmarks often fail to represent low-overlap queries requiring multi-hop or analogical reasoning.Such retrieval is common when matching factual circumstances to cases addressing higher-order legal issues.
  • Benchmark gap: Existing benchmarks often omit paired question-answer tasks, limiting evaluation of downstream reasoning from retrieved information.Retrieval-only benchmarks cannot capture how retriever improvements affect downstream answers.
  • Contributions: The paper introduces Bar Exam QA and Housing Statute QA with approximately 10K labeled query, gold-passage, and answer examples.Bar Exam QA uses bar-exam hypotheticals and judicial opinions; Housing Statute QA uses housing-law questions and statutes across jurisdictions.
  • Contributions: Gold passages are hand-annotated and validated by law students and researchers through processes modeled on lawyers’ legal research.The datasets are designed to represent substantive legal retrieval rather than merely extractive citation relationships.
  • Findings: Common retrieval methods such as BM25 struggle with the datasets’ low query-document lexical similarity.The paper evaluates existing state-of-the-art retrieval pipelines and uses this setting to test legal reasoning in retrieval.
  • Findings: The authors suggest that legal RAG retrievers may need reasoning capabilities through query expansion or stronger embedding models.Their law-inspired query expansion strategy uses generative reasoning roll-outs and improves performance on the datasets.

2 Related Works

Existing general and legal retrieval benchmarks often use easier query-document relationships or fail to support end-to-end legal question answering. The paper positions its datasets as reasoning-focused legal benchmarks with natural questions, expert annotations, and downstream answer evaluation.

  • General IR and QA: BEIR comprises 18 datasets across 9 information-retrieval task types, including question-answering datasets similar to the paper’s benchmarks.Unlike extractive-span QA in Natural Questions and HotpotQA, these benchmarks use multiple-choice answers.
  • General IR and QA: BIRCO and BRIGHT use longer queries and lower lexical similarity than BEIR but do not include comparable legal reasoning tasks.Their domains include natural sciences, computer science, and theorem-based mathematics.
  • Legal retrieval datasets: Earlier statutory-law retrieval systems focused on lexical matching and extensive semantic-feature annotation.This line of work differs from the paper’s emphasis on reasoning-intensive retrieval benchmarks.
  • Legal retrieval datasets: Automatically extracted legal citation datasets can have high query-passage lexical similarity because citing contexts summarize rules from cited cases.Their judge-written queries may also poorly reflect natural questions asked by legal-information seekers or lawyers.
  • Legal retrieval datasets: Few legal retrieval datasets pair retrieval with downstream open-domain QA, and evaluating factual recall in open-ended generation remains difficult.The paper uses multiple-choice QA to make factual correctness easier to evaluate automatically.

3 Datasets

The benchmarks pair substantive legal questions with supporting passages and answers, using expert-informed construction across bar-exam and housing-statute domains. They support retrieval and downstream QA evaluation but remain bounded by domain, format, and realism constraints.

  • Benchmark design: The datasets provide substantive legal questions paired with supporting gold passages and answers for end-to-end evaluation.This enables more fine-grained assessment than datasets designed only for retrieval or only for question answering.
  • Benchmark design: Legal experts, bar-exam writers, law students, and legal researchers hand-annotated questions and passage labels.The annotation processes were intended to resemble legal research rather than extractive citation construction.
  • Benchmark design: The retrieval corpora are substantially larger for these benchmarks, particularly Housing Statute QA, reaching approximately 1–2M documents.Larger corpora increase retrieval difficulty by increasing the relative fraction of irrelevant documents.
  • Limitations: The datasets are restricted in subject-matter domains and multiple-choice answer forms, so they cannot capture the full spectrum of real-world legal-task complexity.Bar-exam questions use stylized, fictional, short fact patterns that may differ from attorneys’ real-world fact patterns.
  • Bar Exam QA: Bar Exam QA contains multistate bar-exam questions with novel scenarios, issue-specific questions, four answer choices, and gold explanation passages.Historical questions receive law-student annotations when answer explanations are unavailable.
  • Housing Statute QA: Housing Statute QA covers statutory housing law across 50+ U.S. jurisdictions using Yes/No questions and up to 10 relevant statutes per sample.Its retrieval task identifies relevant statutes from a larger state-law database.

4 Comparison to Existing Tasks

Bar Exam QA and Housing Statute QA differ from existing general and legal IR tasks in lexical similarity, making retrieval more reasoning-intensive. The paper compares these task properties and baseline retrieval performance across datasets.

  • Existing QA benchmarks often closely match questions, gold passages, and answers lexically, whereas these legal tasks require inference from rules and facts.
  • The benchmark comparison uses NQ and HotpotQA, COLIEE, and CLERC, with lexical-similarity distributions and baseline performance as complexity measures.
  • TF-IDF cosine similarity is used to compare query–gold-passage and gold-passage–answer relationships because it is closely related to BM25.
  • Query–gold-passage similarities average 0.25–0.27 for NQ, HotpotQA, COLIEE, and CLERC, versus 0.07 for Bar Exam QA and 0.08 for Housing Statute QA.
  • Housing Statute QA includes transformed Yes/No questions, so lexical similarity of the original answers is likely higher for that subset.

5 Evaluation

The evaluation tests lexical and dense retrievers, query-expansion strategies, and downstream QA systems on the two legal benchmarks. It includes structured legal reasoning rollouts designed to represent higher-order legal issues and rules.

  • Baseline retrieval evaluates BM25 and E5-family dense models, including E5-small-v2, E5-base-v2, E5-large-v2, and E5-mistral-7b-instruct.
  • Bar Exam QA retrieval uses the full passage corpus, while Housing Statute QA retrieves from jurisdiction-specific statute pools ranging from 10,676 to 155,974 passages.
  • The study compares baseline retrieval with paraphrasing, Chain-of-Thought, and structured legal reasoning query expansion.
  • Structured legal reasoning expands queries with generated reasoning rollouts about higher-order legal hierarchies, including rules implicated by query facts.
  • Downstream QA evaluation uses Llama 3 8B Instruct and GPT-4o-mini with no passage, retrieved passages, reasoning rollouts as pseudo-passages, or annotated gold passages.
  • Figure 3 reports Recall@10 gains for structured reasoning rollout retrieval with 95% bootstrap confidence intervals; Housing Statute QA uses retrieval of at least one gold passage as an upper bound.

6 Results

The benchmarks are difficult for baseline retrievers, particularly when query–passage lexical similarity is low, while structured reasoning query expansion improves retrieval across models. Retrieval gains do not reliably translate into downstream QA gains, which remain constrained by models’ ability to reason over retrieved passages.

  • Bar Exam QA and Housing Statute QA produce lower recall for BM25 and E5-large-v2 than comparison tasks, highlighting their retrieval difficulty.BM25 performs well on tasks with higher query–gold-passage lexical similarity, such as NQ and HotpotQA.
  • 6.28 ± 0.99 Recall@10 for BM25 and 8.86 ± 1.16 for E5-large-v2 are the Bar Exam QA gains from structured reasoning query expansion over baseline retrieval.The gains are statistically significant across the evaluated legal tasks, with larger improvements for more lexically focused retrievers.
  • 10.27 ± 1.08 Recall@10 for BM25 and 2.16 ± 1.15 for E5-large-v2 are the Housing Statute QA gains from structured reasoning query expansion over baseline retrieval.The gains are statistically significant for BM25 and the three smaller E5 models, but not for E5-mistral-7b-instruct, which is already highly performant at baseline.
  • Structured reasoning rollouts outperform verbosity-increasing prompting techniques, while paraphrasing does not improve and can slightly hurt Bar Exam QA retrieval.These results suggest that summarization and synonym substitution alone are insufficient for retrieval tasks requiring additional reasoning.
  • Models perform similarly on high-similarity Housing Statute QA examples, but E5-mistral-7b-instruct most clearly outperforms other models on low-similarity examples.The comparison indicates that harder, lower-similarity examples better distinguish retrieval-model capabilities.
  • A 10% retrieval gain can theoretically yield only a 2% downstream improvement when optimal passage use produces at most a 20% gain.The paper reports that improvements are upper-bounded by how well downstream models can use the gold passage, including only a 20% gain for Llama 3 8B Instruct.
  • GPT-4o-mini gains 23.53 percentage points with retrieved passages over no passage on Housing Statute QA, but still struggles on the more challenging Bar Exam QA.On Housing Statute QA, retrieved passages outperform the generative reasoning pseudo-passage: 71.71% versus 68.51% accuracy.
  • Future legal RAG systems may need retrievers with reasoning abilities and downstream models that can reason about retrieved passages.The paper identifies query expansion and larger embedding-model capacities as possible directions for improving retrieval-side reasoning.

7 Conclusion

The benchmark introduces roughly 10K paired legal retrieval examples and finds that legal retrieval tasks remain challenging for lexically focused systems, while reasoning-oriented query expansion can help. The datasets aim to support evaluation of reasoning-focused retrieval-augmented LLMs, but cover only selected areas of legal practice.

  • The benchmark provides roughly 10K paired queries, gold passages, and answers with human-annotated legal evidence.The examples contain substantive legal questions and supporting law, simulating reasoning-intensive real-world legal retrieval tasks.
  • Bar Exam QA and Housing Statute QA cover reasoning-intensive questions using judicial opinion passages and statutes from different jurisdictions.
  • The datasets do not represent the full distribution of practitioners’ legal questions because they cover only Bar Exam areas and statutory housing law.
  • Lexically focused retrievers find these tasks challenging, while generative query expansion that rolls out reasoning can improve retrieval performance.
  • The findings suggest that retrievers may need reasoning capabilities, including through query expansion or greater embedding-model capacity.
  • The benchmarks are intended as resources for understanding retrieval suitability across legal tasks and for future reasoning-focused retrieval-augmented LLM research.

A Bar Exam QA Dataset Construction

Bar Exam QA transforms multistate bar-exam questions into retrieval examples by pairing each legal scenario with a supporting gold explanation passage. For historical questions, law students identify the governing rule, search Westlaw for relevant cases, and annotate a concise rule statement.

  • The historical MBE gold-passage annotation process took roughly 9 months and was modeled on legal research.
  • Law students first identify the relevant rule of law and develop general and specific search terms from secondary legal sources.
  • Students hand-compose Westlaw Terms and Connector queries, review case descriptions and headnotes, and select cases stating the identified rule.
  • The final gold label is a succinct, generalizable statement of the rule of law from the selected case text.

C Dataset Release and Licenses

The benchmark datasets are publicly released through HuggingFace using source-specific licensing arrangements. Release scope differs across subsets, with some historical exam materials gated and the Barbri subset withheld because of copyright concerns.

  • The datasets are publicly available through HuggingFace, including separate releases for Bar Exam QA and Housing Statute QA.
  • The passage pool draws on Cornell LII, public-domain case law, and public-domain Justia materials under permissive licenses.
  • Historical MBE queries and gold passages are released under CC-BY-NC-SA, while multiple-choice options and answers are available to researchers through a gated restrictive-license release.
  • The Barbri subset is not released because of copyright concerns and is instead used as a private, held-out test set with separate evaluation results.
  • Housing Statute QA is released under CC-BY-SA, and LSC permits download and redistribution.

D Dataset Examples

The paper provides representative dataset examples and query-expansion materials, alongside statistical comparisons of lexical-similarity distributions with other retrieval tasks. These materials illustrate how structured reasoning can encode implicit retrieval steps and latent legal issues.

  • The appendix presents representative examples from Bar Exam QA, Housing Statute QA, Natural Questions, HotpotQA, COLIEE, and CLERC.
  • Tables 13 and 14 document the query-expansion prompts used for Bar Exam QA and Housing Statute QA.
  • For a Bar Exam QA question, paraphrasing, chain-of-thought, and structured-reasoning expansions are compared with the gold passage.
  • The structured reasoning prompt expansion encodes implicit retrieval-task steps and captures latent legal issues.
  • Tables 17–20 report statistical tests comparing lexical-similarity distributions for Bar Exam QA and Housing Statute QA with general- and legal-domain retrieval tasks.

H Retrieval Results

The paper reports retrieval evaluations across the two proposed legal QA benchmarks and four comparison tasks. Evaluation coverage includes aggregate and subset-specific Bar Exam QA results, two Housing Statute QA recall definitions, and task-specific retrieval settings.

  • Bar Exam QA: Bar Exam QA retrieval performance is reported for aggregate, Historical MBE, and Barbri subsets.These results appear in Tables 21–23.
  • Housing Statute QA: Housing Statute QA reports recall based on retrieving at least one gold passage and, separately, all gold passages.The at-least-one measure is an upper bound, while the all-passages measure is a lower bound evaluated at k≥10.
  • Comparison Tasks: Natural Questions and HotpotQA have dedicated retrieval evaluations, with HotpotQA recall requiring both gold passages at retrieval depth k=2.Both gold passages are required to answer each HotpotQA question.
  • Comparison Tasks: COLIEE Task 1.1 and CLERC document-setting retrieval performance are also reported.The results appear in Tables 28 and 29.

I Downstream QA Results

The downstream evaluation reports model performance on Bar Exam QA and Housing Statute QA using multiple language models, passage conditions, and retrieval configurations. Supporting tables also document benchmark examples, query expansion prompts, lexical-similarity analyses, and retrieval results.

  • Downstream QA evaluation: Downstream QA results cover Llama 3 8B and 70B Instruct models on Bar Exam QA, Llama 3 8B Instruct on Housing Statute QA, and GPT-4o-mini on both.The reported tables span Tables 30–34.
  • Benchmark examples: The appendix provides benchmark examples for HotpotQA, COLIEE, CLERC, Bar Exam QA, Housing Statute QA, and Natural Questions.It also includes Housing Statute QA question reformulations into Y/N questions.
  • Analysis and retrieval results: Additional tables document query-expansion prompts, a structured-reasoning expansion example, lexical-similarity tests, and retrieval performance across benchmark tasks.The retrieval tables cover Bar Exam QA, Housing Statute QA, Natural Questions, HotpotQA, COLIEE, and CLERC.
  • Retrieval and reranking: Llama-3-8B-Instruct evaluations use coarse retrieval, optional structured-reasoning query expansion, reranking by predicted answer confidence, and Accuracy (Top k).This setup is described for Bar Exam QA and Housing Statute QA.
  • Passage conditions: The Llama-3-70B-Instruct Bar Exam QA evaluation compares no-passage and gold-passage conditions to assess passage quality and model application.The table caption states that the 8B model struggles to apply the gold passages to the question.
  • Passage conditions: GPT-4o-mini evaluations compare no passage, retrieved passages, a generative reasoning rollout used as a pseudo-passage, and gold passages.These conditions are reported for both Bar Exam QA and Housing Statute QA.
Loading 2505.03970v1…