PaperlayerView exact revision on arXiv

Source-linked AI summary

LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain

Nicholas Pipitone, Ghita Houir Alami

arXiv:2408.10343v1cs.AI

TL;DR

LegalBench evaluates legal LLM generation but leaves retrieval from larger corpora insufficiently assessed. This paper introduces LegalBench-RAG by mapping LegalBench contexts to original legal-text spans and finds that chunking and post-processing choices affect retrieval performance, while general-purpose rerankers show limitations. The benchmark is intended to support precise evaluation of legal retrieval, but it does not test reasoning across multiple documents or very high-hop queries.

  • Problem

    LegalBench evaluates legal LLM generation but does not benchmark extracting correct context from larger legal corpora, leaving a retrieval-evaluation gap.

  • Method

    The paper maps LegalBench’s expert-annotated contexts back to original legal-corpus spans and constructs retrieval queries with corresponding ground-truth snippets.

  • Results

    Recursive Text Character Splitter chunking significantly enhances retrieval performance, while Cohere’s general-purpose reranker shows limitations.

  • Takeaways & Limitations

    LegalBench-RAG provides a dedicated framework for assessing retrieval precision and recall in legal RAG systems.

  • Takeaways & Limitations

    The benchmark does not assess retrieval that requires reasoning across multiple documents or very high numbers of hops.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) systems are showing promising potential, and are becoming increasingly relevant in AI-powered legal applications. Existing benchmarks, such as LegalBench, assess the generative capabilities of Large Language Models (LLMs) in the legal domain, but there is a critical gap in evaluating the retrieval component of RAG systems. To address this, we introduce LegalBench-RAG, the first benchmark specifically designed to evaluate the retrieval step of RAG pipelines within the legal space. LegalBench-RAG emphasizes precise retrieval by focusing on extracting minimal, highly relevant text segments from legal documents. These highly relevant snippets are preferred over retrieving document IDs, or large sequences of imprecise chunks, both of which can exceed context window limitations. Long context windows cost more to process, induce higher latency, and lead LLMs to forget or hallucinate information. Additionally, precise results allow LLMs to generate citations for the end user. The LegalBench-RAG benchmark is constructed by retracing the context used in LegalBench queries back to their original locations within the legal corpus, resulting in a dataset of 6,858 query-answer pairs over a corpus of over 79M characters, entirely human-annotated by legal experts. We also introduce LegalBench-RAG-mini, a lightweight version for rapid iteration and experimentation. By providing a dedicated benchmark for legal retrieval, LegalBench-RAG serves as a critical tool for companies and researchers focused on enhancing the accuracy and performance of RAG systems in the legal domain. The LegalBench-RAG dataset is publicly available at https://github.com/zeroentropy-cc/legalbenchrag.

1. Introduction

LegalBench-RAG addresses the lack of a dedicated benchmark for evaluating retrieval in legal RAG systems. It evaluates whether retrieval mechanisms can pinpoint exact legal references through granular text snippets.

  • LegalBench assesses legal LLM reasoning, but existing benchmarks lack dedicated evaluation of retrieval in legal RAG systems.
  • LegalBench-RAG is introduced as the first legal-domain benchmark designed specifically to evaluate retrieval systems.
  • The benchmark measures whether retrieval mechanisms can pinpoint exact legal references rather than returning imprecise context.

2. Related Work

RAG systems retrieve context from legal documents and use it to generate answers, while existing benchmarks address general retrieval, legal reasoning, and other legal-text challenges. LegalBench specifically evaluates legal reasoning with expert-designed tasks.

  • Retrieval Augmented Generation (RAG): RAG systems segment legal documents, embed chunks and queries, retrieve top-k relevant chunks, and pass them with the query to an LLM.
  • Retrieval Augmented Generation (RAG): A contextual retriever may rerank top-k′ bi-encoder results with a cross-encoder before returning the final top-k contexts.
  • Other legal focused benchmarks: Existing RAG benchmarks such as RGB and RECALL evaluate retrieval and answering in general contexts, while LegalBench targets legal reasoning.
  • LegalBench: LegalBench contains 162 expert-designed tasks spanning six types of legal reasoning.
  • Other legal focused benchmarks: Prior legal-AI research also addresses document review, case summarization, long documents, specialized jargon, and inferential reasoning.

3. A Benchmarking Dataset: LegalBench-RAG

LegalBench-RAG converts LegalBench’s context into a retrieval benchmark by mapping annotations to original legal-corpus spans and constructing queries from document descriptions and interrogatives. Its labels identify the relevant file and span rather than only a classification outcome.

  • Starting Point: LegalBench: LegalBench-RAG is constructed from LegalBench’s ground-truth contexts to evaluate retrieval of correct legal snippets from a larger corpus.
  • Tracing Back to Original Sources: The construction process traces each LegalBench text segment to its original corpus location and pairs queries with relevant character spans.
  • Construction Process: Annotations are transformed into queries using document descriptions and interrogatives derived from category mappings.
  • Construction Process: Each query originates from one source annotation containing a document identifier and annotation category.
  • Construction Process: LegalBench-RAG labels identify the filename and relevant span, whereas LegalBench converts annotation labels into Yes/No answers.
  • Construction Process: The benchmark can represent annotations as arrays of filename-and-index-range tuples when relevant spans are non-adjacent.

3.2. Quality Control

Quality control is treated as essential for trustworthy retrieval evaluation. The benchmark combines expert-created annotations with manual inspection at three critical decision points.

  • Quality control is crucial for ensuring trust in retrieval recall and precision scores.
  • The annotations were created by domain experts through the methods used by the four source datasets.
  • The authors manually inspected every data point and applied quality control at three critical decision points.

Mapping Annotation Categories to Interrogatives

The benchmark construction converts legal annotations into precise query–snippet pairs, while applying document-description and category-quality controls. The resulting corpus and QA pairs support a legally expert-annotated retrieval benchmark and a smaller experimental variant.

  • Document mapping: GPT-4o-mini generated short document descriptions, whose formats and distinctiveness were validated through regex checks, manual inspection, and embedding similarity.Manually indistinguishable description pairs were excluded.
  • Category selection: Categories with inconsistent annotation precision were excluded to preserve benchmark integrity.The inclusion decision followed manual evaluations of category precision.
  • Query construction: Queries pair legal annotations with relevant text spans located at precise character indices in source documents.Each QA pair links a query to snippets, file paths, exact quotations, and character ranges.
  • Dataset composition: 6,889 question-answer pairs span four datasets and almost 80 million corpus characters across 714 documents, with every pair annotated by legal experts.These statistics define the full LegalBench-RAG benchmark.
  • Dataset composition: LegalBench-RAG-mini contains 776 queries, selecting exactly 194 from each of PrivacyQA, CUAD, MAUD, and ContractNLI.The lightweight corpus retains only portions corresponding to those queries.

3.4. Significance of this work

The work provides a publicly available retrieval-focused legal benchmark and a lightweight variant, while acknowledging substantial domain and reasoning-scope limitations. Its source data are costly to reproduce, and its queries generally remain within single-document retrieval.

  • Significance: Legal dataset creation is costly: replicating CUAD is estimated at around $2,000,000 in legal-expert review.The estimate assumes 9,283 pages reviewed at least four times, 5–10 minutes per page, and $500 per hour.
  • Significance: The benchmark is publicly available for commercial and academic evaluation of retrieval in legal RAG pipelines.It is intended to support standardized comparison and iteration across retrieval techniques.
  • Limitations: The benchmark does not cover all legal-industry documents, including structured numerical data parsing and medical-record analysis.Its source datasets comprise NDAs, M&A agreements, commercial contracts, and consumer-company privacy policies.
  • Limitations: Queries are answered by exactly one document, so the benchmark evaluates document and snippet selection rather than cross-document reasoning.Manual inspection nevertheless found several queries requiring multi-hop reasoning, with room for more complex cases.

4. Benchmarking RAG systems using LegalBench-RAG

LegalBench-RAG evaluates retrieval quality through controlled experiments on chunking, reranking, embedding, and top-k design choices. The benchmark compares these choices across datasets using equally weighted metrics.

  • Benchmarking design choices: LegalBench-RAG measures retrieval effectiveness across alternative chunking, reranking, embedding, and top-k design decisions.The section frames retrieval design as a set of tunable pipeline choices affecting performance.
  • Experimental setup: The experiments compare naive fixed-size chunks with 500-character chunks and recursive character splitting, with or without Cohere reranking.The naive method uses no overlap; recursive splitting preserves paragraphs, sentences, and words together.
  • Evaluation protocol: The evaluation weights metrics equally across the four datasets, independently of their document or query counts.This aggregation prevents larger datasets from receiving greater weight solely because they contain more documents or queries.
  • Chunking experiment: The first experiment isolates chunking effects by fixing all other hyperparameters and comparing naive chunking with Recursive Text Character Splitting.Results are reported across datasets and varying values of k.
  • Reranking experiment: The second experiment isolates post-processing by comparing retrieval without reranking against Cohere reranking, including their combination with recursive splitting.The comparisons are reported in Tables 4, 6, and 7.

5. Results and Discussion

Across four datasets and varying k, recursive splitting without reranking produced the strongest retrieval results, while dataset difficulty varied substantially. PrivacyQA was easiest, MAUD hardest, and the findings motivate more specialized legal rerankers and benchmarks.

  • 5.1. Results of the experimentation: The evaluation aggregates Recall@k and Precision@k across four datasets, four method combinations, and k values from 1 to 64.Dataset-level aggregation uses equal weighting for each dataset.
  • 5.1. Results of the experimentation: Recursive Text Character Splitting without reranking achieved the highest precision and recall across the evaluated configurations.Recall increased with k, whereas precision decreased; Cohere reranking performed worse than no reranking.
  • 5.2. Comparison of the four datasets: PrivacyQA was the easiest dataset, reaching 14.38% Precision@1 and 84.19% Recall@64 with RCTS and no reranker.The authors associate this performance with straightforward private-policy questions posed by non-lawyers.
  • 5.2. Comparison of the four datasets: MAUD was the most challenging dataset, reaching 2.65% Precision@1 and 28.28% Recall@64.Its technical legal jargon made relevant information harder to retrieve, and general-purpose reranking performed poorly.
  • 5.2. Comparison of the four datasets: ContractNLI and CUAD were challenging but less difficult than MAUD, likely because their language was somewhat less specialized.The comparison indicates substantial variation in retrieval difficulty across legal datasets.
  • 5.3. Future Work: The results indicate a need for more specialized legal benchmarks and reranking models trained or fine-tuned for legal text.The authors specifically identify room for improvement in reranking, especially on challenging datasets such as MAUD.

6. Conclusion

The paper concludes that LegalBench-RAG fills a legal retrieval-evaluation gap and supports more granular assessment of RAG retrieval mechanisms. Experiments show that chunking and post-processing choices affect retrieval performance, while general-purpose rerankers have limitations.

  • Contribution: LegalBench-RAG provides a benchmark for evaluating legal RAG retrieval precision and recall.It is built by mapping question-answer pairs back to their original legal contexts.
  • Experimental findings: RTCS significantly enhances retrieval performance among the evaluated chunking strategies.
  • Experimental findings: General-purpose rerankers such as Cohere’s model show limitations in legal AI applications.
  • Implications: Specialized legal benchmarks can capture retrieval nuances and support development of legal RAG systems.

A. Appendix A

Appendix A illustrates the benchmark’s question-answer records using abbreviated legal-document examples. The examples expose query, source-file, and answer information while ellipses replace large text regions for readability.

  • Appendix examples: Ellipses replace large regions of text in the appendix examples for readability.
  • Appendix examples: The examples identify the query associated with each benchmark record.
  • Appendix examples: The examples identify the source document through a file path.
  • Appendix examples: The examples include answer text drawn from legal documents.
Loading 2408.10343v1…