Source-linked AI summary
Legal RAG Bench: an end-to-end benchmark for legal RAG
Abdur-Rahman Butler, Umar Butler
TL;DR
Legal RAG lacks reliable end-to-end benchmarks that reflect difficult, grounded legal work. Legal RAG Bench addresses this gap with an expert-built dataset and factorial evaluation methodology, finding that embedding-model choice dominates performance and retrieval quality shapes hallucination rates.
Problem
Reliable end-to-end legal RAG evidence is scarce, while existing benchmarks often weakly reflect real-world legal performance and groundedness requirements.
Method
Legal RAG Bench combines 4,876 legal passages and 100 expert-crafted questions with a full factorial comparison of retrieval and generative models and hierarchical error decomposition.
Results
Embedding-model choice dominates legal RAG performance across evaluation dimensions, with Kanon 2 Embedder producing 17.5-point higher correctness than OpenAI’s Text Embedding 3 Large.
Takeaways & Limitations
Poor retrieval can trigger hallucinations, while a domain-adapted legal embedder can raise performance until reasoning models become the pipeline bottleneck.
Takeaways & Limitations
Model comparisons may reflect interaction effects, including possible matching between Gemini Embedding 001 and Gemini 3.1 Pro, rather than general model superiority.
Abstract
from arXiv · showhide
We introduce Legal RAG Bench, a benchmark and evaluation methodology for assessing the end-to-end performance of legal RAG systems. As a benchmark, Legal RAG Bench consists of 4,876 passages from the Victorian Criminal Charge Book alongside 100 complex, hand-crafted questions demanding expert knowledge of criminal law and procedure. Both long-form answers and supporting passages are provided. As an evaluation methodology, Legal RAG Bench leverages a full factorial design and novel hierarchical error decomposition framework, enabling apples-to-apples comparisons of the contributions of retrieval and reasoning models in RAG. We evaluate three state-of-the-art embedding models (Isaacus' Kanon 2 Embedder, Google's Gemini Embedding 001, and OpenAI's Text Embedding 3 Large) and two frontier LLMs (Gemini 3.1 Pro and GPT-5.2), finding that information retrieval is the primary driver of legal RAG performance, with LLMs exerting a more moderate effect on correctness and groundedness. Kanon 2 Embedder, in particular, had the largest positive impact on performance, improving average correctness by 17.5 points, groundedness by 4.5 points, and retrieval accuracy by 34 points. We observe that many errors attributed to hallucinations in legal RAG systems are in fact triggered by retrieval failures, concluding that retrieval sets the ceiling for the performance of many modern legal RAG systems. We document why and how we built Legal RAG Bench alongside the results of our evaluations. We also openly release our code and data to assist with reproduction of our findings.
1 Introduction
Legal RAG Bench addresses the scarcity of reliable end-to-end legal RAG evaluations. It combines a challenging legal dataset with a methodology designed to assess retrieval, generation, and failure modes relevant to real-world use.
- Motivation: High-quality end-to-end legal RAG benchmarks remain scarce, while many existing legal AI benchmarks have label, methodological, and design flaws.These shortcomings can produce evaluations that correlate weakly with real-world performance and encourage unsuitable models of legal AI quality.
- Contribution: Legal RAG Bench contains 4,876 passages from the Victorian Criminal Charge Book and 100 expert-crafted questions requiring expert knowledge of Victorian criminal law and procedure.Each question is paired with a long-form answer and supporting passage.
- Contribution: The benchmark evaluates end-to-end legal RAG through a full factorial experiment and a hierarchical taxonomy separating hallucinations, retrieval failures, and reasoning failures.The methodology attributes errors to particular retrieval and generative models.
- Findings: The analysis finds retrieval quality is the primary driver of end-to-end legal RAG performance and that most production hallucinations are induced by retrieval failures.The authors release the data, code, and results to support reproducible evaluation.
2 Related work
Existing legal evaluations often have weak expertise involvement, methodological problems, or mismatches between claimed and actual capabilities. The paper argues for evaluations that better represent difficult, grounded legal RAG work.
- Benchmark shortcomings: Many existing legal AI evaluations suffer from poor label quality, methodological flaws, and insufficient involvement of legal subject-matter experts.These problems affect both benchmark design and annotation.
- Retrieval benchmarks: MLEB is described as the largest, most diverse, and most comprehensive open-source legal retrieval benchmark, while other retrieval benchmarks exhibit serious construction failings.The paper specifically discusses problems in AILA Casedocs, AILA Statutes, and the legal split of MTEB.
- Generation benchmarks: The review reports that most examples in an examined HLE subset were inappropriate, poorly framed, or mislabeled.The paper presents the Tommy-and-James property example as a prime case involving insufficient context and an incorrect purported answer.
- Task mismatch: LegalBench and LegalBench-RAG are criticized because most of their data consists of low-value text classification and sentiment-analysis tasks requiring simple yes-or-no answers.The paper presents this as a mismatch between their marketed reasoning and retrieval stress tests and their effective task content.
- Task mismatch: HousingQA and BarExamQA use more challenging questions but closed-ended evaluation cannot simulate real-world legal RAG conditions involving incoherent or correct yet ungrounded answers.The paper notes that the benchmark authors acknowledge this shortcoming.
3 Dataset
Legal RAG Bench is built as a practical criminal-law evaluation set with expert-designed questions, evidence-linked long-form answers, and legal- and AI-informed construction.
- Dataset scope: Legal RAG Bench targets real-world end-to-end legal RAG performance, particularly in criminal law, an under-resourced but important domain.The dataset is presented as an evaluation of retrieval and generative models in larger RAG systems aimed at practical legal advice.
- Dataset construction: The dataset contains 4,876 passages from the Judicial College of Victoria’s Criminal Charge Book paired with 100 complex, hand-crafted questions requiring expert Victorian criminal-law knowledge.Subject-matter expertise in law and AI informed every stage of design and development.
- Corpus processing: Corpus sections were converted to Markdown, split according to their hierarchy, and further chunked when necessary so no chunk exceeded 512 tokens under the Kanon legal tokenizer.The chunking process used heuristics and the semchunk semantic chunking algorithm.
- Question design: Questions were sampled from the corpus and designed to be answerable from individual passages while being lexically dissimilar from relevant passages.The lexical dissimilarity was intended to stress-test models’ semantic understanding.
- Evidence structure: Subject-matter experts hand-crafted long-form answers, producing question-answer-evidence triplets for every example.This structure supports joint evaluation of retrieval and generation components.
4 Evaluation
The evaluation compares retrieval and generative models factorially across correctness, groundedness, and retrieval accuracy, then decomposes errors into hallucination, retrieval, and reasoning categories.
- Experimental design: Legal RAG Bench evaluates every combination of three embedding models and two frontier LLMs in a full factorial experiment.The design enables apples-to-apples comparison of retrieval and generative models’ relative effects.
- Evaluation dimensions: Correctness measures whether an answer entails the reference answer, groundedness whether it is supported by retrieved passages, and retrieval accuracy whether the annotated supporting passage is retrieved.These signals expose component-level failures that overall RAG accuracy can obscure.
- Error taxonomy: The taxonomy triangulates whether errors originate in hallucination, retrieval, or reasoning rather than merely reporting aggregate performance.Figure 1 presents the taxonomy as a flow chart.
- Error taxonomy: The hierarchical taxonomy treats hallucination as the first pipeline failure, followed by retrieval error and reasoning error under the specified conditions.Hallucination means invented facts outside the provided context; retrieval error means a relevant passage was not retrieved and the answer is grounded but incorrect.
5 Results
Results show that embedding-model choice is the dominant driver of legal RAG performance, while LLM effects are generally smaller and can interact with the embedder. Error decomposition links retrieval quality to hallucinations, and the factorial analysis identifies groundedness-specific interactions.
- Evaluation dimensions: Embedding-model choice dominates performance across correctness, groundedness, and retrieval accuracy, whereas LLM choice has a more moderate average effect.Model-level averages compare embedding models across LLMs and LLMs across embedding models.
- Evaluation dimensions: 17.5 points lower correctness, 4.5 points lower groundedness, and 34 points lower retrieval accuracy result when switching from Kanon 2 Embedder to Text Embedding 3 Large.These are the reported performance changes for the three evaluation dimensions.
- Error decomposition: 6.75-point average decrease in hallucinations is associated with Kanon 2 Embedder relative to its general-purpose alternatives, while Gemini Embedding 001 produces a 4.5-point increase relative to Text Embedding 3 Large.The authors infer that better retrieval may make generative models less likely to invent facts.
- Error decomposition: 18% better overall RAG accuracy relative to the sample average is delivered by Kanon 2 Embedder, compared with -3% for GPT-5.2 and +3% for Gemini 3.1 Pro.RAG accuracy is defined as the complement of the sum of all errors.
- Statistical significance and interaction effects: Embedding-model main effects are statistically significant across all three outcomes, while the LLM main effect is not statistically distinguishable from zero for correctness or the combined outcome.The statistical analysis uses ANOVA-style Wald tests from a linear probability model with question fixed effects and clustered standard errors.
- Statistical significance and interaction effects: Groundedness is the only metric with detectable embedder–LLM interactions, so LLM differences in groundedness are not stable across embedding models.Switching from Gemini 3.1 Pro to GPT-5.2 reduces groundedness by 9.0 points with Text Embedding 3 Large and 10.0 points with Gemini Embedding 001, but not detectably with Kanon 2 Embedder.
6 Conclusion
Legal RAG Bench evaluates legal RAG end to end because groundedness and verifiability matter, finding that embedding-model choice dominates performance. Its authors release the benchmark materials to support reproduction.
- Legal RAG Bench addresses the importance of benchmarks that measure both correctness and groundedness in evidence-driven legal work.The conclusion emphasizes that legal conclusions must be verifiable as well as correct.
- Legal RAG Bench assesses legal RAG end to end using a full factorial design and hierarchical error decomposition.The benchmark covers correctness, groundedness, and retrieval-related performance.
- Embedding-model choice dominates RAG performance across all evaluation dimensions.The authors report that poor retrieval can increase hallucinations and that stronger retrieval can shift bottlenecks toward reasoning models.
- The authors openly release the code, data, and results and encourage reproductions.
7 Data Availability
Legal RAG Bench’s dataset, code, and evaluation results are publicly available through Hugging Face, GitHub, and an interactive viewer.
- The dataset is publicly available on Hugging Face, while the code is available on GitHub.
- An interactive viewer presents the benchmark’s results in the accompanying blog post.
8 Disclosures
Isaacus created Kanon 2 Embedder and sponsored the creation of Legal RAG Bench and MLEB.
- Isaacus, whose founders are the authors, created Kanon 2 Embedder and sponsored Legal RAG Bench and MLEB.