Source-linked AI summary
How Much Do Legal RAG Systems Still Hallucinate?
Souvick Das, Sallam Abualhaija, Domenico Bianculli
TL;DR
Legal RAG hallucinations remain a serious, insufficiently characterized problem. This paper analyzes eight systems across two legal corpora using answer- and claim-level evaluation, finding substantial variation, with the least robust systems hallucinating in nearly half of responses.
Problem
Legal RAG systems remain prone to consequential hallucinations, but their behavior across claim severity, question types, and user personas requires domain-specific analysis.
Method
The study evaluates eight legal RAG systems on GDPR and CIVIL using complementary answer- and claim-level analyses across question categories and user personas, with external validation.
Results
Nearly half of responses from the least robust Llama3-8B-based systems contain hallucinations, while BM25+GPT-5 performs best and false-premise questions are particularly challenging.
Takeaways & Limitations
Hallucinations remain a persistent challenge, especially for false-premise questions and non-expert queries, and the observed trends generalize beyond the benchmark.
Takeaways & Limitations
Findings are limited to two legal sources and eight RAG systems and may not generalize to other jurisdictions, languages, retrievers, or foundation models.
Abstract
from arXiv · showhide
Hallucination is a major challenge for retrieval-augmented generation (RAG) systems in the legal domain, where ungrounded answers can lead to serious consequences. To better understand this problem, we conduct a fine-grained analysis of hallucination behavior in eight legal RAG systems across two legal corpora, the GDPR (in English) and a national civil law (in French). Using claim-level and answer-level evaluation, we report on hallucination density and severity, analyze performance across question categories and user personas, and validate our findings on an independent set of 142 legal-expert-authored questions. Our results show that hallucinations remain pervasive, ranging from less than 10% of responses for the best-performing systems to nearly half in the worst case. We further find that false-premise questions, containing incorrect assumptions that must be rejected, produce high hallucination rates on the manually-drafted questions.
1 Introduction
This study investigates hallucination behavior in legal RAG systems through answer- and claim-level analysis, focusing on variation across question categories and user personas. It evaluates eight systems on GDPR and CIVIL corpora and externally validates the observed patterns using 142 manually drafted legal questions.
- Motivation: Legal hallucinations can affect legal decisions, causing serious financial and societal consequences.Real-world incidents have included fictitious judicial decisions generated by AI and cited in legal filings.
- Motivation: Legal hallucination analysis requires domain-specific evaluation because legal interpretation is non-trivial and failures vary across question categories and user personas.Prior work indicates that LLMs often fail to correct incorrect legal assumptions in counterfactual questions.
- Study design: The study analyzes hallucinations at both answer and claim levels across question categories and user personas, with external validation on manually drafted legal questions.Answer-level analysis measures how often hallucinations occur in responses, while claim-level analysis examines minimal factual statements against supporting evidence.
- Study design: Eight state-of-the-art RAG systems are evaluated on GDPR in English and a national civil law corpus in French.The corpora are provided by ClaimRAG-LAW and are referred to as GDPR and CIVIL.
- Findings: Most hallucinations are isolated single-claim errors, whereas severe multi-claim hallucinations occur less frequently but remain present in top-performing systems.The finding comes from the study’s comprehensive analysis of the evaluated legal RAG systems.
- Findings: Hallucination rates are sensitive to question category and user persona, with BM25+GPT-5 lowest, Llama3-8B systems highest, and false-premise questions particularly prone to hallucinations.The study also reports variation across user personas in both GDPR and CIVIL.
- External validation: 142 manually drafted legal questions produce trends similar to benchmark results, indicating that the observed hallucination patterns generalize beyond ClaimRAG-LAW.The independent questions span both GDPR and CIVIL.
2 Related Work
Prior work distinguishes reference-free from reference-based hallucination detection, with this work focusing on methods that compare outputs against trusted evidence. In legal NLP, research has increasingly shifted from broad capability benchmarks toward factual reliability in legal RAG systems, while ClaimRAG-LAW enables fine-grained cross-setting evaluation but emphasizes aggregate metrics.
- Hallucination detection: Reference-based methods compare model outputs against trusted evidence, including gold-standard answers, retrieved documents, or supporting context.The paper focuses on this category of hallucination detection.
- Hallucination detection: Claim-level verification methods decompose generated responses into verifiable claims and assess each claim against supporting evidence.FActScore is an early example that verifies individual claims against retrieved evidence.
- Legal-domain evaluation: Legal benchmarks including LexGLUE, LEXTREME, and LegalBench-RAG evaluate legal language understanding, reasoning, and retrieval across diverse tasks.These benchmarks precede the more recent emphasis on factual reliability in legal RAG systems.
- Legal-domain evaluation: Recent legal RAG research targets factual reliability through retrieve-then-read question answering over Belgian statutory law and citation verification in real-world legal filings.Representative studies include LLeQA and Liu et al. [2026].
- Legal-domain evaluation: ClaimRAG-LAW provides fine-grained evaluation across jurisdictions, languages, question categories, and user personas, but its analysis primarily reports aggregate hallucination metrics.The benchmark supports systematic comparison of legal RAG systems while offering limited insight into variation across question categories and user perspectives.
3 Empirical Evaluation
The evaluation finds that hallucinations remain common in legal RAG systems, but they are usually localized to individual unsupported claims rather than pervasive across answers. GPT-based systems, especially BM25+GPT-5, are generally most reliable, while Llama3-8B systems and false-premise questions remain particularly challenging.
- Overall hallucination behavior: BM25+GPT-5 achieves the lowest claim-level hallucination density on both GDPR (1.5% ± 3.8) and CIVIL (1.2% ± 4.7).GPT-based systems consistently exhibit the lowest CL values, despite often producing more claims per response.
- Overall hallucination behavior: 48.3% answer-level hallucination and 9.5%±12.4 claim-level hallucination occur for E5+Llama3-8B on GDPR, while BM25+Llama3-8B reaches 45.8% and 17.9%±25.8 on CIVIL.High AL and CL values indicate frequent and dense hallucinations across many answers and claims.
- Hallucination severity: H1 is the dominant severity category across both datasets, although E5+GPT-5 produces one answer with H8 on GDPR and one with H9 on CIVIL.Most hallucinated answers contain a single unsupported claim, but rare multi-claim failures remain possible even for strong systems.
- Question categories: BM25+GPT-5 is the most robust system for GR questions, combining GDPR AL of 22.1% with the lowest CL of 1.6±3.6.E5+Llama3-8B has the highest GDPR GR hallucination rate at AL = 46.6%, while BM25+Llama3-8B has the highest CIVIL rate at AL = 39.4%.
- Question categories: BM25+Mixtral-8x7B reaches AL of 100% on CIVIL false-premise questions, whereas GPT-5 remains comparatively robust and reaches 0% in several settings.False-premise questions require systems to verify and reject an incorrect assumption, but many systems implicitly accept the premise.
- User personas: On CIVIL, BM25+Llama3-8B and BM25+Mixtral-8x7B exceed 70% AL for both CO and CI personas, while GPT-based systems remain below 20% in most cases and often below 5% with GPT-5.LE questions generally produce fewer hallucinations than CO questions, despite requiring more sophisticated legal reasoning.
4 Validation on Expert-Authored Questions
External validation on 142 independently drafted legal questions reproduces the principal benchmark trends and system rankings across GDPR and CIVIL. False-premise questions remain especially difficult, while GPT-4 and GPT-5 show distinct hallucination profiles despite broadly comparable overall performance.
- External validation: 142 third-party legal-expert-authored questions, evenly split between GDPR and CIVIL, reproduced the principal hallucination trends observed on ClaimRAG-LAW.The validation used an external question set to assess robustness beyond the benchmark.
- External validation: Consistent system rankings across both legal corpora suggest that the observed patterns generalize to independently curated legal questions rather than being benchmark-specific.Hallucination rates differed from those observed on the benchmark, but rankings remained consistent.
- Question categories: False-premise questions frequently produced some of the highest answer-level hallucination rates across GDPR and CIVIL.Many systems accepted incorrect assumptions instead of explicitly rejecting them, highlighting premise verification as a key challenge.
- Model profiles: GPT-4 and GPT-5 achieved broadly comparable overall performance but exhibited distinct hallucination profiles.GPT-4 often had slightly lower overall CL and AL hallucination rates, whereas GPT-5 consistently had lower strict CLs and ALs hallucination rates.
- Implications: Improving retrieval quality alone is unlikely to mitigate legal RAG hallucinations.The paper calls for stronger faithfulness to retrieved evidence and premise-verification capabilities, especially for misleading or incorrect assumptions.
5 Threats to Validity and Limitations
The study’s validity depends on the benchmark data and diagnostic metrics reported in ClaimRAG-LAW, so annotation or metric errors may affect its hallucination analyses. Expert-validated QA pairs and a standardized evaluation pipeline mitigate this threat by ensuring consistent comparisons across systems.
- Internal Validity: The analysis relies on ClaimRAG-LAW’s benchmark data and diagnostic metrics.Consequently, errors in the benchmark annotations or reported metrics may affect the study’s hallucination rates and analyses.
- Internal Validity: Errors in benchmark annotations or reported metrics may affect the reported hallucination rates and analyses.
- Internal Validity: Expert-validated QA pairs and a standardized evaluation pipeline mitigate this threat by ensuring consistent comparisons across analyzed RAG systems.
6 Conclusion
The study finds that hallucinations remain a persistent challenge across eight legal RAG systems, varying substantially by system, question category, and user persona. Independent expert-authored questions support the generality of these trends beyond the benchmark.
- Conclusion: Eight legal RAG systems were analyzed across the GDPR and CIVIL corpora using complementary answer-level and claim-level evaluations.The analysis builds on ClaimRAG-LAW and examines hallucination behavior at both answer and claim granularity.
- Conclusion: Hallucination rates vary substantially across configurations, from BM25+GPT-5 to Llama3-8B-based systems that hallucinate in nearly half of responses.Most hallucinations are isolated unsupported claims rather than pervasive errors throughout an answer.
- Conclusion: Factual-recall and false-premise questions, along with queries from non-expert users, are particularly challenging.Hallucination behavior varies substantially across question categories and user personas.
- Conclusion: Validation on an independent set of expert-authored legal questions indicates that the observed trends generalize beyond the ClaimRAG-LAW benchmark.The validation provides evidence that the findings are not specific to the benchmark.
- Conclusion: Future work will develop legal-specific evaluation metrics and diagnostic studies to clarify hallucination causes and support more reliable AI-assisted legal applications.The planned extensions target factors contributing to hallucinations in legal RAG systems.