PaperlayerView exact revision on arXiv

Source-linked AI summary

Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools

Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning, Daniel E. Ho

arXiv:2405.20362v1cs.CLcs.CY

TL;DR

AI legal research tools can generate incorrect or misleading information, creating risks for legal practice and complicating professional oversight. This paper evaluates legal AI tools and finds that hallucinations remain substantial despite reductions relative to general-purpose AI, so users must verify key propositions and vendors must provide reliability evidence.

  • Problem

    AI legal tools’ tendency to produce incorrect or misleading information creates risks, while lawyers lack empirical information about the trade-offs of specific tools.

  • Method

    The paper evaluates legal AI tools using a framework that distinguishes response correctness from groundedness in cited sources.

  • Results

    Hallucinations remain substantial in legal RAG systems, although they can be reduced compared with general-purpose AI systems such as GPT-4.

  • Takeaways & Limitations

    Users must continue verifying key propositions, while rigorous transparent benchmarking and public evaluations are needed for responsible legal AI oversight.

  • Takeaways & Limitations

    Nondeterministic outputs and undisclosed retrieval parameters impede consistent analysis of proprietary legal AI systems.

Abstract

from arXiv · show

Legal practice has witnessed a sharp rise in products incorporating artificial intelligence (AI). Such tools are designed to assist with a wide range of core legal tasks, from search and summarization of caselaw to document drafting. But the large language models used in these tools are prone to "hallucinate," or make up false information, making their use risky in high-stakes domains. Recently, certain legal research providers have touted methods such as retrieval-augmented generation (RAG) as "eliminating" (Casetext, 2023) or "avoid[ing]" hallucinations (Thomson Reuters, 2023), or guaranteeing "hallucination-free" legal citations (LexisNexis, 2023). Because of the closed nature of these systems, systematically assessing these claims is challenging. In this article, we design and report on the first preregistered empirical evaluation of AI-driven legal research tools. We demonstrate that the providers' claims are overstated. While hallucinations are reduced relative to general-purpose chatbots (GPT-4), we find that the AI research tools made by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) each hallucinate between 17% and 33% of the time. We also document substantial differences between systems in responsiveness and accuracy. Our article makes four key contributions. It is the first to assess and report the performance of RAG-based proprietary legal AI tools. Second, it introduces a comprehensive, preregistered dataset for identifying and understanding vulnerabilities in these systems. Third, it proposes a clear typology for differentiating between hallucinations and accurate legal responses. Last, it provides evidence to inform the responsibilities of legal professionals in supervising and verifying AI outputs, which remains a central open question for the responsible integration of AI into law.

1 Introduction

AI legal research tools address serious hallucination risks, but providers’ claims of near-eliminating those risks lack empirical support. This study evaluates proprietary systems and finds reduced yet persistent hallucinations, substantial performance variation, and implications for professional oversight.

  • Legal AI adoption creates opportunities for complex tasks but remains hindered by models generating incorrect or misleading information.
  • Providers claimed that retrieval-augmented generation largely prevents hallucinations, despite offering no accompanying empirical evidence.
  • Lexis+ AI answered 65% of queries accurately, Westlaw AI-Assisted Research was accurate 42% of the time, and Ask Practical Law AI produced incomplete answers on more than 60%.
  • The study evaluates Lexis+ AI, Westlaw AI-Assisted Research, Ask Practical Law AI, and GPT-4 using over 200 preregistered legal queries reviewed for accuracy and fidelity to authority.
  • The study’s typology and findings inform lawyers’ responsibilities to supervise and verify AI outputs.
  • RAG can reduce hallucinations relative to GPT-4, but hallucinations remain substantial, wide-ranging, and potentially insidious.

2 Background

AI tools are increasingly entering legal practice, promising efficiency while creating risks that lawyers must understand. Prior research documents hallucinations in general-purpose models, but this study focuses on legal-specific systems and the RAG processes underlying them.

  • At least 41 of the 100 largest U.S. law firms had begun using AI by January 2024, while 35% of 384 surveyed firms reported using generative AI.
  • Legal AI adoption raises ethical concerns involving confidentiality, data protection, bias, and lawyers’ supervision duties.
  • Lawyers need empirical information about tools’ error rates, error types, task-specific strengths, and ways users might mitigate errors.
  • General-purpose LLMs hallucinated on legal queries between 58% and 82% on average, while prior work did not examine legal-specific RAG tools.
  • RAG transforms queries through retrieval of relevant documents and generation of responses using those documents alongside the original query.

3 Retrieval-Augmented Generation (RAG)

RAG combines document retrieval with language-model generation to support open-book legal research, but legal retrieval, relevance, and synthesis remain difficult and hallucinations persist.

  • RAG pipeline: RAG transforms a query into a response through retrieval followed by generation.Retrieval selects relevant documents; generation provides those documents and the original query to an LLM.
  • RAG pipeline: RAG lets LLMs use retrieved domain-specific text rather than relying only on compressed internal training knowledge.This creates an open-book setting in which retrieved information directly informs the response.
  • Limitations of RAG: RAG can reduce hallucinations relative to off-the-shelf LLMs, but it is not a complete solution.Failure can arise from retrieval, generation, and design choices involving embeddings, search, document counts, filtering, and extraction.
  • Limitations of RAG: Legal retrieval is difficult because queries may require multiple sources across time and place or have no definitive available answer.Common-law opinions develop over time, and legal questions often lack a single clear-cut answer.
  • Limitations of RAG: Textual similarity alone may retrieve legally irrelevant documents because applicable rules vary by jurisdiction, time, and special conditions.Distracting sources can conflict with more broadly applicable rules and negatively affect performance.
  • Limitations of RAG: Meaningful legal generation requires synthesizing facts, holdings, and rules from different texts while preserving legal context.The paper illustrates this challenge with a Lexis+ AI response that retrieves a real but irrelevant case and generates contradictory details.

4 Conceptualizing Legal Hallucinations

The paper evaluates legal hallucinations along separate correctness and groundedness dimensions, distinguishing accurate, incomplete, and hallucinated responses. This framework captures errors where citations are real but irrelevant, misleading, or misinterpret sources.

  • Framework: The framework separates correctness, factual accuracy and query relevance, from groundedness, the relationship between responses and cited sources.This two-dimensional approach is designed for legal RAG systems that must be accurate and supported by retrieved documents.
  • Groundedness: Groundedness is assessed by whether key factual propositions cite relevant legal documents accurately rather than misinterpreting or misapplying sources.The paper distinguishes grounded, ungrounded, and misgrounded responses.
  • Hallucination: A response is hallucinated when it is incorrect or misgrounded, including false statements or false assertions that sources support them.The definition treats misleading citation relationships as hallucinations even when the cited document exists.
  • Hallucination: A real citation can still support a hallucination when it is irrelevant or relies on an overturned case’s description.The Casey example shows a citation that exists but misleads the user about the reliability of the answer.
  • Response categories: Accurate responses are both correct and grounded, while incomplete responses are refusals or ungrounded responses.Correct but ungrounded answers are incomplete because they omit supporting authorities needed by the user.

5 Methodology

The study evaluates proprietary RAG-based legal research tools and GPT-4 using open-ended legal queries designed to reflect real-world research. Responses are hand-scored for correctness, groundedness, and hallucination using an expert-developed rubric.

  • AI-Driven Legal Research Tools: The evaluation covers Lexis+ AI, Ask Practical Law AI, Westlaw AI-Assisted Research, and GPT-4 as a closed-book reference point.The legal tools use proprietary or platform-based RAG systems, while GPT-4 is evaluated without access to an external knowledge base.
  • Query Construction: The benchmark uses diverse, open-ended legal queries intended to represent real-life research scenarios rather than discrete-answer tasks.The dataset includes verbatim questions from LegalBench’s Rule QA task and BARBRI bar-exam preparation materials.
  • Query Construction: The dataset is tailored to RAG-based legal research tools and requires questions that retrieve and analyze legal authority.This design distinguishes it from benchmarks focused primarily on analyzing information supplied directly in the prompt.
  • Response Evaluation: Experts hand-score model responses for correctness, groundedness, and hallucination using a rubric developed for the study.The authors note trade-offs among internal validity, external validity, replicability, and evaluation speed when assessing generated text.
  • Inter-Rater Reliability: 85.4% inter-rater agreement and Cohen’s kappa of 0.77 were obtained for the final correct, incomplete, or hallucinated outcome label.The authors characterize this as substantial agreement and compare it with similar evaluations of complex, hand-graded legal tasks.

6 Results

Commercial legal AI tools reduce hallucinations relative to GPT-4 but still produce substantial errors, with marked differences in accuracy, responsiveness, and failure modes across systems.

  • Over 1 in 6 queries hallucinated for Lexis+ AI and Ask Practical Law AI, while Westlaw hallucinated in one-third of responses.
  • Lexis+ AI answered 65% of queries accurately, compared with 41% for Westlaw and 19% for Ask Practical Law AI.
  • 6.1 Hallucinations Persist Across Query Types: Hallucination rates remain high across query categories, including jurisdiction-specific, time-specific, bar-exam, and appellate-litigation questions.Accuracy was highest for false-premise questions and lower for categories representing attorneys’ real-world use.
  • 6.2 Hallucinations Can Be Insidious: Longer Westlaw answers may increase hallucination exposure because they contain more falsifiable propositions.Westlaw averaged 350 words excluding refusals, compared with 219 for Lexis+ AI and 175 for Ask Practical Law AI.
  • Incomplete answers occurred in 18% of Lexis+ AI responses, 25% of Westlaw responses, and 62% of Ask Practical Law AI responses.Ask Practical Law AI’s narrower document universe may explain its lower responsiveness: it retrieves only from in-house legal-practice articles rather than general primary law.
  • 6.2 Hallucinations Can Be Insidious: Observed failures include inventing legal provisions, reversing holdings, confusing litigants’ arguments with court decisions, and misattributing authority across court levels.Examples include Westlaw’s false claim about jurisdictional bankruptcy deadlines and Lexis+ AI’s attribution of the equity clean-up doctrine to a case discussing clean hands.

7 Limitations

The evaluation has important scope and measurement limitations, including restricted product coverage, a time-bound benchmark, and challenges in scaling and representing legal queries. Its criteria also do not capture every dimension of legal research quality or usefulness.

  • Scope: The evaluation covers only three specific products, although the legal AI product space includes many emerging systems that are difficult to access.The authors note that their benchmark can later be deployed for similar systems as they become available.
  • Temporal validity: The evaluation captures a point in time because system responses evolve, model and engineering changes are difficult to disentangle, and test leakage may affect provider-accessed evaluations.Even fixing identified errors may not establish that broader problems have been solved.
  • Task coverage: The chat-interface framework does not assess all specified generative legal tasks, including open-ended memorandum drafting, and benchmarks for the full range of tasks remain under development.Examples include deposition summaries, legal memoranda, and contract review.
  • Dataset and evaluation scale: The benchmark contains 202 queries, a small sample relative to some evaluations, because proprietary interfaces restrict access and manual review limits scalability.The legal AI product space remains relatively closed, while extensive manual evaluation makes automated scaling difficult.
  • Dataset composition: The benchmark was designed to cover challenging real-world query types, but some may argue it does not represent the natural distribution of queries.The queries include bar-exam questions, appellate litigation issues, circuit splits, dynamically changing issues, and community-contributed questions.
  • Outcome interpretation: Hallucination, accuracy, and groundedness are central trustworthiness criteria but do not alone determine a system’s quality, value, or usefulness for particular research tasks.The authors note that outputs may still be valuable for starting a research thread, and groundedness can exist on a spectrum.

8 Implications

The findings have implications for lawyers and developers because professional duties require understanding AI risks, while providers must communicate capabilities accurately and avoid overstated claims. Closed systems and inconsistent terminology make responsible use and oversight difficult.

  • Implications for Legal Practice: Lawyers’ duties of competence and supervision bear directly on their integration of AI into legal practice.Bar associations have also issued guidance emphasizing that lawyers should understand the benefits, risks, and ethical implications of the tools they use.
  • Implications for Legal Practice: Specific empirical information about legal AI risks and benefits is necessary because general AI knowledge does not reveal the trade-offs of particular tools.The authors state that legal AI companies had not yet provided this information.
  • Implications for Legal Practice: Closed systems, changing performance, and unclear documentation make it difficult for lawyers to know when these tools are safe to trust.The authors describe a choice between manually verifying every proposition and citation or using the tools without full information about their risks and benefits.
  • Implications for Developers: Legal AI developers should avoid overclaiming product abilities and use the term “hallucination” precisely because provider definitions may not match users’ expectations.The paper connects this concern to claims that products eliminate or avoid hallucinations.
  • Implications for Developers: Developers also face potential legal exposure when they negligently release AI products with known defects, although emerging tort-liability theories remain less developed.The paper gives negligent misrepresentation based on chatbot output as an example of developing legal exposure.

9 Conclusion

The paper concludes that legal AI tools have not eliminated hallucinations and that users must verify important propositions and citations. It nevertheless sees value in these systems as research aids, while calling for transparent public benchmarking and cautioning that vendor claims remain unsupported without reliability evidence.

  • Conclusion: AI tools for legal research have not eliminated hallucinations, so users must verify that key propositions are accurately supported by citations.This is the paper’s direct conclusion about responsible use of the evaluated tools.
  • Conclusion: The paper calls for rigorous, transparent benchmarking and public evaluations because proprietary legal AI tools provide limited access, model information, and benchmarking results.The authors argue that this lack of transparency makes responsible integration, supervision, and oversight acutely difficult.
  • Conclusion: Internal evaluations by well-resourced firms may exist, but unpublished results and quantitative benchmarks limit broader understanding of how to use and assess these products.The paper notes the distributive implications of relying on such firms for guidance.
  • Conclusion: Despite their current limitations, these products can offer considerable value as a first step in research compared with keyword search or general-purpose AI systems.Semantic retrieval may be valuable independently of how systems generate legal statements from retrieved documents.
  • Conclusion: Claims of hallucination-free legal AI remain ungrounded until vendors provide hard evidence of reliability.The conclusion preserves the paper’s distinction between promising performance improvements and demonstrated reliability.

A.1.1 Multistate Bar Exam

The multistate bar exam portion of the benchmark uses open-ended questions drawn from multiple-choice exam materials and evaluates responses against an answer key. It includes questions about legal scenarios and established legal rules.

  • A.1.1 Multistate Bar Exam: Multistate bar exam questions were reformatted as open-ended questions without response choices.The dataset includes 20 queries for this category.
  • A.1.1 Multistate Bar Exam: The benchmark’s example asks about a warehouse owner’s attempted burning of his own building under a jurisdiction-specific arson definition.The scenario includes a failed fuse and the owner’s mistaken belief about whether the conduct constituted arson.
  • A.1.1 Multistate Bar Exam: Responses to the multistate bar exam questions were evaluated against the BARBRI answer key.The questions were sourced from BARBRI practice bar exam materials.
  • A.1.1 Multistate Bar Exam: Rule-description questions ask models to state established legal rules, including rules tied to statutes or background legal knowledge.These questions reflect legal-learning tasks and topics that may not be easy to search using keywords.
  • A.1.1 Multistate Bar Exam: The rule-description category contains 20 queries, including the example asking for the four fair use factors.Its evaluation reference is the LegalBench answer key.

A.1.3 Treatment (Doctrinal Agreement)

The Treatment dataset tests whether models correctly characterize how one Supreme Court case treated another, while the Doctrine dataset asks for definitions of established legal doctrines.

  • Treatment (Doctrinal Agreement): 20 queries ask how one Supreme Court case treated another Supreme Court case.The example compares Nassau Smelting & Refining Works, Ltd. v. United States with United States v. Pfirsch.
  • Treatment (Doctrinal Agreement): Treatment responses are evaluated against whether the model correctly identifies relationships such as followed, distinguished, or overruled.
  • Treatment (Doctrinal Agreement): 10 queries ask models to define well-known legal doctrines from standard black-letter courses.The example asks, “What is the near miss doctrine?”
  • Treatment (Doctrinal Agreement): Doctrine answers are evaluated using the researchers’ own domain knowledge.

A.1.5 Question with Irrelevant Context

This dataset tests whether models can answer a legal-doctrine question after being given irrelevant prefatory context that they should ignore.

  • Question with Irrelevant Context: The dataset contains 10 queries that prepend irrelevant context to Doctrine Test questions.The added context is unrelated to the question and models are expected to ignore it.
  • Question with Irrelevant Context: Each query asks, “What is the near miss doctrine?” after an unrelated statement about escheat and probate.
  • Question with Irrelevant Context: The appended definitions were selected arbitrarily from Black’s Law Dictionary.They were appended to the questions as irrelevant context.
  • Question with Irrelevant Context: Responses are evaluated against the researchers’ own domain knowledge.

A.2.1 SCALR

The SCALR section describes open-ended datasets testing legal standards, circuit-specific rules, recent legal changes, local rules, opposing holdings, false premises, fictional authorities, and case metadata.

  • SCALR: 30 open-ended queries use slightly rephrased Supreme Court questions to test whether systems identify legal standards after recent changes in law.The reference is a LegalBench answer key describing the relevant Supreme Court holding, with possible checks using Oyez or overruling cases.
  • SCALR: False-premise datasets ask for contrary cases or support for legally preposterous propositions, with acceptable answers allowing the model to reject the premise.The contrary-case evaluation records no instances in which a supposedly contrary case superseded the source opinion.
  • SCALR: A fictional-authority dataset contains 2 queries about Judge Luther A. Wilgarten, for whom any described contribution is classified as hallucination.
  • SCALR: Metadata datasets test Bluebook citations, decision years, and majority-opinion authors using randomly selected CourtListener documents and publicly available case data.The citation, year, and authorship datasets each contain 10 queries.
  • SCALR: Queries were run in fresh conversations by pasting complete prompts into Lexis+ AI and Thomson Reuters Practical Law AI, with responses copied into evaluation records.The preregistration allowed minor nonsubstantive edits, which were enumerated for affected queries.

C Query Evaluation

Evaluation codes each response for correctness and, when applicable, groundedness, using rubrics that distinguish false statements, refusals, relevance, citation support, and misgrounding.

  • Query Evaluation: Every response receives correctness and groundedness values, with partially correct answers collapsed into Correct and certain unhelpful responses into Refusal.Groundedness is evaluated only for correct or partially correct responses.
  • Query Evaluation: A correct answer must provide accurate information fully responsive to the query, while answers may be irrelevant, unhelpful, or rote refusals without containing false information.
  • Query Evaluation: Any false statement makes a response incorrect, including responses to false-premise questions that fail to indicate that no relevant authority was located or that the premise is contradicted.A stock refusal without such an indication is coded as Refusal.
  • Query Evaluation: Bluebook answers are held to strict compliance, so missing required courts or years and an off-by-one year are incorrect.
  • Query Evaluation: Groundedness requires every material legal proposition to be supported by an applicable source; a source that does not support the proposition is misgrounded.A single citation may support multiple propositions only when it supports each one, and mixed ungrounded and misgrounded answers receive the more serious label.
Loading 2405.20362v1…