PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models

Matthew Dahl, Varun Magesh, Mirac Suzgun, Daniel E. Ho

arXiv:2401.01301v2cs.CLcs.AIcs.CY

TL;DR

The paper asks how much legal knowledge LLMs possess and whether their errors vary across the American common law system. It develops a typology and systematically tests popular LLMs with verifiable legal queries. Hallucinations are widespread, models often fail on contra-factual questions and confidence assessment, and current accuracy limitations hinder broader legal-access goals.

  • Problem

    The paper addresses limited systematic evidence about LLM legal knowledge, hallucinations, and their variation across jurisdictions, courts, time periods, and cases.

  • Method

    The study develops a typology and legal knowledge queries, then evaluates popular LLMs using specific, verifiable questions about randomly selected federal court cases.

  • Results

    Legal hallucinations are widespread, occurring between 58% with ChatGPT 4 and 88% with Llama 2 on the tested federal-court questions.

  • Takeaways & Limitations

    The findings caution against rapid, unsupervised integration of popular LLMs into legal tasks and indicate that current shortcomings hinder improved legal access.

  • Takeaways & Limitations

    The paper concludes that challenges remain despite post-hoc recalibration efforts to address models’ shortcomings.

Abstract

from arXiv · show

Do large language models (LLMs) know the law? These models are increasingly being used to augment legal practice, education, and research, yet their revolutionary potential is threatened by the presence of hallucinations -- textual output that is not consistent with legal facts. We present the first systematic evidence of these hallucinations, documenting LLMs' varying performance across jurisdictions, courts, time periods, and cases. Our work makes four key contributions. First, we develop a typology of legal hallucinations, providing a conceptual framework for future research in this area. Second, we find that legal hallucinations are alarmingly prevalent, occurring between 58% of the time with ChatGPT 4 and 88% with Llama 2, when these models are asked specific, verifiable questions about random federal court cases. Third, we illustrate that LLMs often fail to correct a user's incorrect legal assumptions in a contra-factual question setup. Fourth, we provide evidence that LLMs cannot always predict, or do not always know, when they are producing legal hallucinations. Taken together, our findings caution against the rapid and unsupervised integration of popular LLMs into legal tasks. Even experienced lawyers must remain wary of legal hallucinations, and the risks are highest for those who stand to benefit from LLMs the most -- pro se litigants or those without access to traditional legal resources.

1 Introduction

The paper provides systematic empirical evidence on legal hallucinations, examining their prevalence, correlates, contra-factual bias, and implications for access to justice. It finds that hallucinations are widespread and can undermine efforts to make legal services more accessible.

  • These shortcomings significantly hinder LLMs’ potential to make legal information and services more accessible and affordable.The concern is especially salient for pro se and under-resourced litigants who may lack traditional legal resources.
  • Uneven legal knowledge could collapse jurisdictional and doctrinal nuances into a falsely homogeneous legal landscape, perpetuating representational harms.The paper frames this risk as potentially producing an algorithmic monoculture across downstream applications.
  • The study develops legal knowledge queries spanning case existence, holdings, and precedential relationships, applied to a random sample of cases.These queries are designed to elicit different dimensions of an LLM’s legal understanding.
  • Hallucinations vary with case age, salience, and jurisdictional prominence, with newer, more salient, and more prominent-jurisdiction cases performing better.The pattern suggests that legal knowledge may be unevenly distributed across the American common law system.
  • LLMs often provide seemingly legitimate but incorrect answers to contra-factual legal questions and struggle to assess their own certainty accurately.The findings concern both failure to correct erroneous legal assumptions and unreliable self-assessment.

2 Background and Theory

The paper defines language models and develops a typology distinguishing several forms of hallucination in legal settings. It emphasizes that factual legal accuracy is especially important, while reducing one hallucination type may increase another.

  • 2.1 What Is a Language Model?: Language models map text inputs, or prompts, to text outputs, or responses, by selecting probable token sequences.The paper formalizes this as a function from prompts to responses and describes generation as probabilistic.
  • 2.1 What Is a Language Model?: At temperature τ = 0, responses are theoretically deterministic, yet the study finds hallucinations even under this conservative sampling setting.Higher temperature makes responses more stochastic and may increase hallucination frequency.
  • 2.2 The Nature of Legal Hallucinations: The paper argues that hallucination types matter because different legal applications impose different requirements for fidelity, creativity, and factual correctness.This framework also supports empirical analysis and assessment of potential legal liability.
  • 2.2 The Nature of Legal Hallucinations: The paper’s typology distinguishes intrinsic hallucinations from open-domain hallucinations involving training-corpus fidelity or factual accuracy.The categories separate prompt fidelity, derivability from the training corpus, and correspondence with facts in the world.
  • 2.3 Hallucination Trade-offs: Reducing hallucinations of one kind may increase hallucinations of another, so minimizing all modes simultaneously may not be possible.The typology highlights a trade-off between fidelity and spontaneity in some legal tasks.

3 Profiling Hallucinations Using Legal Knowledge Queries

The paper evaluates LLMs on fourteen legal question-answering tasks spanning low, moderate, and high complexity. These tasks test recall of case metadata and text alongside increasingly demanding legal reasoning and synthesis.

  • Fourteen legal QA tasks are grouped into three categories ordered by increasing complexity.The categories progress from relatively easy case information to tasks requiring legal reasoning and synthesis of unstructured prose.
  • 3.1 Low Complexity Tasks: Low-complexity tasks test case existence, court, citation, and opinion-author information that is often available in captions, syllabi, or reporter metadata.The Court task spans SCOTUS, USCOA, and USDC cases, while citation and author tasks assess links to official records and judges.
  • The task designs use real cases and explicit matching rules to assess factual and legal answers.Examples include accepting fuzzy author-name matches, matching citations by volume, reporter, and page, and accepting cited precedents from majority opinions.
  • 3.2 Moderate Complexity Tasks: Moderate-complexity tasks test dispositions, quotations, cited authorities, and overruling years, requiring increasingly broader connections across case information.Disposition is limited to SCOTUS and appellate cases with unambiguous affirm-or-reverse outcomes; overruling year is limited to explicitly overruled SCOTUS cases.
  • 3.3 High Complexity Tasks: High-complexity tasks require synthesizing legal information unavailable directly in standard legal databases, including central holdings and doctrinal agreement.These tasks require legal reasoning about a case’s central issue, holding, or precedential relationship with another case.

4 Experimental Design

The experimental design samples federal cases across judicial levels, places, and years, then estimates hallucinations with reference-based or reference-free queries. The reference-free strategy identifies contradictions but yields only lower bounds on hallucination rates.

  • The study profiles hallucination rates across hierarchy, jurisdiction, time, and case prominence to support statistical inference on these covariates.Case metadata are aggregated from multiple legal databases and repositories to construct the test queries.
  • 5,000 cases are sampled from each of SCOTUS, USCOA, and USDC using stratified sampling across time and place.The sample is drawn from published case law using year, circuit-year, and state-year strata.
  • 4.2 Reference-based Querying: Reference-based queries compare deterministic LLM answers with known ground-truth answers and estimate hallucination rate by averaging mismatches across sampled queries.Abstentions are counted as non-hallucinations and are generally rare enough not to substantively affect findings.
  • 4.3 Reference-free Querying: Reference-free queries obtain two stochastic answers and use logical contradictions between them as evidence of hallucination.GPT-4 assesses contradictions, with manual labels and intercoder reliability analysis used as a validity check.
  • 4.3 Reference-free Querying: Reference-free estimates are lower bounds because logically consistent answers can still be legally unfaithful.The approach avoids assuming that self-consistency is correlated with factuality, but this creates an inferential limitation.
  • More than 800,000 queries are executed across four off-the-shelf LLMs, with both zero-shot and three-shot prompting.The authors share raw API calls and model responses in replication materials.

2. OpenAI’s ChatGPT 3.5 (gpt-3.5-turbo-0613, 2023b),

The experiments include multiple prompting conditions and execute a large query set across the listed language models. Zero-shot prompts ask directly, while three-shot prompts add examples for in-context learning.

  • The listed models include Google’s PaLM 2 and Meta’s Llama 2.
  • The study runs queries under both zero-shot and three-shot prompting setups.Three-shot prompts prepend example questions and responses to enable in-context learning.
  • More than 800,000 queries are executed, exceeding 200,000 queries per LLM.Raw API calls and model responses are shared in the replication materials.

5 Results

Hallucinations are widespread and vary systematically with task complexity, judicial level, jurisdiction, case prominence, decision year, model, and calibration. The results also show susceptibility to contra-factual premises and overconfidence, raising doubts about models’ substantive legal knowledge and reliability for legal assistance.

  • Task complexity: Hallucinations increase with the complexity of the legal research task, reaching at least 59% for Core legal questions and 63% for Central holding queries.Even easier reference-free tasks produce hallucinations at least 49% of the time.
  • Judicial hierarchy: Hallucinations are lowest for SCOTUS tasks, higher for USCOA tasks, and highest for USDC tasks.This pattern suggests stronger performance on authoritative precedents but weaker performance on localized legal knowledge.
  • Case prominence: Case prominence is negatively correlated with hallucination, with a sharp decline around the 90th prominence percentile for GPT 4, GPT 3.5, and PaLM 2.The authors interpret this as a bias toward the most well-known decisions, which is not observed for Llama 2 in the same way.
  • Decision year: Supreme Court hallucinations are most common for the oldest and newest cases and least common for Warren Court cases from 1953–1969.This is a non-linear relationship between hallucination and case age.

6 Discussion

The study systematically documents widespread legal hallucinations and related failures in popular LLMs, including susceptibility to mistaken premises and imperfect awareness of their own errors. These findings temper optimism about unsupervised legal deployment, especially where models’ gaps in legal knowledge could affect access to justice.

  • 6 Discussion: The study provides the first systematic empirical evidence on the nature, frequency, and correlates of legal hallucinations.It frames legal hallucinations as a serious obstacle to adopting LLMs in legal applications.
  • 6 Discussion: Factual legal hallucinations are widespread across ChatGPT 4, ChatGPT 3.5, PaLM 2, and Llama 2 on most profiled legal-knowledge tasks.The study examines four popular models rather than focusing only on legal reasoning performance.
  • 6 Discussion: The models also show susceptibility to contra-factual bias and cannot always know when they are producing legal hallucinations.They frequently provide seemingly genuine answers to legal questions whose premises are false and are imperfect predictors of their own responses’ accuracy.
  • 6 Discussion: The study distinguishes internalizing legal knowledge from engaging in legal reasoning, arguing that effective legal integration requires both capabilities.Gaps in background legal knowledge may prevent models from functioning as reliable sources of legal counsel regardless of in-context reasoning strength.
  • 6 Discussion: These findings temper optimism about off-the-shelf LLMs and indicate especially high risks for litigants in lower courts or less prominent jurisdictions.The discussion connects generic foundation-model risks to efforts to accelerate access to justice.

2. Seeking more complex forms of legal information,

The paper examines increasingly complex legal-information tasks, finding that LLMs can reproduce false premises, uncertain confidence, and source biases. Proposed mitigations may reduce hallucinations, but important technical and normative constraints remain.

  • Seeking more complex forms of legal information: Mistaken-premise questions can elicit seemingly genuine answers instead of corrections.
  • Seeking more complex forms of legal information: LLM reliance may disproportionately burden under-resourced or pro se litigants.
  • Seeking more complex forms of legal information: Limited judicial-source coverage can create a legal monoculture that elides deeper legal nuances.
  • Seeking more complex forms of legal information: Training-corpus and source-selection biases may propagate into downstream legal tools.
  • Seeking more complex forms of legal information: Retrieval-augmented generation and related techniques have reduced hallucinated content and improved output reliability, but remain limited by retrieval quality, ambiguity, computational demands, and conflicting sources.Legal retrieval must distinguish outdated or jurisdictionally irrelevant sources from apposite and binding law.
  • Seeking more complex forms of legal information: Hallucination detection and evaluation are themselves imperfect because datasets and metrics may not reflect real-world reliability.
  • Seeking more complex forms of legal information: Model fidelity to training data, user prompts, and legal facts involves competing normative commitments.
  • Seeking more complex forms of legal information: Developers should make hallucination-reduction choices transparent so users can judge whether legal information is trustworthy.The paper releases a test dataset and urges human-centered integration that augments, rather than replaces, legal professionals and litigants.

A Data Sources

The appendix documents the data, sampling, merging, aggregation, and prompt procedures used to construct the paper’s queries and tasks.

  • The data and sampling strategy are described in Section 4.1 and linked to tasks in Appendix Table 1.
  • The exact procedures for sampling, merging, and aggregating across datasets are available in the paper’s GitHub materials.
  • The appendix provides zero-shot and few-shot prompt templates for SCOTUS, USCOA, and USDC queries.

C Contradiction Detection Approach

The contradiction-detection approach uses repeated stochastic responses and GPT-4 entailment judgments, with human checks supporting the reliability of GPT-4’s labels.

  • For reference-free tasks, repeated higher-temperature queries expose contradictions without ground-truth labels.
  • GPT-4 performs contradiction labeling at scale through an entailment-framed elicitation prompt.The authors report this prompting strategy as the most performant in their setting.
  • GPT-4’s contradiction conclusions were cross-checked against expert knowledge on a subsample.
  • Two researchers independently recoded 100 query responses to assess contradiction-labeling agreement.
  • The reported κ values support using GPT-4 for contradiction detection, with one researcher agreeing more with GPT-4 than with the other human coder.
  • Expected calibration error compares model confidence with the empirical frequency of correct, non-hallucinated responses.
  • Because several closed-source models do not expose conditional probabilities, the study estimates them from K temperature-1 samples compared with the greedy response.

D.2 Temperature Scaling

Temperature scaling generally improves calibration, especially for GPT-4 and Llama 2, but residual miscalibration and compressed confidence distributions preserve important limitations.

  • D.2 Temperature Scaling: Out-of-the-box LLMs are not well-calibrated on legal queries.
  • D.2 Temperature Scaling: Ex post Platt scaling tests whether rescaling raw confidence distributions improves measured ECE.
  • D.2 Temperature Scaling: Rescaling generally improves ECE, especially for GPT-4 and Llama 2.
  • D.2 Temperature Scaling: Llama 2’s pooled ECE improvement results from compressing its entire rescaled distribution into [0.0, 0.3].The rescaled model is therefore not confident in any response.
  • D.2 Temperature Scaling: Overall, rescaling can help, but the results continue to indicate calibration challenges on legal-knowledge queries.

E.1 Fake Case Existence Task

The fake-case existence experiment tests whether apparently strong existence-task performance reflects genuine case knowledge or a tendency to answer “yes” or “no” regardless of truth. Fake citations use plausible party names and court-appropriate reporters.

  • Fake Case Existence Task: Strong performance on real-case existence questions is difficult to interpret because every prompted case is real.The authors note that results could reflect genuine knowledge or a tendency to answer “yes” to existence questions.
  • Fake Case Existence Task: The experiment replaces real cases with fake cases to test whether models can distinguish case existence from prompt patterns.Each fake citation uses plausible party names and an appropriate reporter for the court.

Appendix Table 5 reports the results of this Fake case existence experiment. We see that GPT

The appendix examines model behavior across fake-case existence, jurisdictions, prompting modes, tasks, and abstentions. Results show systematic differences across models and settings, while prompt design and abstention counting affect measured hallucination rates.

  • Fake case existence: GPT 3.5 and PaLM 2 often assert that real or fake cases exist, whereas GPT 4 and Llama 2 tend to deny existence instead.The latter models’ apparent immunity to false assertions reflects opposite-direction bias rather than reliable discrimination.
  • Fake case existence: The fake-case results undermine apparently satisfactory existence-task performance as evidence that models know the true details of cases.Different models can achieve favorable-looking results through opposing existence biases.
  • Jurisdictional heterogeneity: The non-truncated jurisdiction analysis substantively mirrors the truncated analysis, with the Ninth and Second Circuits continuing to perform best.The Federal and Eleventh Circuits show somewhat stronger results in the non-truncated analysis, plausibly because they contain newer cases.
  • Prompting and abstentions: Most analyses use three-shot prompting because examples improve response formatting and may expose embedded knowledge by reducing commercial models’ reticence.Contra-factual tasks requiring negative responses are excluded from few-shot prompting because the analysis targets tuning and behavior rather than model-held information.
  • Prompting and abstentions: Zero-shot prompting preserves the model ordering but reduces the magnitude of each model’s hallucination rate.The reduction is largely driven by abstentions in the Quotation and more complex tasks, which are counted as non-hallucinations.
  • Prompting and abstentions: Abstentions are generally low, but many models abstain in the zero-shot Quotation task.The study counts abstentions as non-hallucinations, making abstention behavior relevant to interpreting hallucination rates.
Loading 2401.01301v2…