Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models

• Because LLMs could offer cheap, accessible legal help, the study assesses whether uneven legal knowledge could instead widen existing service disparities.Source: §1 · p. 5
• The study theorizes a typology of legal hallucinations and tests varied legal-knowledge queries on random cases across district, appeals, and Supreme Courts.Source: §1 · p. 6
• The cited passage reports that LLMs hallucinate between 58% and 88% of the time, with ChatGPT 4 at 58% and Llama 2 at 88%, respectively.Source: §1 · p. 6
• Models perform better on newer, more salient cases and prominent legal jurisdictions, suggesting a risk of legal monoculture.Source: §1 · p. 6
• LLMs often give seemingly legitimate but incorrect answers to contra-factual legal questions and struggle to gauge their certainty without post-hoc recalibration.Source: §1 · p. 6

2.1 What Is a Language Model?

• The study models LLM responses as samples from a learned conditional distribution optimized over a training corpus, and experiments at τ=0 show hallucinations under the most conservative sampling.Source: §2.1 · pp. 7–8

2.2 The Nature of Legal Hallucinations

• Unlike prior work focused on closed-domain legal tasks, this study examines open-domain queries requiring accurate answers to open-ended legal questions.The setting approximates a lawyer or pro se litigant seeking advice through a legal chat interface.Source: §2.2 · pp. 8–9
• The typology distinguishes prompt-unfaithful intrinsic hallucinations, corpus-unsupported extrinsic hallucinations, and responses unfaithful to world facts.The paper regards the world-factuality form as especially alarming where legal work requires correct statements of law.Source: §2.2 · pp. 9–11

2.3 Hallucination Trade-offs

• The paper investigates only factuality-related hallucinations because the first two modes are not always problematic in legal settings.Defining unwanted corpus-unfaithful outputs can require balancing fidelity against spontaneity.Source: §2.2 · p. 10, §2.3 · p. 11
• Lawyers may tolerate some hallucinations, but not factual infidelity to controlling law, which can quickly strip an LLM of professional or analytical utility.Source: §2.3 · p. 11
• Reducing one hallucination mode may increase another: fidelity to a prompt can prevent closed-domain hallucinations yet induce open-domain hallucinations.Source: §2.3 · p. 11
• Users face choices about prompt skepticism, jurisdictional training, and whether to preserve conflicting legal nuance or limit responses to a query’s scope.Source: §2.3 · pp. 11–12
• The preferable balance may differ for pro se litigants and sophisticated lawyers; the authors offer empirical evidence while emphasizing that normative questions require continued research.Source: §2.3 · p. 12

3 Profiling Hallucinations Using Legal Knowledge Queries

• The 14 tasks are grouped into three categories of increasing complexity to assess LLM accuracy across different kinds of legal queries.The framework uses a question-answering approach.Source: §3 · p. 12

3.1 Low Complexity Tasks

• Low-complexity tasks ask models to verify case existence and supply a case’s court, Bluebook citation, or opinion author.These facts are treated as relatively easy to recover from case captions or syllabi, without requiring higher-order legal reasoning.Source: §3.1 · pp. 12, 14

3.2 Moderate Complexity Tasks

• Moderate-complexity tasks test case-specific knowledge through disposition, opinion quotation, cited authority, and the year a case was overruled.The overruling-year task is limited to explicitly overruled Supreme Court cases and accepts only the exact year.Source: §3.2 · pp. 14–15

3.3 High Complexity Tasks

• High-complexity tasks require legal reasoning or synthesis beyond readily available database facts, including doctrinal agreement, factual background, procedural posture, and subsequent history.The doctrinal-agreement task uses Shepard’s treatment codes and is limited to Supreme Court cases.Source: §3.3 · p. 16
• The related-proceedings query tests whether a model knows about proceedings concerning a case after a particular decision, which may change or clarify its legal significance.Source: §3.3 · p. 16
• The core-legal-question query asks models to identify the issue or issues a court addresses, an important factor in assessing whether a case is apposite.Source: §3.3 · p. 17
• The central-holding query tests whether a model can identify the legal principle future cases may rely on or distinguish, the study’s most complex evaluated task.Articulating a holding is described as crucial for legal analysis and argumentation.Source: §3.3 · p. 17

4.1 Data Construction

• The study drew 5,000 cases from each of SCOTUS, USCOA, and USDC using stratified random sampling across year, circuit-year, and state-year strata.The sampling frame comprised cases published in the U.S. Reports, Federal Reporter, and Federal Supplement.Source: §4.1 · p. 17

4.2 Reference-based Querying

• Reference-based queries used case-specific questions with known metadata answers, and the study estimated each task’s hallucination rate by averaging responses over sampled queries.Responses were generated greedily; explicit abstentions counted as non-hallucinations and were generally rare without substantively affecting findings.Source: §4.2 · p. 18

4.3 Reference-free Querying

• For their final five tasks, the authors used reference-free queries pairing a prompt with two LLM answers, the second retrieved by calling the same model again, and asked GPT-4 to identify contradictions.They report GPT-4’s contradiction-labeling reliability as comparable to human reliability.Source: §4.3 · pp. 19–20
• Contradictions establish hallucinations, but the method yields only a lower bound because two logically consistent answers can still lack fidelity to the law.The authors report that even these lower-bound hallucination rates are quite high and informative.Source: §4.3 · p. 20

4.4 Models

• The experiments used four popular, state-of-the-art, off-the-shelf LLMs.Source: §4.4 · p. 20

2. OpenAI’s ChatGPT 3.5 (gpt-3.5-turbo-0613, 2023b),

• The study includes Google’s PaLM 2 (text-bison-001) and Meta’s Llama 2 (Llama-2-13b-chat-hf).The passages identify the model versions and attribute them to Anil et al. 2023 and Touvron et al. 2023, respectively.Source: §2 · p. 20
• Queries were run under both zero-shot and three-shot prompting setups.In zero-shot, the model is asked directly about the case; three-shot prepends example questions and responses for in-context learning (Brown et al. 2020).Source: §2 · p. 21
• The zero-shot setup asks the LLM directly about the given case.It is compared with the three-shot setup, which prepends examples.Source: §2 · p. 21
• Three-shot prompts prepend example questions and responses to enable in-context learning (Brown et al. 2020).The full prompts and few-shot examples for each query are provided in the Online Appendix.Source: §2 · p. 21
• More than 800,000 queries were run, with 200,000+ per LLM.The raw API calls and model responses are shared in the accompanying replication materials.Source: §2 · p. 21
• The study shares its raw API calls and model responses in accompanying replication materials.Full prompts and few-shot examples are available in the Online Appendix.Source: §2 · p. 21

5.1.1 Hallucinations Vary by Task Complexity

• At least 59% of queries on Core legal question and 63% on Central holding produced hallucinations; these high-complexity rates are lower bounds.The paper reports that hallucinations generally increase with task complexity.Source: §5.1.1 · p. 23

5.1.2 Hallucinations Vary by Court

• Across tasks and models, hallucinations were lowest for Supreme Court cases, higher for Courts of Appeals, and highest for district courts.The authors caution that weaker performance at the district-court level may leave LLMs poorly attuned to localized legal knowledge.Source: §5.1.2 · p. 24, Figure 3 · p. 24

5.1.3 Hallucinations Vary by Jurisdiction

• Performance was strongest in the Ninth, Second, Third, and First Circuits, but tended to be worst in central U.S. circuits and was worst in the D.C. Circuit.The D.C. Circuit result runs counter to its perceived influence.Source: §5.1.3 · pp. 25–26

5.1.4 Hallucinations Vary by Case Prominence

• Hallucinations declined sharply as SCOTUS case prominence passed the 90th percentile, indicating more accurate responses about prominent cases.Prominence was measured by PageRank percentile.Source: Figure 5 · p. 26
• Case prominence was negatively correlated with hallucination, with a sharp change near the 90th percentile for GPT 4, GPT 3.5, and PaLM 2, but not Llama 2.This pattern suggests the first three models may be especially skewed toward the best-known decisions, even within the Supreme Court.Source: §5.1.4 · p. 27

5.1.5 Hallucinations Vary by Case Year

• SCOTUS hallucinations were most common for the oldest and newest cases and least common for Warren Court cases from 1953–1969.The authors note that peak performance may lag current doctrine and miss old law that remains applicable.Source: §5.1.5 · pp. 27–28

5.1.6 Hallucinations Vary by LLM

• Overall, GPT 4 performed best, followed by GPT 3.5, PaLM 2, and Llama 2; LLMs tended to overstate justices’ true prevalence more than they understated it.Source: §5.1.6 · p. 28

5.2 Contra-factual Bias

• Contra-factual hallucinations were higher in lower courts and greatest for false overruling premises; Llama 2 often rejected prompts by claiming real cases or justices did not exist.Those rejections count as successful premise rejection under the study’s metric, while reflecting Llama 2’s poor performance on other tasks, especially Existence.Source: §5.2 · p. 31

5.3 Model Calibration

• Across legal queries, PaLM 2 was best calibrated, followed by GPT 3.5, GPT 4, and Llama 2, which was significantly worse than the first three; pooled ECE was 0.057 for PaLM 2 and 0.099 for GPT 3.5.Source: Figure 9 · p. 32, §5.3 · p. 32
• Pooled ECE was 0.190 for GPT 4 versus 0.421 for Llama 2, indicating significantly better calibration for GPT 4.Source: §5.3 · p. 32
• Although GPT 4 had the lowest raw hallucination rate, it was less calibrated than PaLM 2 and GPT 3.5, and its hallucinations were more likely to mislead users.Source: §5.3 · p. 32
• Calibration was poorer on complex tasks such as Doctrinal agreement, tasks involving lower judicial levels, and partially open-ended Court and Author tasks, where models over-reported prominent responses.Source: §5.3 · pp. 32–33
• In every case, calibration error was positive: models overestimated their confidence relative to their actual hallucination rates.Source: §5.3 · p. 33

6 Discussion

• Factual legal hallucinations are widespread across most profiled legal-knowledge tasks in ChatGPT 4, ChatGPT 3.5, PaLM 2, and Llama 2.The authors describe their study as the first systematic empirical test of popular perceptions of this problem.Source: §6 · p. 33
• LLMs often give seemingly genuine answers to legal questions with false premises, showing susceptibility to contra-factual bias.An ex post scaling adjustment explored in the Online Appendix does not resolve the challenge.Source: §6 · p. 33
• Under default configurations, LLMs imperfectly predict their tendency to confidently hallucinate legal falsehoods.The discussion characterizes this as an inability to always “know what they know.”Source: §6 · pp. 33–34
• The study assesses internalized legal knowledge rather than legal reasoning, and knowledge gaps make models unreliable sources of legal counsel regardless of in-context reasoning strength.The authors argue effective integration into the legal profession requires strength in both respects.Source: §6 · p. 34
• The authors identify especially high risks for generic foundation models in lower-hierarchy courts and less prominent jurisdictions, tempering optimism about expanding access to justice.They also report documenting correlates of hallucinations.Source: §6 · pp. 33–34

2. Seeking more complex forms of legal information,

• The risks are highest for under-resourced and pro se litigants, and LLM proliferation may exacerbate rather than eradicate inequalities in access to legal services.The authors suggest user education may mitigate mistaken-premise questions and misplaced trust, while other risks are more intractable.Source: §2 · pp. 34–35
• Ideally, legal LLMs would perform well on localized legal information, correct misguided questions, and express confidence appropriate to their reliability.The authors contrast these goals with SCOTUS-level information, accepting user premises at face value, and hallucinating with conviction.Source: §2 · p. 35
• Technical methods can reduce hallucinated content and improve output accuracy, reliability, and faithfulness, but their effectiveness has important limits.For example, retrieval-augmented generation depends on retrieval quality; databases require continual updates and can contain conflicting or jurisdictionally irrelevant sources.Source: §2 · p. 35
• Hallucination detection and mitigation evaluations are not foolproof, and it remains unclear whether technical improvements can solve the diagnosed conceptual problems.Datasets and metrics may not reflect real-world reliability, embed bias, or adequately capture task-specific hallucination.Source: §2 · p. 36
• Because fidelity to training data, user prompts, and legal facts can conflict, developers should disclose which hallucinations they prioritize minimizing.Transparent choices let litigants decide whether to trust the legal information they seek from LLMs.Source: §2 · p. 36
• The authors release their query-and-answer dataset for continued evaluation and urge practitioners, scholars, and educators to integrate LLMs cautiously and keep human judgment central.Even experienced practitioners should verify prompts and responses, while scholars and educators should guard against importing models’ knowledge biases into applications.Source: §2 · pp. 36–37

A Data Sources

• The study’s data and sampling strategy, described in Section 4.1, are used to construct its queries.The exact procedure for sampling, merging, and aggregating across datasets is available on GitHub.Source: A Data Sources · p. 56
• Appendix Table 1 links the data sources to each task.The supplied table caption identifies it as a table of data used to construct each task.Source: A Data Sources · p. 56, Table 1 · p. 56
• The exact procedure for sampling, merging, and aggregating across datasets is available on GitHub.This procedure is referenced in the data-sources discussion.Source: A Data Sources · p. 56

B Prompt Templates

• Full zero-shot and few-shot prompt templates for all queries appear in Appendix Figures 5–19.The templates show the prompts used for the study’s queries.Source: B Prompt Templates · p. 56
• Few-shot examples for SCOTUS queries are used in the templates, with appropriate cases from other judiciary levels used for USCOA and USDC versions.The examples vary by judiciary level in the USCOA and USDC versions.Source: B Prompt Templates · p. 56
• Appendix Table 2 is identified as reporting intercoder reliability.The supplied caption does not provide reliability values.Source: Table 2 · p. 57

C Contradiction Detection Approach

• For reference-free tasks without ground-truth labels, the authors detect contradictions by comparing LLM responses to repeated queries at higher temperatures.This approach uses the models’ stochastic responses as the basis for contradiction checks.Source: C Contradiction Detection Approach · p. 57
• The contradiction-labeling query for GPT-4 frames detection as an entailment task, which the authors found most performant in their setting.The query template is provided in Appendix Figure 20; the entailment framing follows Wang et al. (2021).Source: C Contradiction Detection Approach · p. 57
• GPT-4’s contradiction labels were checked against two researchers’ independent coding of 100 queries, and the reported κ values support using GPT-4 for detection.One researcher agreed with GPT-4 more than with the other human coder.Source: C Contradiction Detection Approach · p. 57

D.1 Estimation Approach

• Because conditional response probabilities are not always observable, direct calibration assessment was unavailable, including for closed-source ChatGPT 4, ChatGPT 3.5, and PaLM 2.The setup requires these distributions to assess whether model confidence correlates with response accuracy.Source: §D.1 · p. 58
• The authors estimate model response distributions by drawing K samples at temperature 1 and comparing them with the model’s greedy response.This sampling approach is used to calculate ECE for each task.Source: §D.1 · p. 58
• ECE represents the average difference between a model’s confidence in response accuracy and the empirical frequency of correct, non-hallucinated responses.The approach defines X as the estimated response probability and Y as an indicator of whether the response matches the reference response.Source: §D.1 · p. 58
• Following Kumar, Liang, and Ma (2020), the authors use a plug-in ECE estimator with 10 equally sized bins and subtract an approximated bias term.Source: §D.1 · p. 58

D.2 Temperature Scaling

• The authors report that evaluated LLMs are not well-calibrated on legal queries out of the box, while noting that unscaled probabilities alone do not establish that a model is uncalibrated overall.Temperature adjustment is presented as a way for users to alter the probability distribution.Source: §D.2 · pp. 58–59
• The analysis applies ex post Platt scaling to raw model distributions and checks whether temperature adjustment improves measured ECE.The appendix figure displays rescaled calibration curves for all LLMs with resource-aware tasks pooled.Source: §D.2 · p. 59, Figure 1 · p. 59

Appendix Figure 1 visualizes the rescaled ECE for each of our LLMs, Appendix Table 3 re-

• Rescaling generally improves ECE, especially for GPT 4 and Llama 2, but the procedure is not perfect: GPT4, GPT 3.5, and PaLM 2 remain relatively uncalibrated.LLMs face calibration challenges on legal knowledge queries, though ex post rescaling can help.Source: Appendix Figure 1 visualizes the… · p. 59
• Llama 2’s pooled ECE improves substantially because rescaling compresses its entire distribution into [0.0, 0.3], leaving the model unconfident in all responses.Thus, the pooled ECE improvement does not indicate well-calibrated, confident predictions.Source: Appendix Figure 1 visualizes the… · p. 59, Figure 1 · p. 59
• Appendix Table 3 reports temperature-scaled ECE across levels of the federal judiciary, with conditional probabilities estimated from sampled responses and bootstrap standard errors.The note specifies 10 responses at temperature 1 and agreement with the model’s greedy response.Source: Table 3 · p. 60, Appendix Figure 1 visualizes the… · p. 60

E Supplementary Analyses

• The supplementary analyses include a table of estimated hallucination rates for the fake-existence task across levels of the federal judiciary.The table note says standard errors are reported in parentheses; no rate values are supplied in the passage.Source: Table 5 · p. 61, Appendix Figure 1 visualizes the… · p. 61

E.1 Fake Case Existence Task

• Performance on real-case existence questions is generally strong, but because every prompted case is real, genuine knowledge cannot be distinguished from a tendency to answer “yes.”The supplemental analysis addresses this ambiguity by testing fake cases.Source: §E.1 · p. 61
• The supplemental analysis repeats the existence query using fake cases instead of real cases.Each fake citation uses plausible party names and the reporter appropriate for the court, such as the fabricated USCOA citation SolarFlare Technologies v. Armstrong, 656 F.3d 262.Source: §E.1 · p. 61

Appendix Table 5 reports the results of this Fake case existence experiment. We see that GPT

• GPT 3.5 and PaLM 2 often assert that any case exists, whereas GPT 4 and Llama 2 tend to deny any case; PaLM 2 is more discriminating at the SCTOUS level.Thus, apparently satisfactory performance on the Existence task may not reflect knowledge of true case details.Source: Appendix Table 5 reports the results… · p. 61

E.2 Hallucination Rates at the USCOA Level (No Time Cutoff)

• Without a time cutoff, the Ninth and Second Circuits remain best, while the Federal and Eleventh Circuits show somewhat stronger performance, plausibly because their case pools are newer.The Eleventh Circuit split from the Fifth in 1981, and the Federal Circuit began in 1982.Source: §E.2 · p. 62

E.3 Hallucination Rates by State

• State-level USDC results show particularly low hallucination rates in New York, with Hawaii and Idaho also performing strongly.New York comprises four federal districts; Hawaii and Idaho each have one district court.Source: §E.3 · p. 62

E.4 Zero-shot Prompting

• Zero-shot prompting preserves the model ranking but reduces hallucination rates, largely because abstentions in Quotation and more complex tasks count as non-hallucinations.The zero-shot figure summarizes the reference-based tasks pooled.Source: §E.4 · pp. 63–64, Figure 4 · p. 64
• The main-manuscript trends for task complexity, court level, jurisdiction, case importance, and case year remain in zero-shot results but are less pronounced.The authors attribute the weaker patterns to higher abstention rates.Source: §E.4 · p. 64

F Abstention Rates

• Abstention rates are generally low, except in the zero-shot Quotation task, where many models abstain.Abstentions are counted as non-hallucinations.Source: F Abstention Rates · p. 64, Table 6 · p. 65