Explaining Legal Concepts with Augmented Large Language Models (GPT-4)

1 Introduction

• The study compares direct GPT-4 explanations with explanations augmented by relevant case-law information retrieval.The authors investigate both GPT-4's limitations and whether retrieved context improves explanation quality.Source: §1 · pp. 1, 3, 4
• Statutory provisions use vague, open-textured terms to accommodate varied circumstances, making their interpretation difficult.Legal interpretation often requires determining how courts have applied a term in prior cases.Source: §1 · p. 2
• The proposed system aims to automatically draft case-law-based explanations that assist legal professionals interpreting statutory terms.The envisioned explanations target users with legal training and may support lawyers, judges, and public access to law.Source: §1 · pp. 2–3
• Directly generated explanations may appear reasonable while remaining incomplete, inaccurate, or fictitious.The paper illustrates this concern with automatically generated explanations of the statutory phrase “navigation equipment.”Source: §1 · p. 2
• The paper contributes an investigation of GPT-4 for professional legal explanation and a benchmark comparing direct and retrieval-augmented setups.The authors describe this as the first study to make these contributions in the AI & Law community.Source: §1 · p. 4

2 Related Work

• Earlier research provided case-law extracts or summaries to help users assess how open-textured concepts apply.Related studies addressed support for laypeople and automatic summarization of individual cases.Source: §2 · p. 4
• Legal information-retrieval research has investigated retrieving and ranking sentences useful for interpreting statutory terms.Prior ranking methods used pre-trained language models and outperformed earlier approaches.Source: §2 · p. 4
• Language-model augmentation can inject retrieved external information into prompts to guide model outputs.The literature includes sparse and dense retrieval, reasoning prompts, and models that actively formulate searches.Source: §2 · p. 4
• This paper uses retrieved explanatory passages from several cases as context for GPT-4, which transforms them into a compact explanation of a legal term.The retrieval methodology follows earlier work on identifying useful case-law sentences.Source: §2 · p. 4

3 Proposed Method

• GPT-4 is configured with temperature 0.7, a 1,000-token output limit, top_p 1, and zero frequency and presence penalties.These settings govern creativity, output length, and token repetition during generation.Source: §3 · p. 5

Baseline model

• The baseline directly asks GPT-4 to explain the term from the source provision and instructions without additional context.This setup relies on information potentially present in training data but remains vulnerable to hallucination.Source: Baseline model · p. 6

augmented model.

• The augmented model supplies relevant case-law context so GPT-4 functions primarily as a transformer of provided information into a concise explanation.The approach is designed to reduce reliance on the model’s internal training data for factual context.Source: Baseline model · p. 6
• The augmented architecture adds high-value case-law sentences to the prompt alongside the statutory term and source provision.Relevant cases are retrieved, segmented into sentences, evaluated for explanatory value, and inserted into the prompt.Source: Baseline model · pp. 6–7

4.1 Data

• The dataset contains 42 statutory phrases and 26,959 case-law sentences, including 1,853 manually labeled high-value sentences used for augmentation.High-value sentences are intended to define or elaborate the meaning of a term.Source: §4.1 · pp. 7–8

4.2 Experimental Design

• Each system generates one-sentence and ten-sentence explanations for every term, producing 168 explanations and 84 paired comparisons.Two legal scholars with extensive semantic-annotation experience evaluate the paired outputs.Source: §4.2 · p. 8
• The annotation dimensions are factuality, clarity, relevance, information richness, and on-pointedness.These dimensions respectively address accuracy, communication quality, pertinence, comprehensiveness, and focus on the legal term and its implications.Source: §4.2 · pp. 8–9
• Annotators compare paired explanations while blinded to system details and randomized to explanation order.Each term is handled by a single annotator, who evaluates both short and long versions for that term.Source: §4.2 · pp. 8–9

5 Results and Discussion

• The augmented system was preferred overall, with the preference more pronounced for long explanations than short explanations.Short-explanation preferences are reported in Fig. 3, while long-explanation preferences are reported in Fig. 4.Source: §5 · p. 9, Figure 3 · p. 10, Figure 4 · p. 11

5.1 Factuality

• Augmentation eliminated observed citation and case-content hallucinations: annotators found no non-existing cases or misrepresentations in augmented explanations.Direct GPT-4 explanations included unverifiable citations and invented or irrelevant case content, affecting both short and long explanations.Source: Augmented LLM · p. 11, Baseline · p. 11

Augmented LLM

• Augmented explanations still inherited retrieval problems, including state cases for federal statutes, misleading or outdated authorities, dissenting opinions, and context mismatches.The paper attributes these issues to the legal information-retrieval component and warns that problematic retrieved material can propagate into final outputs.Source: Augmented LLM · pp. 11–12

5.2 Clarity

• Annotators preferred augmented explanations for clarity, although improperly formatted citations distracted from reading in both systems.The clarity assessment covered language, structure, coherence, conciseness, and consistency.Source: §5 · p. 9, §5.2 · p. 12

5.3 Relevance

• Annotators preferred augmented explanations for relevance, with no systematic baseline issue identified beyond factual inaccuracies.The relevance assessment concerned pertinence, importance, applicability, and currency; the reason for the preference was left for future investigation.Source: §5.3 · p. 12

5.4 Information Richness

• Annotators preferred augmented explanations for information richness, without identifying specific problems in either model.Information richness was assessed through depth, breadth, perspectives, contextual details, and nuance.Source: §5.4 · pp. 12–13

5.5 On-pointedness

• Annotators decisively preferred augmented explanations for on-pointedness, and reported no specific issues with either model.On-pointedness was assessed through focus, precision, and integration.Source: §5.5 · p. 13

• Augmentation significantly enhanced explanation quality across factuality, clarity, and on-pointedness, overcoming issues related to hallucinations.The approach combines traditional legal information retrieval with GPT-4 to produce accurate and useful summaries of how statutory terms have been interpreted.Source: §6 · p. 13
• Direct GPT-4 explanations appeared high-quality at the surface level, but in-depth analysis revealed serious factuality limitations, including citations to nonexistent case law.Source: §6 · p. 13
• The augmented approach remains sensitive to shortcomings in the information-retrieval method, which can create problems in generated explanations and propagate into final outputs.Future work should improve retrieval, extend explanations to laypeople, and apply augmented LLMs to other legal tasks.Source: §6 · p. 13