Source-linked AI summary
Can GPT-3 Perform Statutory Reasoning?
Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme
TL;DR
Statutory reasoning is a basic legal skill, but GPT-3’s capabilities on this task require direct evaluation. The paper tests GPT-3 on SARA and synthetic statutes, finding some gains over prior SARA results alongside persistent errors on simple synthetic cases. Its findings point to statutory reasoning as an area for further AI research.
Problem
Statutory reasoning—applying natural-language statutes to facts—is a basic legal skill, and the paper examines how capable GPT-3 is at it.
Source: §1 · p. 1Method
The paper evaluates GPT-3 on SARA, probes its knowledge of the U.S. Code, and tests it on synthetic statutes designed to be unfamiliar to the model.
Source: §1 · p. 1Results
Some GPT-3 prompt settings surpass the prior SARA state of the art, but performance varies; zero-shot accuracy is 78% on 2-wide, 2-deep synthetic statutes, and two-shot reasoning reaches 81% on 3-wide, 3-deep statutes.
Source: §3.5 · p. 4, CONCLUSION · p. 9Takeaways & Limitations
The results identify statutory reasoning as an area for further research to improve large language models.
Source: §1 · p. 1, CONCLUSION · p. 9The synthetic statutes are much simpler than many U.S. Code sections: they lack negation and ambiguity and use balanced structures.
Source: §5.1 · p. 6
Abstract
from arXiv · showhide
Statutory reasoning is the task of reasoning with facts and statutes, which are rules written in natural language by a legislature. It is a basic legal skill. In this paper we explore the capabilities of the most capable GPT-3 model, text-davinci-003, on an established statutory-reasoning dataset called SARA. We consider a variety of approaches, including dynamic few-shot prompting, chain-of-thought prompting, and zero-shot prompting. While we achieve results with GPT-3 that are better than the previous best published results, we also identify several types of clear errors it makes. We investigate why these errors happen. We discover that GPT-3 has imperfect prior knowledge of the actual U.S. statutes on which SARA is based. More importantly, we create simple synthetic statutes, which GPT-3 is guaranteed not to have seen during training. We find GPT-3 performs poorly at answering straightforward questions about these simple synthetic statutes.
1 INTRODUCTION
The paper evaluates GPT-3 on statutory reasoning, a basic legal skill, using the SARA benchmark and synthetic statutes. Although GPT-3 improves on prior SARA results, it has imperfect knowledge of U.S. statutes and makes regular errors on simple synthetic statutes.
1 INTRODUCTION
- Prior reports of GPT-3’s legal-task performance—including low but passing law-school exam grades—motivate testing its statutory reasoning directly.The cited examples include bar-exam questions and law professors’ final exams.Source: §1 · p. 1
- GPT-3’s straightforward application improves on prior work on SARA, but it also makes clear mistakes.SARA is a handcrafted benchmark covering nine sections of the U.S. tax code.Source: §1 · p. 1
- The paper tests whether GPT-3 can perform statutory reasoning, applying natural-language legislative rules to natural-language facts.The evaluation begins with SARA, a handcrafted computational benchmark covering nine U.S. tax-code sections.Source: §1 · p. 1
- GPT-3 has imperfect knowledge of the U.S. Code and regularly errs on straightforward questions about human-interpretable synthetic statutes.The paper presents an example involving a short synthetic statute in Figure 1.Source: §1 · p. 1, Figure 1
- The authors identify statutory reasoning as an area for new AI research and future improvements in language models.They frame the synthetic-statute errors as an additional reason to investigate the capability.Source: §1 · pp. 1–2
2 RELATED WORK
Related work situates statutory reasoning among legal NLP tasks and studies of language-model reasoning. It reviews prompting methods and GPT-3 evaluations on legal and other reasoning benchmarks.
Statutory reasoning
- SARA machine-reading efforts have had limited success, while its curated cases avoid semantic ambiguity; statute entailment and legal judgment prediction are related legal NLP tasks.Semantic ambiguity can itself challenge statutory interpretation, as in Nix v. Hedden.Source: §2 · p. 2
Prompting GPT-3
Reasoning with GPT-3
- Prior reasoning studies report that added context or problem decomposition can help, while GPT-3 may lose track across multi-step proofs and rely on background knowledge.Saparov and He [17] report that reasoning ability decreases as a target domain overlaps less with real-world knowledge.Source: §2 · p. 2
GPT-3 on legal tasks
- Legal-task evaluations find GPT-3 can outperform earlier BERT-based models on statute entailment, yet remains below expert performance on legal multiple-choice questions.Prompting GPT-3 to rank bar-exam answer choices approaches passing scores in some subject areas.Source: §2 · p. 2
3 SARA
This section evaluates GPT-3 on SARA using several prompting setups, with and without statutory text. Some setups outperform the BERT-based state of the art, but performance varies and examples reveal errors in GPT-3’s statutory knowledge.
3.1 Data and Experimental Setup
- SARA contains nine U.S. tax-code sections and 376 cases; the study excludes 100 tax-calculation cases and reports results separately for 72 numeric and 28 nonnumeric test cases.The dataset also translates statutes, facts, and questions into Prolog, which solves the questions with 100% accuracy.Source: §3.1 · p. 2
3.2 Three Approaches
- The SARA experiments compare dynamically selected four-shot examples, zero-shot prompts, and fixed ten-shot prompts with human-written chains of thought.When GPT-3 does not end with “Entailment” or “Contradiction,” the researchers use a second prompt to extract the answer.Source: §3.2 · p. 3
3.3 "Let’s think step by step."
- Adding “Let’s think step by step” is tested with zero-shot and four-shot dynamic prompts, but not with the ten-shot prompts that already include step-by-step explanations.The prompt phrase occasionally improves performance, but not systematically.Source: §3.3 · p. 3
3.4 With or Without Statute
- The experiments test each setup with and without relevant statutory text, using the most relevant section for four-shot and zero-shot prompts.When statutes are omitted, “section” references in the SARA cases are changed to “I.R.C. section.”Source: §3.4 · pp. 3–4
3.5 Discussion
- Some GPT-3 prompt settings surpass the BERT-based state of the art on the 100 entailment cases, with far less human input.Only the best model significantly exceeds the prior state of the art; on nine of ten setups, performance is at least as good on cases without numbers as on cases with numbers.Source: §3.5 · p. 4, Table 1 · p. 4
- Zero-shot tests without statutory text perform at chance or worse, while examples show GPT-3 misidentifying statutory subsections and confusing half-siblings with step-siblings.The subsection errors also occur when statutory text is included in the prompt.Source: §3.5 · p. 4
3.6 Does GPT-3 know about SARA?
- GPT-3’s responses to questions about SARA and 20 randomly selected cases did not indicate that it recognized the SARA dataset.Its answer to what SARA was described a multiple-choice test for legal professionals; none of the 20 source-identification answers implicated SARA.Source: §3.6 · pp. 4–5
4 GPT-3’S AWARENESS OF THE U.S. CODE
The authors probe GPT-3’s familiarity with the U.S. Code by asking it to identify statutes from their text and recite statutes from citations. GPT-3 often recognizes statutory sources, but its recitations are rarely faithful to the requested section.
4.1 Given statutory text from the U.S. Code, GPT-3 can often identify the title number
- Among 265 cases where GPT-3 identified the title but not the section, it supplied a numerical section for 250; 36 were off by one and 100 were off by nine or less.Source: §4.1 · p. 5
- GPT-3 identified the title and exact section for eight of the nine statutes used in SARA, misidentifying 26 U.S.C. §3301 as §3111.The authors suggest these tax-code sections may be more familiar because they are central to individuals’ tax calculations and likely received more discussion in GPT-3’s training corpus.Source: §4.1 · p. 5
- Given the body of U.S. Code text without its section name, section number, or title number, GPT-3 identified it as U.S. Code correctly 89% of the time and its title number 63%.Source: Table 2 · p. 5, §4.2 · p. 6
4.2 Given a U.S. Code citation, GPT-3 can recite plausible but incorrect statutory language
- Mean unpenalized-BLEU was 7.11 and median was 4.52 across 530 recitations; only 28 scored above 20, and GPT-3 never recited statutory text exactly.Its outputs often matched the Code’s formal style while omitting or inserting language, or captured an entirely different section’s gist.Source: §4.2 · p. 6
- Recall@1 was 1.5% and recall@5 was 4.7%; the correct section’s normalized rank averaged 0.48, suggesting GPT-3 paid little attention to the requested section number.A normalized rank of 0.5 was expected if it paid no attention to the section number.Source: §4.2 · p. 6
5 GPT-3 STRUGGLES WITH SIMPLE REASONING ON SYNTHETIC STATUTES
The authors test GPT-3 on novel synthetic statutes that are deliberately simple, then compare zero-shot and two-shot reasoning. GPT-3 makes frequent errors without examples; two examples improve accuracy, but performance still falls on wider or deeper statutes.
5.1 Creation of Synthetic Statutes
- The synthetic statutes encode repeated A=>B definitions; width counts parallel relations and depth the maximum number of applications, while the balanced, ambiguity-free rules are simpler than many Code sections.Terms were generated as nonce words or IDs to ensure the statutes were novel to GPT-3.Source: §5.1 · p. 6
5.2 Zero-Shot Experimental Setup
- The zero-shot tests pair synthetic statutes with one fact and an applicability question; parallel tests use semantically identical numbered sentences, and a second prompt requests a definitive Yes-or-No answer.Tests used both nonces and generated IDs, and the authors balanced positive and negative ground-truth answers.Source: §5.2 · p. 7
- In zero-shot tests, GPT-3 achieved around 78% accuracy even on the simplest 2-wide, 2-deep statutes, and accuracy declined as statutes grew wider or deeper.Source: Table 4 · p. 7, §5.2 · p. 7
- Of 2,272 errors in 9,000 nonce runs, 2,204 were false positives and 61 were false negatives; seven responses were non-definitive.Source: §5.2 · p. 8
5.3 Two-Shot Experimental Setup
- With two examples, GPT-3 reached 100% accuracy on 2-wide, 2-deep statutes but only 81% on 3-wide, 3-deep statutes, with accuracy tending to fall as width or depth increases.Two-shot performance was uniformly better than zero-shot performance; the tested statutes were far less complex than many U.S. Code sections.Source: Table 5 · p. 9, §5.3 · p. 9
- Two-shot tests give GPT-3 two correctly answered examples—one Yes and one No—before a test question about a different section.The examples use the same section as each other, while all three questions use different names and terms.Source: §5.3 · p. 9
CONCLUSION
GPT-3 performs better than the previous state of the art on SARA, but its results on unfamiliar synthetic statutes reveal persistent difficulty with statutory reasoning. The authors call for further research and plan to test newer language models.
CONCLUSION
- GPT-3 achieved 71% accuracy on SARA, above the previous state of the art, but scored 78% on simple 2-wide, 2-deep synthetic statutes and 81% on 3-wide, 3-deep statutes even with two examples.The authors note that the tested synthetic statutes are far less complex than statutes found in the U.S. Code.Source: CONCLUSION · p. 9
- The authors hope the work motivates research to improve language-model performance on statutory reasoning and plan to test newer models such as GPT-4.Source: CONCLUSION · p. 9