Source-linked AI summary
Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence
John J. Nay, David Karamardian, Sarah B. Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H. Choi, Jungo Kasai
TL;DR
The paper examines how well LLMs apply tax law, an important question for legal services and AI governance. It evaluates models on generated tax questions and varies legal-context retrieval and prompting. Newer OpenAI models perform better, while the authors note that LLMs do not yet provide the contextual advice, ethical counsel, and nuanced judgment of most human lawyers.
Problem
The paper addresses the need to evaluate LLM legal analysis, which could affect legal services and AI governance.
Source: Introduction · pp. 2–3Method
The authors test models on newly generated tax-law questions and vary legal-text retrieval and prompting approaches.
Source: Our Approach to Validating LLM Legal… · p. 4Results
Newer OpenAI models show improving performance on thousands of tax-law inquiries, and GPT-4 benefits strongly from few-shot prompting.
Source: Introduction · p. 4, Results · pp. 10–11Takeaways & Limitations
The results demonstrate emerging tax-law understanding in LLMs when sufficient model capability is paired with prompting that elicits reasoning.
Source: Implications · p. 12LLMs do not yet provide contextual advice, ethical counsel, and nuanced judgment as consistently as most human lawyers.
Source: Implications · p. 12
Abstract
from arXiv · showhide
Better understanding of Large Language Models' (LLMs) legal analysis abilities can contribute to improving the efficiency of legal services, governing artificial intelligence, and leveraging LLMs to identify inconsistencies in law. This paper explores LLM capabilities in applying tax law. We choose this area of law because it has a structure that allows us to set up automated validation pipelines across thousands of examples, requires logical reasoning and maths skills, and enables us to test LLM capabilities in a manner relevant to real-world economic lives of citizens and companies. Our experiments demonstrate emerging legal understanding capabilities, with improved performance in each subsequent OpenAI model release. We experiment with retrieving and utilising the relevant legal authority to assess the impact of providing additional legal context to LLMs. Few-shot prompting, presenting examples of question-answer pairs, is also found to significantly enhance the performance of the most advanced model, GPT-4. The findings indicate that LLMs, particularly when combined with prompting enhancements and the correct legal texts, can perform at high levels of accuracy but not yet at expert tax lawyer levels. As LLMs continue to advance, their ability to reason about law autonomously could have significant implications for the legal profession and AI governance.
Introduction
The paper argues that systematic evaluation is needed because LLMs are black boxes, benchmark performance can be inflated by training-data overlap, and legal analysis has practical stakes. It focuses on tax law and reports improving performance across successive OpenAI models.
Introduction
- Systematic legal evaluation matters because LLMs may behave unpredictably on new tasks, and benchmark scores can be inflated when evaluation data overlaps with training data.The paper notes that inflated scores can create an illusion of understanding when performance reflects recognition instead.Source: Introduction · p. 2
- Legal analysis evaluations can inform AI governance, support more efficient legal services, and help identify inconsistencies in existing laws.The paper also discusses potential uses in legal advice, government transparency, and identifying outdated or silent areas of law.Source: Introduction · pp. 2–3
- Tax law supports automated evaluation because its authority is concentrated in the CFR and U.S. Code, many questions have definitive answers, and problem-solving can require logic and mathematics.The authors also cite tax law's regular significance to citizens' and companies' economic lives.Source: Introduction · pp. 3–4
- Across thousands of tax law inquiries, newer OpenAI models show improving performance, which the authors interpret as evidence of emerging legal understanding.The experiments compare models both alone and with legal-text retrieval and varied retrieval techniques.Source: Introduction · p. 4
Our Approach to Validating LLM Legal Understanding
The authors validate legal understanding using generated multiple-choice tax questions with known answers and legal citations. The questions use randomly generated facts, names, and numbers to avoid overlap with internet training data.
Our Approach to Validating LLM Legal Understanding
- Randomly generated facts, names, and numbers make the validation problems new and absent from the internet, reducing the risk that models encountered them during training.Each problem is multiple-choice with one correct answer.Source: Our Approach to Validating LLM Legal… · p. 4
- Each generated problem includes a legal question and answer options, the correct option, an explanation, and a citation to the governing legal text.Questions can test qualitative rules or arithmetic, including basis calculations.Source: Our Approach to Validating LLM Legal… · p. 4
- The evaluation uses separate CFR and U.S. Code exams, with three and four 100-question sections, respectively.Each section targets a specific tax-law question type.Source: Our Approach to Validating LLM Legal… · p. 4
- GPT-4 grades model answers by comparing each predicted option with the ground truth, enabling evaluation across more than 28,000 questions.The authors state that manual grading at this scale was infeasible and that model outputs were not consistently parseable.Source: Our Approach to Validating LLM Legal… · p. 5
Our Approach to Retrieval-Augmented Generation and LLM Prompting
The experiments compare models answering without supplied legal context against retrieval-based approaches and few-shot prompting. The authors also validate the GPT-4 grading method on a subset of questions.
Our Approach to Retrieval-Augmented Generation and LLM Prompting
- The bypass_retrieval baseline supplies only the question and answer options, testing what the model can answer from internal knowledge.It provides no additional legal context.Source: Our Approach to Retrieval-Augmented… · p. 5
- The GPT-4 evaluator graded nearly all validation-subset answers correctly, erring less than 1% of the time on ambiguous outputs from davinci.Manual review found these errors did not significantly affect the reported overall trends.Source: Our Approach to Retrieval-Augmented… · p. 5
- Similarity search retrieves four legal-source documents by cosine similarity and adds them to the prompt, while gold_truth supplies the correct legal source directly.The authors use gold_truth to estimate the effect of ideal retrieval and isolate reasoning errors associated with inaccurate similarity-search retrieval.Source: Our Approach to Retrieval-Augmented… · p. 6
- Few-shot prompting provides three question-answer examples matched to the problem type, excluding evaluation questions and supplying no source documents.The retrieval experiments use different prompt templates, and retrieved context may be truncated for models with smaller context windows.Source: Our Approach to Retrieval-Augmented… · pp. 5–6
The LLMs
The experiments use four increasingly advanced OpenAI models and include a final GPT-4 setting that combines gold_truth retrieval, few-shot examples, and chain-of-thought prompting. Figure 1 depicts comparisons across models, retrieval techniques, and prompting techniques.
The LLMs
- The four tested models are davinci, text-davinci-002, gpt-3.5-turbo, and gpt-4, spanning increasingly advanced OpenAI releases.The paper characterizes gpt-4 as more capable than any GPT-3.5 model and able to do more complex tasks.Source: The LLMs · p. 6
- The mega_run combines GPT-4 with gold_truth retrieval, few-shot prompting, and chain-of-thought prompting to assess an upper bound on performance with these techniques.The authors describe it as their most powerful model-and-technique combination.Source: The LLMs · p. 7
- Figure 1 compares experimental performance across multiple-choice tax law exams, LLMs, retrieval techniques, and prompting techniques.Source: Figure 1 · p. 7
Results
Accuracy rises with newer OpenAI models on both the CFR and U.S. Code exams. For GPT-4, CoT and few-shot prompting help, and combining them with the most relevant legal texts produces the best overall accuracy.
Results
- CoT does not consistently improve results across models and retrieval methods, but it boosts GPT-4 performance.The authors suggest models may need a certain capability level to benefit from additional reasoning.Source: Results · p. 8
- GPT-4’s example responses calculate an $8-per-share employer deduction at vesting and a -$4,000 loss on inherited property.Source: Results · pp. 8–9
- Accuracy improves slightly as the LLM receives more relevant legal source material through successive retrieval methods.Source: Results · p. 10
- Few-shot prompting strongly helps GPT-4 but is less consistently useful for weaker models on difficult tax questions.The “mega_run” combines few-shot examples with CoT and “gold truth” sources.Source: Results · p. 10
- GPT-4’s “mega_run,” combining few-shot and CoT prompting with “gold truth” legal sources, achieves the best overall accuracy on both exams.The setup combines the two prompting enhancements with the most relevant legal sources.Source: Figure 3 · p. 10, Results · p. 10
- Newer OpenAI models show higher overall answer accuracy on both the CFR and U.S. Code exams.The paper identifies the underlying LLM as the primary experimental factor consistently associated with accuracy increases.Source: Figure 4 · p. 11, Results · p. 11
Implications
The paper frames legal analysis by LLMs as relevant to legal services and AI governance, while emphasizing that current models remain short of professional tax lawyers and that the study covers U.S. tax law.
Implications
- The experiments are limited to U.S. tax law, although finding relevant authorities and applying them to facts are central legal-work capabilities that could generalize to other practice areas.The authors describe the work as a step toward adapting LLMs to reason about law autonomously and reliably.Source: Implications · p. 11
- Current LLMs underperform professional tax lawyers, and clear-cut legal questions cover only a small part of practising lawyers’ work.The paper says LLMs are not yet as consistent as most human lawyers at contextual advice, ethical counsel, and nuanced judgment.Source: Implications · pp. 11–12
- LLMs could assist lawyers with drafts or provide basic legal information, potentially increasing productivity and lowering legal-service costs.As autonomous basic legal advice becomes more capable, policymakers might reconsider rules on legal-service delivery, including unauthorised practice of law.Source: Implications · p. 12
- Legal-analysis capabilities could support AI governance through models that self-police or separate models that check whether another AI complies with law.The paper connects these approaches to aligning AI with humans.Source: Implications · p. 12
- The study demonstrates emerging tax-law understanding when models have sufficient general capability and prompting elicits reasoning behavior.The authors say these capabilities could affect law-making and require changes to legal-services regulation and emerging AI governance regimes.Source: Implications · p. 12
Related Work
Related work spans prompting, agents, evaluation, and external-knowledge augmentation. This paper studies simple prompting and legal-context retrieval for tax-law question answering, while situating legal understanding among capabilities that may emerge with scale.
Other prompting applications
- Studies of prompting and task workflows extend beyond legal QA, including prompting for scientific tasks and human-in-the-loop decomposition for science question answering.Source: Related Work · p. 13
Prompting
- Prompting research explores demonstrations, explanations, self-refinement, and reasoning-action patterns; this study tests simple prompting techniques for legal question answering.Prior work includes CoT and self-talk for compositionality, MORALCoT for rule-breaking judgments, and prompt-design approaches; the paper leaves more advanced legal-domain prompting for future work.Source: Related Work · pp. 12–14
LLM agents
- Research on LLM agents covers tool use, planning, memory, and interaction with environments, with some studies reporting improved task performance under particular methods.Examples include Cicero’s human-level Diplomacy play with strategic reasoning and planning improvements on more than 70 Minecraft tasks.Source: Related Work · pp. 14–15
Evaluation
- LLM evaluation research includes increasingly specific benchmarks, including G-Eval for language generation and AmbiEnt for ambiguous meanings.Source: Related Work · p. 15
Augmentation and question answering
- Augmentation research supplies external knowledge or tools to improve grounding, while question-answering work includes retrieval-augmented approaches that place grounding documents in model inputs.This paper evaluates retrieval of legal authorities as context for tax-law answers.Source: Related Work · p. 16
Emergent capabilities
- Some complex capabilities may emerge nonlinearly after models reach a sufficient scale, and the paper’s experiments suggest legal understanding may be one such ability.The passage contrasts GPT-2’s 1.5 billion parameters with GPT-3’s 175 billion as examples of model scale.Source: Related Work · p. 17
Next Steps
The authors identify prompt length, retrieval quality, advanced prompting, and legal-domain training as directions for further work. They especially note that similarity search did not consistently provide the gold-truth legal sources used in stronger GPT-4 results.
Next Steps
- The authors propose investigating whether longer prompts reduce accuracy, since a wider context window does not necessarily make filling it fully optimal.Source: Next Steps · p. 17
- Self-reflection and self-refinement prompting are identified as candidates for improving performance, including prompting a model to identify ambiguity and generate topics for further legal research.Source: Next Steps · p. 17
- GPT-4 performed better with gold-truth legal documents than with similarity-search retrieval, suggesting the search setup often missed the most relevant sources.The authors identify better retrieval as a way to narrow the gap between similarity search and gold truth.Source: Next Steps · p. 17
- Real-world retrieval needs testing because users will not supply the exact legal documents; the authors propose varying embeddings, retrieval methods, and retrieval length and count.Source: Next Steps · p. 18
- Future work could compare general-purpose pre-trained LLMs with models pre-trained and fine-tuned for legal reasoning, and develop fine-tuning practices for real-world legal tasks.Source: Next Steps · p. 18
Appendix
The appendix outlines CFR and U.S. Code tax-question types, then shows a GPT-4 prompt template using legal-context and question-answer-example placeholders.
CFR Exam Question Types
- CFR questions cover entity classification and elections, restricted property transferred as employee compensation, and the tax treatment of transfers that combine gifts and sales.The part-gift/part-sale questions ask about recognized gain or the recipient’s basis.Source: Appendix · p. 18
U.S. Code Exam Question Types
- U.S. Code questions cover restricted-property employer deductions, gains on sales of unrestricted compensation, basis rules for gifts and inheritances, part-donation/part-sale transfers, and the qualified business income deduction.The deduction question concerns Section 199A.Source: Appendix · pp. 18–19
GPT-4 “mega_run” Prompt Template
- The GPT-4 “mega_run” template instructs the model to use legal context and three multiple-choice question-answer examples, with a final question to answer.The shown example inserts “gold_truth” sources as context and includes a check-the-box question; other examples concern Rhode Island and Colombian entities.Source: Appendix · p. 20