PaperlayerView exact revision on arXiv

Source-linked AI summary

Legal Prompting: Teaching a Language Model to Think Like a Lawyer

Fangyi Yu, Lee Quartey, Frank Schilder

arXiv:2212.01326v2cs.CLcs.AI

TL;DR

The paper asks whether prompting and explanation-based fine-tuning can improve large-language-model reasoning on Japanese legal entailment, a task evaluated on relatively small COLIEE datasets. It tests zero-shot, few-shot, chain-of-thought, legal-reasoning, and fine-tuning approaches. Legal-reasoning prompts achieve the strongest 2021 result, while 8-shot prompting performs best in 2022, improving prior best accuracies from 0.7037 to 0.8148 and from 0.6789 to 0.7431.

  • Problem

    The paper investigates how prompting and fine-tuning approaches perform on specialized legal reasoning, where COLIEE answers require determining entailment from Japanese statutory articles.

    Source: §1 · p. 1, §2 · p. 2, §5 · p. 8

  • Method

    The study evaluates pretrained GPT-3 with zero-shot, few-shot, chain-of-thought, legal-reasoning, and fine-tuning prompts, including generated or premise-based explanations.

    Source: §1 · p. 2, §4.3 · p. 4, §4.4 · p. 6

  • Results

    2021 accuracy rose from 0.7037 to 0.8148, while 2022 accuracy rose from 0.6789 to 0.7431; legal-reasoning prompts led in 2021 and 8-shot prompting in 2022.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Few-shot approaches were the most consistent across years, whereas zero-shot legal-reasoning prompts produced an outstanding result in one year but lower performance in the other.

    Source: §1 · p. 2, §5 · pp. 8–9

    The small COLIEE datasets constrain the evaluation, and the authors question whether prompting actually teaches a language model to think like a lawyer.

    Source: §2 · p. 2, §5 · p. 9

Abstract

from arXiv · show

Large language models that are capable of zero or few-shot prompting approaches have given rise to the new research area of prompt engineering. Recent advances showed that for example Chain-of-Thought (CoT) prompts can improve arithmetic or common sense tasks significantly. We explore how such approaches fare with legal reasoning tasks and take the COLIEE entailment task based on the Japanese Bar exam for testing zero-shot/few-shot and fine-tuning approaches. Our findings show that while CoT prompting and fine-tuning with explanations approaches show improvements, the best results are produced by prompts that are derived from specific legal reasoning techniques such as IRAC (Issue, Rule, Application, Conclusion). Based on our experiments we improve the 2021 best result from 0.7037 accuracy to 0.8148 accuracy and beat the 2022 best system of 0.6789 accuracy with an accuracy of 0.7431.

1 Introduction

The paper examines prompting and fine-tuning strategies for legal reasoning on Japanese bar-exam entailment data, where binary answers require appropriate legal rationale. Its experiments find that legal-reasoning prompts and few-shot or explanation-based approaches can improve performance, with the strongest results varying by test year.

1 Introduction

  • Legal professionals must apply potentially confusing and contradictory laws to varied scenarios and explain the rationale behind their conclusions.The paper motivates legal reasoning as more demanding than producing a simple binary response.Source: §1 · p. 1
  • 15.79% higher accuracy on the 2021 COLIEE test set came from a zero-shot legal-reasoning prompt, while an 8-shot approach improved 2022 accuracy by 9.46%.The paper evaluates zero-shot, few-shot, and fine-tuning approaches with and without explanations.Source: §1 · p. 2
  • 15.79% improvement over the prior COLIEE result was achieved on 2021 data, whereas 2022 performance improved by 9.46% with an 8-shot approach.These results motivate comparing prompt designs across years rather than relying on one evaluation setting.Source: §1 · p. 2
  • Few-shot and explanation-based fine-tuning produced good and consistent results across the two test sets, while zero-shot legal reasoning was strongest for only one year.The authors caution that zero-shot legal reasoning may be more prone to overfitting to a specific test set.Source: §1 · p. 2

2 Legal Entailment task

COLIEE task 4 asks systems to determine whether retrieved Japanese statutory articles entail a legal hypothesis. The task uses binary accuracy evaluation on relatively small datasets, making entailment reasoning the central challenge.

2 Legal Entailment task

  • Task 4 determines whether relevant statutory articles entail a hypothesis or its negation, producing a binary YES or NO answer.The organizers formalize the task as entailment between retrieved articles and Q or not Q.Source: §2 · p. 2
  • A worked example contrasts a hypothesis allowing rescission without a request with statutory premises requiring a request from specified parties.The example illustrates how the hypothesis must be checked against the retrieved legal provisions.Source: §2 · p. 2
  • About 0.5 accuracy is expected from random or always-YES/always-NO predictions because most test sets approximately balance positive and negative answers.Accuracy is the evaluation metric for the binary entailment task.Source: §2 · p. 2
  • 806 answered questions from prior competitions form the available pool, with 81 questions in the 2021 test set and 109 in 2022.The paper identifies the relatively small training and test sets as a main challenge.Source: §2 · p. 2

3 Prior work

Prior COLIEE task-4 systems largely used BERT-based models, while earlier GPT-3 work found that unprompted models performed poorly on Japanese legal entailment. Prompting and explanation-based methods provide the main conceptual foundation for the paper’s experiments.

3 Prior work

  • Most task-4 systems were BERT-based, including ensembles, while other approaches used DistilRoBERTa, LEGAL-BERT, T5, Electra, rule-based methods, or graph neural networks.The prior systems addressed the small training set with varied model and representation choices.Source: §3 · pp. 2–3
  • Earlier GPT-3 experiments without fine-tuning or prompting relevant Japanese statutes performed below the random baseline, indicating that stronger prompting or fine-tuning was needed.That work jointly addressed retrieval and entailment through the organizers’ task-5 setting.Source: §3 · p. 3
  • Chain-of-Thought prompting improved zero-shot and few-shot approaches on earlier reasoning tasks, motivating its evaluation for legal entailment.The cited prompting literature targets commonsense and mathematical reasoning as well as more complex tasks.Source: §3 · p. 3
  • STaR automatically retains correct model-generated rationales and fine-tunes on them, improving over answer-only fine-tuning on several reasoning tasks.The method is presented as an explanation-based alternative to manually created rationales.Source: §3 · p. 3

4 Experiments and results

The experiments compare zero-shot, few-shot, chain-of-thought, explanation-based fine-tuning, and legal-reasoning prompts on COLIEE 2021 and 2022 using GPT-3. Performance varies by prompting strategy and test year, with legal-reasoning prompts producing the strongest reported results.

4 Experiments and results

  • The evaluation uses GPT-3 text-davinci-002 on COLIEE 2021 and 2022, with accuracy as the metric because both test sets approximately balance positive and negative answers.The experiments cover zero-shot, few-shot, chain-of-thought, and fine-tuning variants.Source: §4 · p. 3, §4.1 · p. 3

4.1 Zero-shot (ZS)

  • 0.7407 accuracy: a more specific zero-shot prompt outperforms the 2021 winner’s 0.7037 accuracy by 3.70 percentage points.Replacing “the given premise” with “Japanese civil code statutes” reduces accuracy to 0.7037.Source: §4.1 · pp. 3–4, Table 1 · p. 4

4.2 Few-shot (FS)

  • 8-shot achieves the best 2022 few-shot performance, while 3-shot and 8-shot tie on 2021 and all few-shot settings beat their respective winners.The comparison is against 0.7037 for 2021 and 0.6789 for 2022.Source: §4.2 · p. 4, Table 2 · p. 4

4.3 Zero-shot-Chain of Thought (ZS-CoT)

  • 0.6296 and 0.7064 accuracy: ZS-CoT underperforms standard ZS on 2021 but outperforms the 2022 COLIEE winner by 4.05%.The pipeline first extracts reasoning and then extracts a formatted binary answer.Source: §4.3 · pp. 4–5, Figure 2 · p. 5

4.4 Fine-tuning LM with and without explanations

  • 23.99% improvement: fine-tuning with an instructive prompt outperforms fine-tuning without a prompt on the 2021 test set.Fine-tuning uses the 2021 COLIEE training set with binary-answer completions or explanations.Source: §4.4 · pp. 5–6, Table 3 · p. 6
  • Pseudo-explanation fine-tuning surpasses GPT-3-generated-explanation fine-tuning, but both underperform fine-tuning with instructive prompts and labels alone.Pseudo-explanations select the premise sentence most similar to the hypothesis using MPNet embeddings.Source: §4.4 · pp. 6–7

4 Experiments and results

  • Few-shot and explanation-based fine-tuning are comparatively consistent across both test sets, whereas zero-shot and label-only fine-tuning are more inconsistent.The authors caution that zero-shot legal reasoning performs best for only one year and may overfit a specific test set.Source: §4.5 · p. 7

4.5 Legal Reasoning (LR) prompt

  • 0.8148 accuracy: TRRAC beats the 2021 COLIEE winner by 15.79%, while IRREAC and IRRAC surpass the 2022 winner by 5.41%.These are the best-performing legal-reasoning approaches selected for the two test years.Source: Table 4 · p. 8, §4.5 · p. 8, Figure 6 · p. 8

5 Conclusions and discussion

The experiments find that legal-reasoning prompts and explanation-based fine-tuning can improve legal entailment performance, but results vary across test years. Prompting may improve performance without yet establishing that language models learn to think like lawyers.

  • 0.8148 accuracy improved the 2021 best system from 0.7037, while 2022 performance improved the best system from 0.6789 to 0.7431.The strongest 2021 result came from zero-shot legal-reasoning prompts, whereas 8-shot prompting performed best on 2022 data.Source: §5 · p. 8
  • A 1-shot approach underperformed zero-shot by 2.47 percentage points on the 2021 test set, suggesting that irrelevant examples may mislead the model.The authors propose selecting few-shot examples whose topics match those in the test set.Source: §5 · p. 9
  • Few-shot approaches produced good and consistent results across both COLIEE test sets, unlike zero-shot and label-only fine-tuning approaches.Zero-shot legal reasoning achieved the best result for one year but appeared more prone to overfitting to a specific test set.Source: §5 · p. 9
  • The study questions whether prompting actually teaches a language model to “think like a lawyer” and calls for testing explanation fine-tuning and legal-reasoning prompts across legal and other domains.The authors identify broader datasets and domains as areas requiring further exploration.Source: §5 · p. 9

A Appendix

The appendix presents legal entailment examples in which hypotheses are evaluated against statutory premises. The examples include both false statements corrected by the cited rule and a true statement supported by the premise.

  • The examples pair hypotheses with statutory premises and provide binary answers plus explanations grounded in the relevant legal rule.The appendix includes examples involving mortgages, assistance, agency, good-faith acquisition, leases, rescission, business entrustments, and partnership property.Source: A Appendix · pp. 10–12
  • A mortgage is not extinguished by prescription unless the secured claim is extinguished simultaneously.The example marks the contrary hypothesis false.Source: A Appendix · p. 11
  • Assistance applies to a person whose capacity is inadequate, whereas extremely inadequate capacity concerns curatorship rather than assistance.The example therefore marks the hypothesis as false.Source: A Appendix · p. 11
  • An unauthorized agent’s liability is determined by the counterparty’s choice, not by the agent’s choice.Article 117 makes the agent liable for performance or compensation unless authority or ratification is established.Source: A Appendix · p. 11
  • Good-faith acquisition of the jewelry fails because the possessor lacked the required permission and did not know the jewelry belonged to another.The example labels the hypothesis true while explaining why ownership cannot be acquired.Source: A Appendix · pp. 11–12
  • A lessor may assert agreed lease cancellation against a sublessee when the lessor had a cancellation right based on the lessee’s non-performance.This statutory exception makes the stated hypothesis false.Source: A Appendix · p. 12
  • The appendix also shows that rescission may require a specified party’s request, Article 656 applies mutatis mutandis to non-juridical business entrustments, and partnership property cannot be divided before liquidation.These examples correct hypotheses concerning court rescission, business entrustments, and partnership property.Source: A Appendix · p. 12
Loading 2212.01326v2…