Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Texts

1 INTRODUCTION

• The central question is how successfully general GPT-3.5 can classify short legal snippets using brief definitions of a non-hierarchical semantic type system.The evaluated tasks concern contract review, statutory and regulatory provisions, and case-law analysis.Source: §1 · p. 1
• The study compares GPT-3.5 annotations with a random forest and fine-tuned RoBERTa on the three legal-document tasks.GPT-3.5 is not fine-tuned, and its predictions use compact, one-sentence semantic-type definitions as prompts.Source: §1 · p. 1
• The evaluation focuses on batches of short text snippets from legal documents rather than whole-document annotation.The snippets are classified according to the categories in the type system.Source: §1 · p. 1

2 RELATED WORK

• Prior legal applications of GPT include entailment, bar-exam assessment, trademark distinctiveness, legal reasoning, and Supreme Court judgment modeling.The cited work reports improved state-of-the-art performance on COLIEE entailment [45] and successful GPT-3.5 and GPT-4 bar-exam applications [4].Source: §2 · p. 1
• Adjudicatory-decision research classifies short text segments by rhetorical, functional, or argumentative role, as well as identifying case outcomes.The cited examples span courts, administrative decisions, international courts, arbitration, and multiple jurisdictions,,, [1],,,, [32],.Source: §2 · p. 1
• Legal-norm classification includes statutory categories such as definitions, prohibitions, obligations, publication provisions, and scope of change.Other work analyzes obligations, permissions, agents, themes, and legal concepts,,,.Source: §2 · p. 1
• Contract research classifies clauses such as termination, governing law, and jurisdiction, while other datasets address lease entities and red flags.This study uses twelve selected semantic types from CUAD [16]; MAUD is another merger-agreement dataset [37].Source: §2 · pp. 1–2

3 DATA

• The experiments use three existing manually annotated datasets of short legal text, which the authors further filter and process for this study.The datasets support tasks involving different types of legal documents and include expert annotations.Source: §3 · p. 2
• The BVA dataset comprises 50 arbitrarily selected decisions from 2013–2017 concerning veterans’ PTSD claims, with sentences about factual issues annotated for rhetorical roles.The decisions were analyzed by Walker et al., and the sentences were manually annotated.Source: §3 · p. 2
• CUAD contains 510 lawyer-supervised commercial contracts and more than 13,000 annotations across 41 clause types; this study selects its 12 most common types.The dataset was released by Hendrycks et al. [16].Source: §3 · p. 2
• PHASYS covers public-health emergency-preparedness and response laws and regulations; this study classifies provisions by purpose into three categories.Relevant spans were coded using a codebook, and statutory and regulatory text units can comprise non-contiguous spans referenced by citations.Source: §3 · p. 2

4 EXPERIMENTS

• The Jaccard baseline assigns each snippet the label whose definition has the highest token-set similarity to that snippet.The comparison uses the task’s available type definitions.Source: §4 · p. 2
• The random forest is evaluated with document-level 10-fold cross-validation and grid search, while RoBERTa is fine-tuned for 10 epochs using the same splits.The random-forest grid search varies n-gram type, estimator count, and maximum tree depth; RoBERTa uses batches of 16 and sequence length 512, with Adam at 4e-5.Source: §4 · pp. 2–3
• Table 1 presents experimental results in micro-F1, identifies the Jaccard similarity baseline, and labels supervised-system training sizes with @N, where @Max denotes use of all available data points.Source: Table 1 · p. 3
• To assess labeled-data needs, the supervised models are trained on samples of 20, 50, 100, 250, 500, and 1,000 data points.The training documents are randomly sampled from each cross-validation training set.Source: §4 · p. 3
• To test text-davinci-003, snippets were batched within a 4,097-token prompt limit, leaving space for predictions, at sizes of 50 for BVA sentences, 20 for CUAD clauses, and 10 for PHASYS provisions.Source: §4 · p. 3
• The batch construction aims for cost effectiveness, which the authors note may somewhat limit the evaluated GPT-3.5 model’s performance.The snippets are batched as large as the prompt capacity allows.Source: §4 · p. 3

• The reported comparison is .35 vs .73 on BVA, .38 vs .86 on CUAD, and .24 vs .54 on PHASYS, with GPT-3.5 outperforming Jaccard in each.Source: §5 · p. 3
• Table 2 rows are predictions and columns human labels; errors include 654 BVA Evidence sentences labeled Reasoning and PHASYS Emergency Response provisions labeled Preparedness.CUAD confusions included Minimum Commitment, Profit Sharing, and Volume Restrictions.Source: Table 2 · p. 4, §5 · p. 4
• GPT-3.5 was competitive with limited in-domain data, but supervised models trained on large portions outperformed it; RoBERTa needed several hundred examples and random forest often close to a thousand.Source: §5 · p. 4
• For workflows with low tolerance for annotation errors, the authors say zero-shot predictions may need expert quality assurance; human review could also produce data for fine-tuning.Source: §5 · p. 4
• Performance varied across datasets: CUAD was described as reasonable, while BVA Reasoning remained difficult even for supervised systems, with RoBERTa at F1 = .71.The authors suggest compact definitions or word-occurrence features may miss the nuance of Reasoning labels.Source: §5 · p. 4
• PHASYS performance was unsatisfactory; Response made up 62.4% of its data, and unclear definitions and fine-grained Response–Preparedness distinctions may have contributed.Source: §5 · p. 4

6 CONCLUSIONS

• (Micro) F1 was .73 for adjudicatory-decision rhetorical roles, .86 for contractual-clause types, and .54 for statutory and regulatory emergency-response and preparedness purposes.The evaluated model was text-davinci-003.Source: §6 · p. 4
• The approach is presented as a way for legal professionals, educators, and scholars to lower costs in existing high-volume semantic-annotation workloads.Source: §6 · p. 5
• It could also unlock annotation workflows that would not be economically feasible to perform manually or using supervised ML.Source: §6 · p. 5