Source-linked AI summary
ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmark
Ilias Chalkidis
TL;DR
This paper tests ChatGPT’s performance on legal text classification amid claims of emergent zero-shot capabilities in large language models. It evaluates GPT-3.5-turbo on LexGLUE with templated instructions and finds that it exceeds random baselines but trails smaller fine-tuned models.
Problem
The paper examines how the first available ChatGPT model performs on legal text-classification tasks amid reported emergent zero-shot language-model capabilities.
Source: §1 · p. 1Method
The authors evaluate GPT-3.5-turbo on LexGLUE using templated instruction-following examples in a zero-shot setting.
Source: §1 · p. 1Results
Takeaways & Limitations
The authors recommend smaller fine-tuned models for similar practical legal text-classification applications.
Source: §2.3 · p. 3The authors caution against expecting out-of-the-box adequacy on similar tasks with fixed or predefined label sets.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Following the hype around OpenAI's ChatGPT conversational agent, the last straw in the recent development of Large Language Models (LLMs) that demonstrate emergent unprecedented zero-shot capabilities, we audit the latest OpenAI's GPT-3.5 model, `gpt-3.5-turbo', the first available ChatGPT model, in the LexGLUE benchmark in a zero-shot fashion providing examples in a templated instruction-following format. The results indicate that ChatGPT achieves an average micro-F1 score of 47.6% across LexGLUE tasks, surpassing the baseline guessing rates. Notably, the model performs exceptionally well in some datasets, achieving micro-F1 scores of 62.8% and 70.2% in the ECtHR B and LEDGAR datasets, respectively. The code base and model predictions are available for review on https://github.com/coastalcph/zeroshot_lexglue.
1 Introduction
The paper situates ChatGPT’s legal-task evaluation amid claims of emergent zero-shot capabilities in large language models. It audits GPT-3.5-turbo on LexGLUE legal text classification and reports performance above guessing baselines, with stronger results on two datasets.
1 Introduction
- Large language models have demonstrated emergent zero-shot natural language understanding capabilities as model sizes scale.The paper notes that these developments challenge the traditional supervised fine-tuning paradigm.Source: §1 · p. 1
- ChatGPT is described as an instruction-following transformer model aligned through reinforcement learning from human feedback, with reported capabilities extending to law.The paper identifies it as a successor to Instruct-GPT and a GPT-3.5 model.Source: §1 · p. 1
- Earlier legal bar-exam testing found 50.3% accuracy for text-davinci-003, while accounting certification testing reported a 14.4% correct rate.Source: §1 · p. 1
- ChatGPT achieves 49.0% average micro-F1 across LexGLUE tasks in zero-shot evaluation, exceeding baseline guessing rates and reaching 62.8% and 70.1% on ECtHR B and LEDGAR.The audit evaluates GPT-3.5-turbo using examples in a templated instruction-following format.Source: §1 · p. 1
2 Experiments
The experiments evaluate ChatGPT on seven LexGLUE legal text-classification tasks using sampled test examples and templated instructions. ChatGPT exceeds random-guess baselines, but smaller fine-tuned models perform better; few-shot results vary with label-set size.
2.1 LexGLUE Datasets
- LexGLUE comprises seven legal text-classification tasks, and the evaluation samples 1,000 test examples from each dataset using task-specific templates for documents, questions, and label options.Four datasets have about 1,000 test examples; EURLEX, LEDGAR, and CaseHOLD have about 5,000, 10,000, and 4,000, respectively.Source: §2.1 · p. 2
2.2 Experimental Setup
- The March 2023 gpt-3.5-turbo model receives one instruction-following example at a time; evaluation first checks exact label matches, then uses SentenceBERT cosine similarity if none match.Few-shot evaluation was limited to datasets with documents short enough for the 4k-token processing limit.Source: §2.2 · p. 2, §2.3 · p. 2
2.3 Results & Discussion
- Results compare random and training-distribution-biased guessing, zero- and few-shot ChatGPT, and fine-tuned LegalBERT models.Few-shot results are unavailable for some long-document datasets because of the 4k-token limit, including the template.Source: §2.3 · p. 2, Table 2 · p. 3
- ChatGPT exceeded random baselines but remained below smaller fine-tuned models in zero- and few-shot settings, scoring 62.8% and 70.1% micro-F1 on ECtHR B and LEDGAR.Source: Table 2 · p. 3, §2.3 · p. 3
- Few-shot prompting with K = 8 substantially improved performance on UNFAIR-ToS with 9 labels, while performance deteriorated on EURLEX and LEDGAR with 100 labels.The authors hypothesize that demonstrations covering few labels may bias generated labels toward that restricted set.Source: §2.3 · p. 3
- The authors conclude that smaller fine-tuned models should be preferred for similar practical legal text-classification applications.Source: §2.3 · p. 3
3 Limitations
The findings are partly restricted by the instruction templates and the decision not to tune prompts for computational reasons. The authors report that alternative prompts can bring minor gains, but do not expect them to overturn the comparison with fine-tuned models.
3 Limitations
- Findings are partly restricted to the developed instruction-following templates because prompt engineering or tuning was omitted to limit computational cost.Source: §3 · p. 3
- Three ECtHR B templates vary question framing and label detail; prompt engineering can lead to minor performance improvements.Source: §3 · p. 3, Table 3 · p. 3
- Predictions used a fixed sampling temperature of 0 for deterministic responses; other temperature values were not tested.Source: §3 · p. 3
- Other non-conversational GPT-3.5 models were omitted because they would cost 10× more, or approximately $500 total; the authors cite OpenAI’s claim that gpt-3.5-turbo is on par with Instruct Davinci.Source: §3 · p. 3
- The authors expect careful prompt engineering to add performance points but not make zero-shot ChatGPT competitive with fine-tuned models.Source: §3 · p. 4
4 Conclusions
In zero- and few-shot templated LexGLUE evaluation, ChatGPT exceeded random-guess baselines but was outperformed by smaller fine-tuned models. The authors characterize its zero-shot classification performance as poor overall and caution against expecting out-of-the-box adequacy in similar fixed-label settings.
4 Conclusions
- 49.0% average micro-F1 across LexGLUE tasks exceeded random-guess baselines, but smaller fine-tuned models outperformed ChatGPT by approximately 30%.Source: §4 · p. 4
- ChatGPT achieved micro-F1 scores of 62.8% on ECtHR B and 70.1% on LEDGAR.Source: §4 · p. 4
- The authors characterize zero-shot legal text classification as poor overall and caution against expecting out-of-the-box adequacy on similar tasks with fixed label sets.Source: §4 · p. 4
- The authors call for further study of more delicate prompt engineering and more flexible evaluation, and expect legal-oriented GPT models trained on legal corpora and similar task instructions to perform better.Source: §4 · p. 4