Source-linked AI summary
GPT Takes the Bar Exam
Michael Bommarito, Daniel Martin Katz
TL;DR
Legal language and Bar Exam questions pose a demanding test for a general-purpose language model. The paper evaluates GPT-3.5 on the MBE with zero-shot prompting, prompt engineering, hyperparameter optimization, and fine-tuning. GPT-3.5 outperformed guessing and reached passing performance in two categories, while its top-ranked alternatives were strongly correlated with correctness.
Problem
Because GPT-3.5 is not domain-specific and legal language is complex, its ability to succeed on legal assessments through zero-shot evaluation was an open question.
Method
The study experimentally evaluates GPT-3.5 on the NCBE MBE using zero-shot prompts, varied prompts and hyperparameters, and an attempted fine-tuning procedure.
Results
GPT-3.5 significantly exceeded random guessing, achieved passing rates in Evidence and Torts, and showed strong correctness correlation in its ranked alternatives.
Takeaways & Limitations
The results suggest that an LLM may pass the MBE component of the Bar Exam in the near future.
Takeaways & Limitations
Interpretation is limited by nascent scientific understanding of LLMs and the proprietary nature of GPT, including limited information about its provenance and retraining.
Abstract
from arXiv · showhide
Nearly all jurisdictions in the United States require a professional license exam, commonly referred to as "the Bar Exam," as a precondition for law practice. To even sit for the exam, most jurisdictions require that an applicant completes at least seven years of post-secondary education, including three years at an accredited law school. In addition, most test-takers also undergo weeks to months of further, exam-specific preparation. Despite this significant investment of time and capital, approximately one in five test-takers still score under the rate required to pass the exam on their first try. In the face of a complex task that requires such depth of knowledge, what, then, should we expect of the state of the art in "AI?" In this research, we document our experimental evaluation of the performance of OpenAI's `text-davinci-003` model, often-referred to as GPT-3.5, on the multistate multiple choice (MBE) section of the exam. While we find no benefit in fine-tuning over GPT-3.5's zero-shot performance at the scale of our training data, we do find that hyperparameter optimization and prompt engineering positively impacted GPT-3.5's zero-shot performance. For best prompt and parameters, GPT-3.5 achieves a headline correct rate of 50.3% on a complete NCBE MBE practice exam, significantly in excess of the 25% baseline guessing rate, and performs at a passing rate for both Evidence and Torts. GPT-3.5's ranking of responses is also highly-correlated with correctness; its top two and top three choices are correct 71% and 88% of the time, respectively, indicating very strong non-entailment performance. While our ability to interpret these results is limited by nascent scientific understanding of LLMs and the proprietary nature of GPT, we believe that these results strongly suggest that an LLM will pass the MBE component of the Bar Exam in the near future.
Introduction
Legal language combines unusual complexity with extensive domain knowledge, historically challenging NLP systems. This paper therefore evaluates whether general-purpose GPT-3.5 can handle the MBE using zero-shot prompts.
- Legal systems generate extensive textual demands across search, legal aid, contracts, e-discovery, and judicial analysis.
- Legal language uses specialized grammar, precise phrasing, and domain-specific meanings that require substantial education and training to understand.
- Traditional NLP systems often struggled with legal task assessments because legal text requires broad semantic and domain-specific knowledge.
- GPT-3.5 is not domain-specific, leaving its ability to succeed on legal assessments through zero-shot performance an open question.
- The study tests GPT-3.5’s text-davinci-003 model on the MBE using zero-shot prompts through OpenAI’s text completion API.
Data
The MBE is a standardized, multiple-choice component of the Bar Exam that tests legal knowledge and demanding reading comprehension. The study uses NCBE practice materials to assess model performance across law categories.
- The Bar Exam culminates years of legal education and exam-specific preparation, requiring accumulated theoretical knowledge and exam-specific question-solving skills.
- The Uniform Bar Examination includes multiple-choice, essay, and performance components, with the MBE typically worth 50% of the overall score.
- A raw MBE score of roughly 60% may produce a scaled score of approximately 133, sufficient to pass in several jurisdictions.
- MBE questions combine legal knowledge with reading comprehension, using embellished fictional situations containing both important and distracting details.
- The MBE contains approximately 200 questions spanning eight categories, including seven specific areas of law and one experimental category.
- The researchers purchased NCBE MBE practice questions and simulated exams, basing assessments on a downloadable study aid and answer key.
Methods
The experiment evaluates GPT-3.5 with varied prompts and API hyperparameters on NCBE MBE materials, including rank-ordering strategies and an attempted fine-tuning procedure. Rank-ordering substantially improved correctness, while the authors could not explain why.
- The researchers tested GPT-3.5 with zero-shot prompts through the text-davinci-003 completion API and documented prompt, hyperparameter, and fine-tuning procedures.
- Prompt Engineering and Responses: Prompt engineering varied requests for single answers, explanations, backup choices, re-prompting, and ranking two or all four choices.
- Prompt Engineering and Responses: Rank-ordering the top three choices improved model correctness substantially, unlike many other prompt variations whose results changed little.
- Responses were logged, parsed for scoring, and handled through exception cases for rare formatting variations; none were manually altered or human-evaluated.
- (Hyper)parameters for GPT-3: The researchers varied temperature, top p, best of, and maximum token settings across predefined values.
- The study also attempted to fine-tune GPT-3.5, although the supplied methods passages do not report the resulting comparison.
Results
GPT-3.5 substantially exceeded random guessing on the MBE but did not yet pass the overall multiple-choice exam. Performance reached passing levels in Evidence and Torts, while gaps varied considerably across legal categories.
- GPT-3.5 reached the average passing rate for Evidence and Torts.The reported result applies without fine-tuning.
- 17% was GPT-3.5’s average deficit relative to human test-takers across categories.The gap was negligible or single-digit for Evidence, Torts, and Civil Procedure, but reached 36% for Criminal Law.
- 50% average accuracy exceeded the 25% random-guessing baseline, but GPT-3.5 did not yet pass the overall multiple-choice exam.The MBE is only one component of the overall Bar Exam, so this result does not alone establish passage of the full exam.
- The authors identify missing knowledge and confusing exam language as possible contributors to GPT-3.5’s category-specific performance gaps.They state that interpretation is constrained by limited information about GPT-3.5’s provenance, architecture, and retraining changes.
- 71% of GPT-3.5’s top two choices and 88% of its top three choices were correct, indicating strong correlation between answer ranking and correctness.Across categories, top-two answers exceeded the 50% random-chance rate and exceeded NCBE-reported averages in five of seven categories.
- Civil Procedure was the notable exception where GPT-3.5’s top-ranked alternatives did not significantly exceed random chance.Civil Procedure was also reported as the worst category for human test-takers.
Conclusion and Future Work
GPT-3.5 outperformed random guessing on the MBE, passed two categories, and matched human test-takers in one without fine-tuning. The authors propose extending evaluation beyond the MBE and toward inspectable alternative models.
- GPT-3.5 significantly outperformed random guessing across all prompts and hyperparameter values, passed two Bar categories, and achieved parity with human test-takers on one.
- The authors report that GPT-3.5’s answer rankings were strongly correlated with correctness beyond random chance, indicating general legal-domain understanding.
- The authors caution that interpreting GPT-3.5’s answer selection is limited by nascent understanding of LLMs and GPT’s proprietary nature.
- Future work will investigate models that can be directly inspected, including GPT-J, GPT-Neo, and Bloom-family models.
- Future evaluations will examine GPT-3.5 and other models on the Bar Exam’s essay and situational-performance sections, beyond the MBE.