PaperlayerView exact revision on arXiv

Source-linked AI summary

LEXam: Benchmarking Legal Reasoning on 340 Law Exams

Yu Fan, Jingwei Ni, Jakob Merane, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Etienne Salimbeni, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold, Mrinmaya Sachan, Alexander Stremitzer, Christoph Engel, Elliott Ash, Joel Niklaus

arXiv:2505.12864v7cs.CLcs.AIcs.LG

TL;DR

Long-form legal reasoning remains challenging for LLMs, while prior legal benchmarks have mainly assessed final-answer correctness rather than intermediate reasoning. LEXAM benchmarks open-ended and multiple-choice exam questions in English and German, and its results show that models struggle particularly with structured, multi-step open questions.

  • Problem

    Legal reasoning is difficult and under-assessed in NLP, and prior legal benchmarks have primarily evaluated final outputs rather than the intermediate steps needed for legally sound conclusions.

    Source: §1 · pp. 1–2

  • Method

    LEXAM pairs multilingual law-exam questions with reference answers and open-question reasoning guidance, and uses an ensemble LLM judge validated against human experts to assess reasoning quality.

    Source: §1 · pp. 2–3

  • Results

    State-of-the-art models struggle with multi-step reasoning and applying legal rules to novel scenarios, especially on open questions requiring structured legal reasoning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LEXAM offers a resource for legal NLP research and a scalable way to assess legal reasoning beyond simple accuracy metrics.

    Source: §1 · p. 3

    The benchmark is a research and educational resource, not a tool for real-world legal practice, and current LLMs should not be relied upon for legal decisions without qualified human oversight.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Long-form legal reasoning remains a key challenge for large language models (LLMs) in spite of recent advances in test-time scaling. To address this, we introduce LEXam, a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels. The dataset comprises 7,537 law exam questions in English and German. It includes both long-form, open-ended questions and multiple-choice questions with varying numbers of options. Besides reference answers, the open questions are also accompanied by explicit guidance outlining the expected legal reasoning approach such as issue spotting, rule recall, or rule application. Our evaluation on both open-ended and multiple-choice questions present significant challenges for current LLMs; in particular, they notably struggle with open questions that require structured, multi-step legal reasoning. Moreover, our results underscore the effectiveness of the dataset in differentiating between models with varying capabilities. Deploying an ensemble LLM-as-a-Judge paradigm with rigorous human expert validation, we demonstrate how model-generated reasoning steps can be evaluated consistently and accurately, closely aligning with human expert assessments. Our evaluation setup provides a scalable method to assess legal reasoning quality beyond simple accuracy metrics. Project page: https://lexam-benchmark.github.io/.

1 INTRODUCTION

Legal reasoning requires specialized knowledge and complex inference, yet LLM abilities in law remain under-assessed, particularly beyond final-answer correctness. LEXAM addresses this gap with a multilingual benchmark and an expert-validated approach to evaluating legal reasoning.

1 INTRODUCTION

  • Earlier legal benchmarks primarily scored final outputs, leaving the intermediate steps behind legally sound conclusions unevaluated.The paper frames legal reasoning as a high-stakes capability requiring specialized knowledge and abilities such as precedent interpretation, statutory analysis, and legal inference.Source: §1 · pp. 1–2
  • LEXAM contains 7,537 English- and German-language law exam questions, including long-form open questions and multiple-choice questions, with reference answers and normative guidance for open questions.The questions come from 340 exams across 116 University of Zurich law courses at bachelor’s and master’s levels.Source: §1 · p. 2
  • Current state-of-the-art models struggle with multi-step reasoning, applying legal rules to novel scenarios, and structured open questions.The paper also reports that model-generated reasoning steps can be evaluated consistently and accurately against human expert assessments.Source: §1 · pp. 2–3
  • An ensemble LLM-as-a-Judge framework provides scalable assessment of legal reasoning quality beyond simple accuracy metrics and closely aligns with human expert assessments.The paper also presents LEXAM as a resource for NLP research and a practical tool for educators evaluating AI capabilities.Source: §1 · p. 3

2 DATASET

LEXAM is constructed from law exams at the University of Zurich and provides open questions and generated multiple-choice formats with metadata for studying their distributions and characteristics.

2.1 CONSTRUCTION

  • The source exams span 340 exams and 116 University of Zurich courses from 2016–2023, with questions retained from courses in four legal areas.Although institutionally Swiss, the dataset covers international, comparative, and interdisciplinary topics; adding jurisdictions, particularly common-law contexts, is identified as future work. Human performance data are unavailable due to institutional restrictions, and MCQs do not appear in prior exams.Source: §2.1 · p. 3

2.1.1 OPEN QUESTIONS

  • For courses with at least ten open questions, five were randomly assigned to development and the remainder to test, yielding 300 development and 2,541 test questions..Source: §2.1.1 · p. 4

2.1.2 MULTIPLE-CHOICE QUESTIONS

  • Authors parsed MCQs into stems and statements, then generated questions with two to five statements and one correct option plus 3, 7, 15, or 31 distractors.True/false questions were also converted to this format, producing 4,696 MCQs; an auxiliary 385-question subset varies answer-choice counts from 4 to 32 while holding stems and statement order constant.Source: §2.1.2 · p. 4

2 DATASET

  • The dataset comprises 2,841 open questions and 4,696 multiple-choice questions with between 4 and 32 options.Questions are distributed across development and test sets; Figure 3 shows distributions by legal area, language, and jurisdiction.Source: §2.2 · p. 4, Figure 3 · p. 4

2.2 DATASET STATISTICS AND STRUCTURED FEATURES

  • Metadata records legal area, course, jurisdiction, language, exam year, and MCQ features including statement count and negative phrasing.English and German questions come from distinct exams rather than parallel translations.Source: §2.2 · pp. 4–5

3 EXPERIMENTAL SETUP

The experimental setup evaluates open legal answers with a specialized LLM judge and multiple-choice responses with accuracy, using selected models and standardized inference settings.

3.1 OPEN QUESTIONS

  • A specialized judging prompt tracks professor-written doctrinal structure and penalizes domain-specific errors, while an Alternative Annotator Test compares judges against human experts.Carefully constructed minimum-score ensembles of open-source judges can surpass human judges, and minimum-score ensembles consistently enhance evaluation performance.Source: §3.1 · pp. 5–6

3.2 MULTIPLE-CHOICE QUESTIONS

  • Multiple-choice evaluations use accuracy, with balanced choice labels making it an unbiased measure; stability is assessed by bootstrapping subsets.A random predictor achieves approximately 25% accuracy on four-choice questions, and a subset of frontier models is also evaluated on perturbed MCQs.Source: §3.2 · p. 6

3.3 LLM SETTINGS

  • The study evaluates 35 LLMs on open questions across reasoning, large, and small model categories, selecting models from diverse families and with multilingual abilities.Conventional models use temperature 0 and a maximum length of 4,096; reasoning models follow their official recommended settings.Source: §3.3 · p. 6

4 RESULTS

LEXAM results show substantial differences among models, with the strongest scores on open questions coming from reasoning models. Performance varies across question formats and metadata, and robustness tests underscore sensitivity to MCQ phrasing and structure.

4.1 RESULTS ON OPEN QUESTIONS

  • 70.20 was GPT-5’s highest average score on open questions, while GPT-5 and Gemini-2.5-Pro achieved the best performance.The results are reported in Table 1, which presents long-form open-question performance graded by an ensemble of LLM judges.Source: §4.1 · p. 7, Table 1 · p. 7
  • On open questions, reasoning models consistently outperformed large and small models; English scores exceeded German scores, with the largest language gap among small models.Interdisciplinary and public-law topics scored higher than criminal and private-law topics, while generic and international-law tasks scored higher than Switzerland-specific ones.Source: §4.1 · p. 7

4.2 RESULTS ON MULTIPLE-CHOICE QUESTIONS (16 CHOICES)

  • 52.53% for GPT-5.2 and 52.42% for Claude-4.6-Sonnet were the top two accuracy scores on MCQ-16.Bootstrap standard errors were typically below 1.5% across model groups.Source: §4.2 · p. 7
  • On MCQ-16, accuracy dropped for German courses, Swiss law, and negative phrasing, while public law scored below private and interdisciplinary law.Reasoning models maintained comparatively stable performance when the number of statements increased from four to five, while large and small models degraded; negative phrasing produced a particularly pronounced drop for reasoning models.Source: §4.2 · pp. 7–8

4.2.1 ROBUSTNESS CHECK ON MULTIPLE-CHOICE QUESTIONS WITH PERTURBATIONS

  • Perturbation results show significant performance differences across tested settings, indicating variability in model robustness and accuracy.The authors caution that standard MCQs may yield overly optimistic results because of sensitivity to spurious signals and guessing behavior.Source: §4.2.1 · p. 8

5 EXPERT EVALUATION FOR OPEN QUESTIONS

Expert ratings show substantial agreement, and the ensemble LLM judge passed a strict alternative-annotator test. Further analyses found close human-score alignment across languages, alongside differences in model leniency and larger gaps on borderline judgments.

5.1 INTER-ANNOTATOR AGREEMENT

  • Three legal experts’ ratings had average Pearson correlation r=0.70, mean quadratic weighted κ=0.49, and mean absolute error 1.95 on the 0–10 scale.The experts independently and blindly rated 50 randomly drawn question–answer pairs.Source: §5.1 · p. 9

5.2 ALTERNATIVE ANNOTATOR TEST

  • The Alt-test used three leave-one-out trials, compared the LLM judge with each held-out expert, and corrected p-values using Benjamini–Yekutieli at α=0.05.The test used m=3 experts, n=50 items, and ε=0.Source: §5.2 · p. 9
  • With ε=0, the ensemble LLM judge achieved a winning rate of ω=1.00, exceeding the recommended ω ≥0.5 threshold.The Alt-test compared the ensemble against three experts in a leave-one-out design.Source: §5.2 · p. 9, Table 3 · p. 9

5.3 FURTHER ROBUSTNESS ANALYSES OF LLM JUDGES

  • Judge models aligned closely with human scores in English and German, with only minor, statistically insignificant language effects.Leniency differed across models, and the largest model–human evaluation gaps occurred where human disagreement was higher.Source: §5.3 · p. 9
  • Qualitative review found recurring flawed legal reasoning, incorrect or fictitious authorities, and weaker German or multilingual proficiency in smaller models.The authors say these findings motivate an ensemble judge with minimum-score integration.Source: §5.3 · p. 9

6 RELATED WORK

The related-work discussion positions LEXAM as a challenging, broad legal-reasoning benchmark distinct from narrower task-specific datasets. It also distinguishes the work from prior Swiss legal NLP datasets based largely on court rulings and labels extracted through regexes.

6.1 INTERNATIONAL BENCHMARKS

  • LEXAM covers 78 legal subdomains and is described as the first general legal-reasoning dataset with challenging long-form ground-truth answers and the only dataset with an expert-tuned LLM judge.The comparison with prior legal QA datasets is summarized in Table 4.Source: §6.1 · p. 10, Table 4 · p. 10
  • Because prior legal NLP benchmarks use narrow, task-specific formulations, aligned direct comparisons with LEXAM would require substantial reformatting and are not meaningful.The authors position LEXAM as complementary to task-oriented datasets and provide descriptive comparisons instead.Source: §6.1 · p. 10

6.2 SWISS BENCHMARKS

  • Prior Swiss legal NLP tasks mainly use court rulings or legislation, and the authors report no prior Swiss legal dataset based on law-school exams.They collect real university law exams with detailed reference responses written by law professors; prior work often extracted labels using regexes rather than manual human annotation.Source: §6.2 · p. 10

7 CONCLUSION

The conclusion presents LEXAM as a benchmark for outcome- and process-based evaluation using 340 law exams. The authors report model variability, particular difficulty with complex open questions, and declining MCQ accuracy as answer-option counts increase.

7 CONCLUSION

  • LEXAM comprises 340 law exams and is designed for outcome- and process-based evaluation of LLM legal reasoning.Source: §7 · p. 10
  • The authors report substantial variability in model performance on MCQs and especially complex open questions.Source: §7 · p. 10
  • Increasing the number of MCQ options consistently reduced model accuracy, while the evaluation framework assesses legal reasoning quality beyond simple accuracy metrics.Source: §7 · p. 10

ETHICS STATEMENT

The benchmark uses publicly available, professor-written exam questions and model solutions obtained with permission, excluding personal, confidential, or sensitive material. The authors restrict its use to research and education, not real-world legal practice.

ETHICS STATEMENT

  • All questions and reference answers were obtained with permission from the University of Zurich Faculty of Law and are publicly available academic materials.The dataset excludes student-written answers and other personal, confidential, or sensitive content.Source: ETHICS STATEMENT · p. 11
  • No anonymization or institutional ethics review was required because the dataset contains no personal or human-subject data.The assessment was confirmed through consultation with multiple legal experts.Source: ETHICS STATEMENT · p. 11
  • The benchmark is for research and education, not legal practice, and its analyses do not constitute legal advice.The authors caution against relying on current LLMs for legal decisions without qualified human oversight.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

LEXam releases its dataset and evaluation pipeline for replication and extension. Its supporting analyses describe the Swiss legal-education context, dataset composition, and patterns in model performance on open questions.

B.2 DATASET STATISTICS

  • The authors release the full LEXAM dataset, including exam questions, metadata, and reference materials, together with the complete evaluation code, enabling replication and extensions of the benchmark.Source: REPRODUCIBILITY STATEMENT · p. 11

B.1 INSTITUTIONAL BACKGROUND

  • Swiss legal education emphasizes statutory interpretation and deductive reasoning, within a Bologna-model Bachelor’s and Master’s structure.The Bachelor’s curriculum is more fixed, while the Master’s permits greater specialization and electives.Source: §B.1 · p. 18

C.1 OPEN QUESTIONS

  • Performance is higher on non-challenging courses and the top ten courses than on challenging courses and the bottom ten.The challenging-course proxy covers German courses focused on Swiss law without an interdisciplinary approach; it was manually reviewed by legal experts.Source: §C.1 · pp. 22–23
  • Model performance trends downward on more recent open questions, especially from 2019 onward, while reasoning models remain ahead of other groups.Course availability and composition vary across years, so some variation may reflect changes in the course set rather than model effects.Source: §C.1 · p. 23
  • Performance stays relatively stable as question length increases but tends to fall as answer length grows; reasoning models remain higher and more stable in both conditions.The plots cap question length at 400 words and answer length at 600 words, covering about 90% of the data in each case.Source: §C.1 · pp. 23–24

C.2 MULTIPLE-CHOICE QUESTIONS

On multiple-choice questions, reasoning models lead the reported four-choice accuracy results, but performance varies with course features, question length, and the number of answer choices. Accuracy drops substantially as perturbation settings grow from four to 32 choices.

C.2 MULTIPLE-CHOICE QUESTIONS

  • 62.65% accuracy for GPT-5 and 58.01% for Claude-4.5-Sonnet lead the reported MCQ-4 results; GPT-4.1 follows at 54.40%.Bootstrap standard errors are typically below 1.3%.Source: §C.2 · pp. 24–25, Table 10 · p. 25
  • Models perform worse on challenging courses and modestly worse when an answer set includes a “none” option; accuracy declines across groups from 2018 onward.Reasoning models retain a consistent lead, and accuracy does not systematically fall as the number of legal statements increases.Source: §C.2 · p. 25, Figure 10 · p. 26
  • Accuracy falls to a local minimum around 450-word questions, then rebounds beyond that point, particularly for reasoning models above around 500 words.The authors suggest very long questions may provide richer context or clearer constraints.Source: §C.2 · p. 26

D FURTHER ANALYSES ON MCQS WITH PERTURBATIONS

  • Gemini-2.5-Pro leads across tested perturbation settings, but its accuracy drops from 68.6% at four choices to 35.6% at 32 choices.DeepSeek-V3 drops from 58.6% to 16% across the same settings, the largest reported decline.Source: D FURTHER ANALYSES ON MCQS WITH… · p. 26

D.1 PERFORMANCE PER COURSE

The further analyses show that model performance varies by course, language, and legal area, and falls as MCQ choice counts rise. The appendices describe the prompts, inference setup, expert annotation process, and agreement results used to examine and validate model outputs.

D.1 PERFORMANCE PER COURSE

  • Accuracy falls across courses as MCQ choice counts rise, with performance uniformly low across models and courses at 32 choices.In the four-choice setting, Gemini exceeds 75% in half the courses and peaks at 87.5% in criminal law and criminal procedure law; by 32 choices, even leading models fail to maintain reliable performance.Source: §D.1 · pp. 26–27

D.2 PERFORMANCE PER LANGUAGE

  • At 32 choices, Gemini achieves 49.6% accuracy in English versus 27.6% in German, a wider gap than at four choices.Across difficulty levels, models perform better in English than German.Source: §D.2 · p. 28, Figure 13 · p. 28

D.3 PERFORMANCE PER LEGAL AREA

  • At 32 choices, Gemini scores 49.6% in interdisciplinary law, 39.6% in criminal law, and 12.4% in public law.Interdisciplinary questions score highest and public law lowest across difficulty levels.Source: §D.3 · p. 28, Figure 14 · p. 29

E.1 PROMPT TO ANSWER OPEN QUESTIONS

  • The open-question prompt requests structured, exam-style legal analysis, while the MCQ prompt adds stepwise reasoning and elimination of incorrect options.Both prompts instruct models to assume Swiss law unless otherwise specified.Source: §E.1 · p. 29, §E.4 · p. 32

E.2 PROMPT TO JUDGE OPEN QUESTIONS ANSWERS

  • The judge compares model answers with professor-written reference answers using legal-reasoning criteria; a sample exposes errors such as citing Art. 193 PILA instead of the relevant Art. 176 PILA.The judging prompt was refined to account for domain-specific expectations, including fabricated statutory references and legally permissible solution paths.Source: §E.2 · p. 30, §E.3 · pp. 31–32

F INFERENCE HYPERPARAMETERS, COSTS, AND LLM ENDPOINT

  • Reasoning-model inference uses model-specific settings, while small conventional models run locally and closed-source models use official APIs or Together AI.Reported costs include about 7 A100 GPU hours for small models and $100 for GPT-4o judging.Source: F INFERENCE HYPERPARAMETERS, COSTS,… · p. 33

G.1 EXPERT RECRUITMENT AND EVALUATION DATA SELECTION

  • Three Swiss legal experts independently assessed 50 examples each; annotations used a 0–10 match score, with misleading content penalized by severity.The same 50 question–answer pairs were annotated under a law professor’s supervision; additional legally sound content was not penalized.Source: §G.1 · p. 33, §G.2 · pp. 33–34

G.3 FULL EXPERT-AGREEMENT BREAKDOWN

  • Expert pairs show correlations of r = 0.66–0.75, with moderate-to-substantial κ agreement, supporting the reliability of human scores.The appendix reports Pearson correlation, quadratic weighted Cohen’s κ, and mean absolute error for pairwise agreement.Source: §G.3 · p. 34, Table 11 · p. 34

G.4 EXPERIMENT DESIGN ON LLM JUDGE AND DETAILED RESULTS ON ALT-TEST

The authors test LLM judges against expert assessments, finding that minimum-score pooling performs strongly while constant-score baselines do not. Judge outputs align with human assessments across English and German, with no significant language effect in the reported analysis.

G.4 EXPERIMENT DESIGN ON LLM JUDGE AND DETAILED RESULTS ON ALT-TEST

  • Twelve judge settings are assessed on 50 expert-labeled examples, with human-comparable performance, impartiality, and reasonable cost set as desiderata.The Alt-test evaluates judges at two tolerance levels.Source: §G.4 · p. 34
  • Minimum-score pooling achieves a winning rate of 1.00 and advantage probability up to 0.76, outperforming individual judgments; the cost-efficient three-model ensemble reaches a 1.0 winning rate at tolerance 0.The ensemble combines GPT-4o, DeepSeek-V3, and Qwen3-32B.Source: §G.4 · p. 34, Table 12 · p. 35

G.5 ALTERNATIVE ANNOTATOR TEST: NAIVE BASELINES

  • Constant-score baselines never reach ω ≥0.5 at either ε level; their advantage probability ρ peaks around 0.507, below the LLM judges.The tested baselines assign the same score from 1 to 10 across all 50 items.Source: §G.5 · p. 35

G.6 FULL RESULTS FOR JUDGE MODEL ENSEMBLING

  • Qwen3-32B shows self-bias, and both Qwen3-32B and GPT-4o show family bias in assessments of long-form open questions.Table 14 reports model performance with bootstrapped standard errors and marks results potentially affected by self- or family bias.Source: §G.6 · p. 35

H QUANTITATIVE ANALYSES ON CROSS-LINGUISTIC CONSISTENCY OF JUDGE OUTPUTS

  • Judge scores correlate strongly with expert scores in English and German; language has a nonsignificant, small effect on score differences, and Gemini-2.5-Pro has the lowest R².The analyses use the 50 examples annotated by three legal experts.Source: H QUANTITATIVE ANALYSES ON… · p. 36

I QUALITATIVE ANALYSIS OF COMMON FAILURE MODES ACROSS DIFFERENT LEGAL DOMAINS

The analysis identifies recurring reasoning, legal-reference, language, and question-interpretation errors in model outputs, alongside differences in judge scores and human assessments. It also discusses limits on direct human–model comparison, the benchmark’s breadth, and plans to expand and update its data.

I QUALITATIVE ANALYSIS OF COMMON FAILURE MODES ACROSS DIFFERENT LEGAL DOMAINS

  • Frequent failures include flawed legal reasoning, incorrect legal references, misread questions, and inaccurate statutory paraphrases, sometimes paired with confident but shallow doctrinal conclusions.Some responses answer which law governs validity rather than whether PILA applies, missing the question’s focus.Source: I QUALITATIVE ANALYSIS OF COMMON… · pp. 36–37
  • Smaller models show weaker German and multilingual proficiency, sometimes mixing languages or producing incoherent responses.This language-related issue is reported as unique to smaller models.Source: I QUALITATIVE ANALYSIS OF COMMON… · p. 37
  • GPT-4o scores average +1.19 points versus human ratings, while Claude-4-Sonnet and Gemini-2.5-Pro score -0.39 and -0.33; DeepSeek-R1 scores +0.69, and discrepancies tend to rise when human annotators disagree more.Source: I QUALITATIVE ANALYSIS OF COMMON… · p. 37

J HUMAN PERFORMANCE ON LEXAM

  • Direct human–model comparisons on open questions are not appropriate because answer styles and judge-prompt fit differ, while sampling the dataset’s full topic range would overburden an individual student.Students typically answer under time constraints and are trained to be concise; a separate prompt for human answers would create a different evaluation standard.Source: J HUMAN PERFORMANCE ON LEXAM · p. 38
  • On 30 Swiss-law four-option questions, a bachelor-level law student scores 60% accuracy and a doctoral-level jurist 80%, offering a preliminary indication that advanced legal expertise outperforms current reasoning models on this subset.The authors note that the sample size is limited.Source: J HUMAN PERFORMANCE ON LEXAM · p. 38

K FUTURE PLANS FOR DATA COLLECTION

  • Because legal interpretations and applicable laws may change after exam questions are formulated, some answers may become outdated; metadata can identify affected subsets, and update detection is proposed.The authors cite a mechanism similar to LiveLongBench as a promising direction.Source: K FUTURE PLANS FOR DATA COLLECTION · pp. 38–39
  • The release prioritizes depth over breadth, emphasizing Swiss law while including other jurisdictions; broader common- and civil-law coverage is planned to support cross-system and cross-cultural analysis.The expansion effort names the United States, Germany, Mainland China, Hong Kong, Israel, and additional Swiss regions.Source: K FUTURE PLANS FOR DATA COLLECTION · pp. 38–39
Loading 2505.12864v7…