PaperlayerView exact revision on arXiv

Source-linked AI summary

LexGLUE: A Benchmark Dataset for Legal Language Understanding in English

Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, Nikolaos Aletras

arXiv:2110.00976v4cs.CL

TL;DR

Legal NLU lacks unified evidence about whether state-of-the-art models generalize across diverse legal tasks. LexGLUE standardizes seven legal datasets and evaluates generic and legal-oriented models, finding consistent advantages for the latter across multiple tasks, while remaining English-only and excluding anonymization.

  • Problem

    Legal NLP resources and evaluations are fragmented, leaving open how well current models generalize across diverse legal tasks.

  • Method

    LexGLUE unifies seven legal datasets and evaluates generic and legal-oriented Transformer models across standardized legal NLU tasks.

  • Results

    Legal-oriented models perform overall better, with approximately +2-4% p.p. m-F1 over equally sized generic Transformer models on SCOTUS and CaseHOLD.

  • Takeaways & Limitations

    LexGLUE provides a standardized basis for evaluating legal NLU models across multiple tasks and supports analysis of domain-specific pre-training.

  • Takeaways & Limitations

    The current LexGLUE version evaluates only English models and omits anonymization because suitable datasets were unavailable.

Abstract

from arXiv · show

Laws and their interpretations, legal arguments and agreements\ are typically expressed in writing, leading to the production of vast corpora of legal text. Their analysis, which is at the center of legal practice, becomes increasingly elaborate as these collections grow in size. Natural language understanding (NLU) technologies can be a valuable tool to support legal practitioners in these endeavors. Their usefulness, however, largely depends on whether current state-of-the-art models can generalize across various tasks in the legal domain. To answer this currently open question, we introduce the Legal General Language Understanding Evaluation (LexGLUE) benchmark, a collection of datasets for evaluating model performance across a diverse set of legal NLU tasks in a standardized way. We also provide an evaluation and analysis of several generic and legal-oriented models demonstrating that the latter consistently offer performance improvements across multiple tasks.

1 Introduction

Legal practice produces and depends on extensive textual data, motivating NLU systems that can handle legal language across tasks. LexGLUE introduces a standardized benchmark and tools to evaluate generic and legal-oriented models on diverse legal NLU datasets.

  • Motivation: Legal practice generates massive textual corpora, including briefs, statutes, contracts, patents, regulations, and judicial decisions.Analyzing and interpreting this text is central to lawyering and legal practice.
  • Motivation: Legal language contains specialized vocabulary, altered meanings, archaic expressions, foreign terms, unusual word order, and unusually long documents.These properties distinguish legal text from generic corpora and challenge standard language models.
  • Contribution: LexGLUE is a benchmark based on seven existing English legal NLP datasets for evaluating models across legal tasks.Its design is inspired by general-purpose multitask benchmarks such as GLUE and SuperGLUE.
  • Future work: Future versions may add datasets, tasks, and languages while prioritizing validity, annotation quality, statistical power, and social-bias checks.The authors explicitly identify these criteria for selecting additional resources.
  • Contribution: The benchmark aims to support generic models with limited task-specific fine-tuning and provide an accessible entry point for legal NLP research.Python APIs integrated with Hugging Face facilitate dataset access and model evaluation.
  • Contribution: LexGLUE is intended to improve interdisciplinary participation, transparent evaluation, and adoption of legal NLP methods.The authors contrast this goal with commercial claims that have received little independent evaluation.

2 Related Work

Legal NLP research spans prediction, classification, information extraction, question answering, and explanation across jurisdictions and languages. LexGLUE addresses scattered resources and inconsistent evaluation by unifying documented, public English datasets into a standardized benchmark.

  • Existing tasks: Legal NLP studies address court-outcome prediction, legal topic classification, information extraction, question answering, and decision explanation.These tasks have been explored across multiple legal document types and jurisdictions.
  • Existing tasks: Research covers legal materials from jurisdictions including the United States, China, France, the Philippines, Turkey, Thailand, the United Kingdom, Germany, and Switzerland.The literature includes both prediction and interpretability work on court decisions.
  • Pre-trained models: Transformer-based models pre-trained on legal corpora have achieved state-of-the-art results on several legal NLP tasks relative to generic-corpus models.This motivates evaluating domain-oriented models within a common benchmark.
  • Benchmark gap: The legal NLP literature is extensive, but its resources are scattered, documentation is often unavailable, and evaluation measures vary across studies.These inconsistencies make comparisons across work difficult.
  • Benchmark gap: LexGLUE creates a unified benchmark from representative, publicly available, documented English datasets of suitable size for modern NLP methods.The benchmark also introduces simplifications to improve standardization and accessibility.

3 LexGLUE Tasks and Datasets

LexGLUE selects seven English legal datasets using criteria covering language, legal substance, difficulty, availability, and size. The datasets span classification, multilabel prediction, and multiple-choice legal reasoning tasks, with documented simplifications and chronological splits where available.

  • Dataset desiderata: LexGLUE comprises seven datasets selected to evaluate understanding and reasoning about legal text on meaningful practitioner-oriented tasks.The benchmark’s first version is restricted to English.
  • Dataset desiderata: The selection favors tasks where state-of-the-art performance leaves substantial room for improvement and does not exclude tasks requiring legal expertise.This criterion distinguishes LexGLUE from benchmarks designed around near-saturated performance.
  • Dataset desiderata: Datasets must be public, documented by published articles, and sufficiently large; very small datasets are excluded to improve training and evaluation reliability.The authors cite unstable results from small test sets as a motivation.
  • Tasks and datasets: ECtHR Tasks A and B use case facts to predict violated versus allegedly violated ECHR articles, with chronological splits and a simplified ten-article label set.The source contains approximately 11K cases, split into training, development, and test periods.
  • Tasks and datasets: SCOTUS classifies Supreme Court opinions into 14 issue areas using SCDB metadata, with chronological training, development, and test splits.The dataset covers cases from 1946 through 2016 across the stated splits.
  • Tasks and datasets: UNFAIR-ToS labels sentences with sets of eight potentially unfair contractual-term types, while CaseHOLD selects masked holdings from five choices.The datasets use chronological splits where available, but CaseHOLD lacks chronological information for re-splitting.

4 Models Considered

The paper compares linear TF-IDF-SVM and several pre-trained Transformer models, including legal-oriented variants, while adapting models to legal documents that often exceed standard input limits.

  • 4.1 Linear SVM: The linear baseline is a Support Vector Machine using TF-IDF features from the top-K frequent training n-grams with n ∈[1, 2, 3].
  • 4.2 Pre-trained Transformer Models: BERT, RoBERTa, DeBERTa, Longformer, BigBird, Legal-BERT, and CaseLaw-BERT represent the examined pre-trained Transformer-based models.
  • 4.2 Pre-trained Transformer Models: Legal-BERT is pre-trained on legislation, contracts, and court cases with a vocabulary built from scratch for legal terminology, whereas CaseLaw-BERT is trained exclusively on US case law.
  • Hierarchical Variants: Because legal documents can exceed 512 sub-word units, the paper uses hierarchical variants that encode paragraphs separately and then process paragraph representations with a second-level Transformer.
  • Task Models: For text classification, document representations feed label-specific dense layers with sigmoid or softmax activation, while CaseHOLD scores five prompt–answer pairs with a softmax.
  • Implementation: The experiments use publicly available checkpoints and release pre-processed datasets, code, and instructions for loading the data and running experiments.

5 Experiments

The experiments evaluate generic and legal-oriented models across LexGLUE tasks using test results and aggregated scores, finding stronger overall performance for legal-oriented models but no universal winner.

  • 5.2 Experimental Results: Table 3 reports test results for all examined models across LexGLUE tasks, using hierarchical variants for starred datasets except Longformer and BigBird.
  • 5.2 Experimental Results: Legal-BERT and CaseLaw-BERT perform overall better, especially on m-F1, which accounts for class imbalance by treating all classes equally.
  • 5.2 Experimental Results: +2-4% p.p. m-F1 is the approximate improvement of legal-oriented models over equally sized generic Transformer models on SCOTUS and CaseHOLD.
  • 5.2 Experimental Results: On SCOTUS, the TFIDF-based linear SVM performs better than all generic Transformer models because its vocabulary and IDF scores are customized per task.
  • 5.2 Experimental Results: Legal-BERT performs as well as or better than CaseLaw-BERT on all datasets, despite CaseLaw-BERT being trained solely on US case law.

6 Vision – Future Considerations

The paper identifies longer documents, document structure, larger legal pre-training corpora, and larger language models as directions that could advance LexGLUE and legal NLP.

  • Long Documents: Long-document models support sequences up to 4096 sub-words, which is exceeded in three of seven LexGLUE tasks.
  • Long Documents: The hierarchical model handles up to 8192 sub-words, but its additional paragraph-level Transformer blocks are not pre-trained and may negatively affect performance.
  • Structured Text: Current long-document models do not consider document structure such as sentences, paragraphs, and sections, while SMITH hierarchically encodes increasingly larger blocks.
  • Large-scale Legal Pre-training: Existing legal language models are relatively small at 12–36 GB of pre-training data, and some cover narrowly defined legal domains.
  • Even Larger Language Models: Larger RoBERTa substantially improves performance over its base version and can match or exceed legal-oriented models in some cases.
  • Even Larger Language Models: The paper therefore indicates potential gains from pre-training larger legal-oriented models on larger legal corpora.

7 Limitations and Future Work

LexGLUE is representative but bounded in language, data access, coverage, and evaluation. Future versions aim to broaden datasets and languages while improving validity, annotation quality, statistical power, and bias assessment.

  • LexGLUE cannot cover the full legal domain, although it includes representative English datasets grounded partly in practically interesting applications.
  • The current benchmark evaluates only English models, while scarce non-English datasets make multilingual expansion challenging.
  • Copyright, trade-secret, bureaucratic, technological, and data-protection barriers restrict the creation and public availability of additional legal datasets.
  • LexGLUE omits anonymization because suitable datasets and benchmarks are unavailable, with inclusion planned for a later version.
  • Human evaluation is missing; re-annotation by multiple legal experts could estimate human-level performance and inter-annotator agreement, but would be costly.
  • Applying NLP to law also requires study of multidisciplinary limitations, fairness, robustness, and broader legal considerations of AI.

Ethics Statement

The benchmark relies mainly on publicly available legal resources and describes their licensing, privacy, and research-use basis. It also emphasizes reproducibility and transparent evaluation while identifying dataset and labeling considerations.

  • Except for SCOTUS, all LexGLUE datasets are publicly available and previously published, with users encouraged to cite the original works.
  • The authors report no ethical grounds for concern and present transparent benchmarking as a way to clarify tool capabilities and potential risks.
  • LexGLUE comprises seven reusable datasets, with data generally partially anonymized under applicable national law.
  • The datasets draw on public court decisions, legislation, contracts, terms of service, and case-law snippets, with dataset-specific privacy and legal-use rationales.
  • The work used no crowd-workers or annotators, and its experiments are intended to be reproducible through released access routes and code.
  • The paper states that it does not foresee harms to vulnerable or marginalized populations and does not knowingly use identifying characteristics for them.
  • The released repository supports Transformer-based models and replication scripts using five random seeds for each dataset.
  • For unlabeled instances in selected tasks, training uses binary cross-entropy, while evaluation adds an explicit no-label class so F1 reflects correct all-zero predictions.

C Additional Results

Additional results report development performance, variability, training time, and long-document comparisons. Hierarchical BERT substantially improves over truncated standard BERT on long-document tasks, while its training is faster than Longformer and BigBird.

  • Tables 5 and 6 report development µ-F1 and m-F1 results across all LexGLUE tasks using means and standard deviations from three selected seeds.
  • Development-score variability is relatively low, reaching up to 0.5% for µ-F1 and 1% for m-F1 across models and datasets.
  • Development results are generally higher than test results in many cases.
  • Hierarchical variants are used for starred datasets, whereas Longformer and BigBird remain exceptions.
  • Hierarchical variants are approximately 2× faster than Longformer and BigBird on ECtHR and SCOTUS tasks.
  • 10.2% p.p. is the ECtHR Task A gain from hierarchical BERT over standard BERT with truncated documents.
  • 7.5% p.p. is the ECtHR Task B gain, while SCOTUS gains 4.9% p.p. under the same hierarchical-versus-truncated comparison.
  • The authors attribute larger ECtHR gains to combining multiple distant facts, unlike SCOTUS topic classification, where long-range reasoning is not needed.

E Use of Roberta Large

RoBERTa-large substantially improves performance across all LexGLUE tasks compared with RoBERTa-base, with results comparable to or better than legal-oriented models. Its larger size and the models’ pretraining-data difference indicate potential gains from larger legal-oriented models trained on larger legal corpora.

  • Model configuration: RoBERTa-large uses 24 Transformer blocks, 1024 hidden units, and 18 attention heads.The evaluation uses AdamW with a 1e-5 maximum learning rate, 0.1 warm-up ratio, 0.06 weight decay, and mini-batches of 8 examples.
  • Results: RoBERTa-large, with 355M parameters, substantially improves performance over RoBERTa-base across all tasks.Table 8 reports development and test results using the run with the best development score.
  • Results: RoBERTa-large achieves results comparable to or better than Legal-BERT and CaseLaw-BERT in some cases.The comparison concerns the development and test results reported across LexGLUE tasks.
  • Interpretation: The legal-oriented models are much smaller and were pretrained with 5−10× less data than the evaluated large model.This provides a strong indication that larger legal-oriented models pretrained on larger legal corpora may improve performance.
  • Training considerations: Large models are sensitive to parameter updates, especially during initial training, making smaller learning rates and warm-up steps important.This training consideration accompanies the RoBERTa-large evaluation.

F Other Tasks and Datasets Considered

The authors considered CUAD and legal information-retrieval datasets but excluded them from the first LexGLUE version because of annotation problems, poor model results, long-document processing demands, and task-specific evaluation requirements.

  • CUAD: CUAD contains 510 contracts annotated with 41 contractual insights and formulates extraction as SQuAD-like question answering.Inputs pair a question naming an insight with a contract paragraph, and models identify the answer span.
  • CUAD: Models scored approximately 10–20% token-level F1 on CUAD, while Exact Answer Accuracy was 0% for all models.Legal-BERT performed slightly better than the other models by +5% F1; the original work used more lenient measures as well.
  • CUAD: CUAD annotations mix short entity-level answers with long paragraph-level answers, creating heterogeneous target lengths.Examples include document names and parties alongside lengthy termination provisions.
  • CUAD: Short and long annotations for the same insight introduce evaluation noise because the correct answer length becomes unclear.The ambiguity affects both training and evaluation.
  • CUAD: Indirect mentions, semi-redacted text, and fully redacted text further complicated CUAD annotations.These practices may hide sensitive information but could have been excluded from a benchmark dataset.
  • CUAD exclusion: These annotation issues and very low F1 scores led the authors to exclude CUAD from LexGLUE.Related contract-extraction studies used simpler, separate tasks and much larger datasets with better accuracy.
  • Information retrieval: Legal information-retrieval datasets were excluded because they process multiple long documents and require task-specific architectures and different evaluation measures.The examples target relevant case-law retrieval and regulatory compliance.

G Dataset Examples

LexGLUE training examples are presented as input–output or input–label pairs, with additional examples available through the Hugging Face dataset preview.

  • Example format: Table 9 presents training examples as pairs of inputs and outputs or labels for LexGLUE datasets and tasks.The table organizes examples under Dataset, Input(s), and Output(s) / Label(s).
  • Access: The dataset preview functionality on Hugging Face provides access to more LexGLUE training examples.The paper directs readers to the online dataset card for additional inspection.
  • Result correction: Corrected TFIDF-SVM results show that the method is significantly outperformed in most cases, including on SCOTUS.A grid-search bug had refit the model on both training and development data, overestimating development and test scores.
Loading 2110.00976v4…