Source-linked AI summary
LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain
Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, Ilias Chalkidis
TL;DR
Legal NLP lacks a standardized multilingual benchmark, despite the prevalence of multilingual and multinational legal systems and the need to assess models across languages. LEXTREME surveys legal NLP literature and compiles datasets into a multilingual benchmark. Model performance varies across its datasets, with XLM-RLarge leading the aggregate but not most individual datasets.
Problem
Standardized multilingual legal benchmarks are lacking, despite the prevalence of multilingual and multinational legal systems and the need to evaluate models across languages.
Source: §1 · p. 2Method
LEXTREME surveys legal NLP literature and selects 11 datasets covering 24 languages to create a multilingual legal benchmark.
Source: §1 · p. 2Results
Performance varies substantially across datasets; XLM-RLarge tops the aggregate but achieves the best performance on only six of eleven datasets.
Source: §6 · p. 8Takeaways & Limitations
The authors release baselines and an open benchmark, invite future multilingual model evaluation, and envision community extensions with suitable datasets.
Source: §7 · p. 9The benchmark cannot establish model generality or whether a model understands natural legal language because task selection does not follow theories designed to prove generality.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Lately, propelled by the phenomenal advances around the transformer architecture, the legal NLP field has enjoyed spectacular growth. To measure progress, well curated and challenging benchmarks are crucial. However, most benchmarks are English only and in legal NLP specifically there is no multilingual benchmark available yet. Additionally, many benchmarks are saturated, with the best models clearly outperforming the best humans and achieving near perfect scores. We survey the legal NLP literature and select 11 datasets covering 24 languages, creating LEXTREME. To provide a fair comparison, we propose two aggregate scores, one based on the datasets and one on the languages. The best baseline (XLM-R large) achieves both a dataset aggregate score a language aggregate score of 61.3. This indicates that LEXTREME is still very challenging and leaves ample room for improvement. To make it easy for researchers and practitioners to use, we release LEXTREME on huggingface together with all the code required to evaluate models and a public Weights and Biases project with all the runs.
1 Introduction
Legal NLP is growing, but existing benchmarks are often saturated and predominantly English, while no standardized multilingual legal benchmark was available. LEXTREME addresses this gap with a broad dataset collection and baseline evaluations, while showing that model performance remains uneven.
1 Introduction
- Legal benchmarks are needed because existing benchmarks can be saturated, while legal models and resources remain predominantly English despite multilingual legal settings.The introduction notes that legal documents have specialized vocabulary and knowledge, and can be longer than general-domain texts.Source: §1 · pp. 1–2
- LEXTREME surveys legal NLP literature from 2010–2022 and selects 11 datasets covering 24 languages across eight subdivisions and two language families.The paper also describes evaluating five widely used multilingual encoder-based models.Source: §1 · p. 2
- DistilBERT scores 36.7 versus MiniLM’s 19.0 LEXTREME score despite having only marginally more parameters, 135M versus 118M.The passage also reports a correlation between model size and LEXTREME performance.Source: Figure 1 · p. 1, §1 · p. 2
- The authors release LEXTREME with evaluation code and a public leaderboard, frame it as a living benchmark, and integrate it with LexGLUE into HELM.They invite community contributions of multilingual legal datasets and aim to ease curated-benchmark evaluation of large language models.Source: §1 · p. 2
2 Related Work
Related work covers general and cross-lingual benchmarks, legal-domain benchmark suites, and legal language models. The paper positions LEXTREME among these resources as a multilingual legal benchmark spanning diverse datasets and languages.
2.1 Benchmarks
- GLUE and SUPERGLUE aggregate task scores, while MMLU targets pretrained knowledge and XTREME evaluates cross-lingual generalization across 40 languages.XTREME includes six predictive tasks over ten datasets; SUPERB and SUPERB-SG unify speech datasets but do not propose an aggregated score.Source: §2.1 · pp. 2–3
2.2 Benchmarks for the Legal Domain
- Legal benchmark suites include English-language LEXGLUE, multilingual fairness benchmark FairLex, Korean-focused LBOX, and the collaborative LEGALBENCH initiative.LEXTREME is motivated by LEXGLUE and LBOX and seeks to encourage multilingual models, diverse tasks, and datasets for legal NLP.Source: §2.2 · pp. 3–4
2.3 Legal Language Models
- Legal language models have been developed for several individual languages, and multilingual legal models were released recently but were unavailable at this paper’s submission.The authors therefore do not report results for those multilingual models.Source: §2.3 · p. 4
3 LEXTREME Datasets and Tasks
LEXTREME selects legal NLP datasets through a literature review and applies explicit inclusion and exclusion criteria. Its datasets cover three task types, with dataset configurations forming the basis of analyses and fine-tuning.
3.1 Dataset and Task Selection
- Selection required legal text and legal or NLU tasks, European-language coverage, human annotation, public availability, and exclusion of datasets without redistribution rights or with ML-generated labels.Source: §3.1 · p. 4
- 11 datasets were selected from 108 candidate papers after a review of legal NLP literature from 2010 to 2022.Three authors reviewed candidates and resolved disagreements through discussion and majority voting.Source: §3.1 · p. 4
3.2 LEXTREME Datasets
- LEXTREME datasets span single-label classification, multilabel classification, and named entity recognition, using existing splits when available and otherwise random 80/10/10 splits.Source: §3.2 · p. 5
- Classification tasks include Brazilian case outcomes and judicial unanimity, German argumentative functions, Greek legal topics, and Swiss appeal judgments.Source: §3.2 · p. 5
- Online Terms of Service contributes sentence-level unfairness classification and multilabel detection of nine clause topics.Source: §3.2 · p. 5
- C19 supports sentence-level classification of COVID-19 measures, while MultiEURLEX classifies EU laws using EUROVOC labels at three taxonomy levels.Source: §3.2 · pp. 5–6
- The NER datasets cover Greek, Romanian, Brazilian, and multilingual EUR-Lex legal texts, with MAP annotations at coarse and fine granularity.Source: §3.2 · p. 6
4 Models Considered
The paper evaluates multilingual and selected monolingual encoder models, using hierarchical variants for long legal documents. This addresses the 512-token limit of the evaluated pretrained models, while multilingual long-context alternatives were unavailable.
4 Models Considered
- The evaluated models are encoder-only and no model larger than 1B parameters was fine-tuned, reflecting the benchmark's NLU focus and resource constraints.Source: §4 · p. 6
4.1 Multilingual
- Five multilingual models trained on at least 100 languages were considered, including both base and large XLM-R; ChatGPT was evaluated zero-shot on text-classification tasks with fewer than 50 labels.Few-shot evaluation and sophisticated prompting were omitted because of long inputs and computational cost.Source: §4.1 · p. 6
4.2 Monolingual
- Monolingual models were fine-tuned only for languages represented in at least three datasets, distinguishing general-purpose Native-BERTs from legal-pretrained NativeLegalBERTs.Source: §4.2 · p. 6
4.3 Hierarchical Variants
- Hierarchical variants address long documents by encoding 128-token chunks, contextualizing chunk representations with a two-layer transformer, and max-pooling them into document representations.The resulting document representation is passed to a classification layer.Source: §4.3 · p. 6
- The evaluated pretrained models support at most 512 tokens, and multilingual long-context models such as Longformer and SLED were unavailable and predominantly English-trained.Source: §4.3 · p. 6, Table 4 · p. 7
5 Experimental Setup
The experimental setup uses macro-F1 across tasks and balances datasets and languages through harmonic-mean aggregation. Scores are reported by dataset subset, while jurisdiction is outside the benchmark's scope.
5 Experimental Setup
- Random oversampling was applied to imbalanced SLTC datasets except GLC, whose many labels would have made the resulting training data and fine-tuning costs grow drastically.BCD-U is cited as an example with a minority class of about 2%.Source: §5 · p. 7
5.1 Evaluation Metrics.
- Macro-F1 is used for every dataset because it applies to both text classification and NER, enabling comparability across the benchmark.Additional metrics and per-seed scores are available in the public Weights and Biases project.Source: §5.1 · p. 7
5.2 Aggregate Score
- Because dataset sizes and task complexity vary, scores are also reported for each dataset subset to support fine-grained analysis.Source: §5.2 · p. 7
- The aggregate combines dataset and language scores, using harmonic means to weigh datasets and languages equally.The dataset and language aggregates are each built through successive harmonic means across their respective configurations and datasets or languages.Source: §5.2 · p. 7
- Jurisdiction is not addressed because the authors consider it beyond this work's scope.Source: §5.2 · p. 7
6 Results
Results show a broad advantage for larger multilingual models, but performance varies substantially by dataset and less across languages. Monolingual models also perform strongly, while the tested supervised models outperform ChatGPT.
Larger models are better
- Larger models perform better overall, although individual dataset and language scores are more erratic; XLM-RBase's low MEU-3 performance weighs on both aggregates.The dataset aggregate uses a harmonic mean, and MEU-3 affects all 24 language scores.Source: §6 · p. 8
Differing model variance across datasets
- 52 points: the OTS gap between worst-performing MiniLM and best-performing XLM-RLarge illustrates substantial dataset variation, while XLM-RLarge leads only 6 of 11 datasets overall.Models differ by less than 6 points on GAM, SJP, and MAP; MiniLM also achieves the best performance on GAM.Source: §6 · p. 8
Less variability across languages
- 21 of 24 languages: XLM-RLarge outperforms the other multilingual models on most languages, with less variability across languages than across datasets.The authors report no language-level extremes like those seen for GLC, OTS, or C19.Source: §6 · p. 8
Monolingual models are strong
- 13 of 19 available languages: NativeBERT models reach top performance, while the few available NativeLegalBERT models slightly outperform multilingual models of the same size.NativeBERT models have the top language aggregate score; performance exceptions are Bulgarian, Spanish, Polish, and Slovak.Source: §6 · p. 8, Table 6 · p. 8
ChatGPT underperforms
- Much smaller supervised models outperform ChatGPT on all tested tasks, with very large gaps on GAM and OTS.ChatGPT was evaluated on 1,000 validation samples to avoid using test data.Source: §6 · p. 8, Table 7 · p. 8
7 Conclusions and Future Work
The authors present LEXTREME as a first multilingual legal NLP benchmark, built by selecting datasets from surveyed literature. They invite community extension and identify additional tasks, languages, models, and analyses for future work.
7 Conclusions and Future Work
- The authors select 11 datasets from 108 papers using rigorous criteria to compile the first multilingual legal NLP benchmark and provide baselines for five multilingual encoder models.They open-source the dataset and code and invite researchers and practitioners to evaluate future multilingual models on the benchmark.Source: §7 · p. 9
- LEXTREME is intended as a living benchmark that the community can extend with suitable datasets.The authors also plan to extend it with other NLU tasks and generation tasks such as summarization, simplification, and translation.Source: §7 · p. 9
- Future work includes more languages and models, deeper study of general-purpose versus legal-trained monolingual models, and analysis of jurisdiction information in language models and fine-tuning.The authors specifically name mT5 as a model to evaluate and propose adding summarization, simplification, and translation tasks.Source: §7 · p. 9
Limitations
The authors caution that benchmark coverage, data, and aggregate scores are limited, and that benchmark use can have negative consequences. They also note that many outcome-prediction datasets use judges’ decisions rather than parties’ documents.
Limitations
- LEXTREME cannot establish model generality or whether a model understands natural legal language because its tasks were not selected to prove generality.The authors say task selection reflects what is available or of interest to the community.Source: Limitations · p. 9
- LEXTREME uses real-world-scenario tasks and data to try to reduce the gap between benchmark and real-world performance, but benchmarks remain context-limited and subjective.The authors note that task formats may be more homogeneous than human activities and datasets can contain biases.Source: Limitations · p. 9
- Benchmark competition can encourage misbehavior, steer research toward benchmark-specific techniques, and support marketing that hides product flaws.The authors say these behaviors cannot be forecast in advance.Source: Limitations · p. 9
- Aggregated scores are intuitive but imprecise; LEXTREME weights tasks and languages equally rather than evaluating only an arbitrarily selected set of datasets.The authors acknowledge potential downsides while arguing for responsible use of aggregate evaluation.Source: Limitations · p. 10
- Many outcome-prediction datasets use judges’ decisions as input rather than documents submitted by the parties.The authors report that limited access to complaints, especially multilingual complaints, makes more realistic complaint-based datasets difficult to create.Source: Limitations · p. 10
Ethics Statement
The paper describes responsible-use concerns, experiment settings, and the multilingual and monolingual models covered by its evaluation. It recommends that these technologies assist human experts with notices on their limitations.
Ethics Statement
- The authors say legal technologies should assist human experts, with notices on their limitations, because irresponsible deployment is a plausible risk.They frame LEXTREME as supporting responsible and ethical deployment while acknowledging shortcomings.Source: Ethics Statement · p. 10
A.1 Maximum Sequence Lengths
- Maximum sequence lengths vary by dataset, from 256 tokens for several tasks to 8192 for MultiEURLEX when seeking maximum performance.The appendix also gives conditional settings, including 4096 or 8192 for MultiEURLEX and 2048 or 4096 for Swiss Judgment Prediction.Source: §A.1 · p. 16
A.2 Total compute
- The experiments used 689 GPU days.Source: §A.2 · p. 16
A.3 Hyperparameters
- Experiments used learning rate 1e-5, batch size 64, three random seeds, and validation-loss early stopping with patience five, with specified exceptions for MultiEURLEX.The full-dataset multilingual MultiEURLEX runs trained for one epoch and were evaluated every 1000 steps; mixed precision was used except with mdeberta-v3-base.Source: §A.3 · p. 16
B Model Descriptions
- MiniLM uses teacher-student distillation of final-layer self-attention, while DistilBERT reduces BERT’s layers by half and removes its pooler and token-type embeddings.Multilingual MiniLM was trained using XLM-RBASE; DistilBERT uses the same corpus as BERT.Source: B Model Descriptions · p. 16
- DeBERTa represents content and position separately in its attention mechanism; mDeBERTa-v3 further uses replaced-token detection and gradient-disentangled embedding sharing.Source: B Model Descriptions · pp. 16–17
- XLM-R is a multilingual model pretrained on a corpus comprising 100 languages, using RoBERTa-like objectives without next-sentence prediction.Source: B Model Descriptions · p. 17
C Monolingual Models Overview
- Table 8 lists the monolingual models, with batch size abbreviated as BS and pretraining corpora described in their publications where available.The authors note that specifications or publications were unavailable for some models.Source: Table 8 · p. 18
D Dataset Splits
The appendix reports split sizes for language-specific subsets of multilingual tasks, including both example counts and label counts. Values are ordered as train, validation, and test.
D Dataset Splits
- Table 9 reports train, validation, and test example counts for each language-specific subset of multilingual tasks.Source: Table 9 · p. 20
- Both tables order their split values as train, validation, then test.Source: Table 9 · p. 20, Table 10 · p. 21
- Table 10 reports train, validation, and test label counts for each language-specific subset of multilingual tasks.Source: Table 10 · p. 21
E Detailed Multilingual Results
The appendix provides multilingual and monolingual model results for validation and test sets. The tables report arithmetic mean macro-F1 and standard deviation over seeds.
E Detailed Multilingual Results
- Table 11 reports multilingual-model validation-set arithmetic mean macro-F1 and standard deviation over seeds, with best scores bolded.Source: Table 11 · p. 23
- Table 12 reports multilingual-model test-set arithmetic mean macro-F1 and standard deviation over seeds, with best scores bolded.Source: Table 12 · p. 23
F Detailed Monolingual Results
- Tables 13 and 14 report monolingual-model arithmetic mean macro-F1 and standard deviation over seeds for validation and test sets, respectively.Source: Table 13 · p. 25, Table 14 · p. 26
G Original Paper Results
Original-paper scores are reported across dataset configurations, but unavailable configurations have no scores and differences in evaluation procedures make direct comparisons challenging. The accompanying histograms show input sequence-length distributions, measured with multilingual-model tokenizers and truncated at the 99th percentile.
G Original Paper Results
- Different training and evaluation procedures across original papers make direct score comparisons challenging, and unavailable configurations have no reported scores.LEXTREME scores are calculated as the arithmetic mean across three seeds.Source: G Original Paper Results · p. 27
- Dataset-specific tables report original-paper scores for configurations including BCD-J and BCD-U, but the supplied captions provide no score values.The tables identify the best scores in bold.Source: Table 15 · p. 27, Table 16 · p. 27
H Histograms
- Input sequence-length histograms use token counts from five multilingual-model tokenizers and group values beyond the 99th percentile at the end.The histograms cover the LEXTREME datasets.Source: H Histograms · p. 32