Source-linked AI summary
LEGAL-BERT: The Muppets straight out of Law School
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, Ion Androutsopoulos
TL;DR
BERT adaptation in specialised domains lacked systematic evidence, particularly for legal language. The paper compares out-of-the-box use, further pre-training, and training from scratch across legal tasks, while broadening fine-tuning search. It finds that adaptation strategies outperform out-of-the-box BERT, smaller models can remain competitive, and the best strategy varies by domain and task.
Problem
BERT adaptation guidelines had been insufficiently investigated in specialised domains, and legal-domain adaptation had not been studied systematically.
Method
The paper systematically compares out-of-the-box BERT, further pre-training on legal corpora, and pre-training from scratch, using broader fine-tuning searches across legal datasets.
Results
Further pre-training or training from scratch performs better than out-of-the-box BERT on domain-specific tasks, while the best strategy varies and smaller models remain competitive.
Takeaways & Limitations
Domain adaptation should be evaluated systematically rather than following fixed pre-training and fine-tuning guidelines, and smaller models may suit low-resource settings.
Takeaways & Limitations
The released models are evaluated on three legal end-tasks and future work considers specific legal sub-domains such as EU legislation.
Abstract
from arXiv · showhide
BERT has achieved impressive performance in several NLP tasks. However, there has been limited investigation on its adaptation guidelines in specialised domains. Here we focus on the legal domain, where we explore several approaches for applying BERT models to downstream legal tasks, evaluating on multiple datasets. Our findings indicate that the previous guidelines for pre-training and fine-tuning, often blindly followed, do not always generalize well in the legal domain. Thus we propose a systematic investigation of the available strategies when applying BERT in specialised domains. These are: (a) use the original BERT out of the box, (b) adapt BERT by additional pre-training on domain-specific corpora, and (c) pre-train BERT from scratch on domain-specific corpora. We also propose a broader hyper-parameter search space when fine-tuning for downstream tasks and we release LEGAL-BERT, a family of BERT models intended to assist legal NLP research, computational law, and legal technology applications.
1 Introduction
BERT can underperform in specialised domains because generic pre-training does not capture domain-specific language. This paper compares out-of-the-box use, further pre-training, and training from scratch, and releases LEGAL-BERT models for legal NLP.
- Adaptation strategies: Specialised-domain BERT can be used out of the box, further pre-trained on domain corpora, or pre-trained from scratch with a new sub-word vocabulary.All three strategies include a final downstream fine-tuning step.
- Motivation: Generic-corpus BERT has been reported to underperform on specialised-domain text, motivating domain adaptation.Legal language has specialised vocabulary, formal syntax, and domain-dependent semantics.
- Key findings: Further pre-training and training from scratch outperform out-of-the-box BERT on domain-specific tasks, with mostly comparable performance across three legal datasets.
- Key findings: A broader fine-tuning hyper-parameter search than prior guidelines can substantially improve performance.
- Release: The authors release LEGAL-BERT, a family of models intended for legal NLP research, computational law, and legal technology applications.
- Key findings: LEGAL-BERT-SMALL achieves comparable performance to larger models while training approximately 4 times faster and using fewer resources.
2 Related Work
Prior domain-adaptation studies report gains from further pre-training or training from scratch, but they rarely compare strategies systematically or examine smaller models and broader fine-tuning choices.
- Existing approaches: Earlier studies applied further pre-training or training from scratch to biomedical, scientific, educational, and other specialised text, reporting downstream improvements.
- Open gaps: Previous work rarely investigates how the number of additional pre-training steps affects performance.Lee et al. is identified as the exception, although extending training to 1M steps showed no clear further improvement.
- Open gaps: Prior studies generally adopt Devlin et al.’s fine-tuning hyper-parameter guidelines without further investigation.
- Open gaps: The effectiveness and efficiency of smaller BERT-based models in specialised domains had not been considered.The paper notes that specialised-domain regularity and narrower topic ranges may reduce the need for larger models.
- Open gaps: Although BERT was state of the art in several legal NLP tasks, its adaptation to the legal domain had not been studied.
3 LEGAL-BERT: A new family of BERT models for the legal domain
LEGAL-BERT is built from diverse legal corpora using both further pre-training and training-from-scratch variants, including a substantially smaller model for efficiency.
- Training data: The authors collected 12 GB of diverse English legal text covering legislation, court cases, contracts, and other fields.
- Model variants: LEGAL-BERT-FP further pre-trains BERT-BASE on legal corpora for up to 500k additional steps, extending beyond the suggested 100k.
- Pre-training analysis: The paper compares training losses across all LEGAL-BERT versions during pre-training.
- Model variants: LEGAL-BERT-SC uses the BERT-BASE architecture with a newly created vocabulary of equal size to BERT’s vocabulary.
- Model variants: LEGAL-BERT-SMALL has 6 layers, 512 hidden units, 8 attention heads, and 35M parameters, versus BERT-BASE’s 110M parameters.It trains approximately 4 times faster and requires fewer hardware resources.
4 Experimental Setup
The experiments evaluate LEGAL-BERT variants on three legal classification and sequence-tagging datasets while expanding fine-tuning search beyond standard BERT guidelines.
- Pre-training: Pre-training uses 1M steps, batches of 256 samples, up to 512 sentencepiece tokens, Adam, and a 1e−4 learning rate.
- Evaluation datasets: The models are evaluated on EURLEX57K, ECHR-CASES, and CONTRACTS-NER across classification and named-entity-recognition tasks.CONTRACTS-NER contains contract-header, dispute-resolution, and lease-details subsets.
- Evaluation datasets: EURLEX57K supports large-scale multi-label classification of EU laws and is suitable for few- and zero-shot learning.
- Evaluation protocol: The experiments replicate earlier BERT fine-tuning studies across the legal datasets.
- Fine-tuning: The fine-tuning search uses batch sizes of 4, 8, 16, and 32, broader than the standard range.The authors use validation-loss early stopping because some models still underfit after four epochs.
5 Experimental Results
The experiments compare LEGAL-BERT variants across pre-training and downstream settings, showing that adaptation benefits vary by dataset and that smaller models can remain competitive.
- Pre-training Results: LEGAL-BERT-SC performs better on pre-training objectives than LEGAL-BERT-SMALL.LEGAL-BERT-SMALL reaches a loss similar to generic-corpus BERT-BASE by the end of pre-training.
- Pre-training Results: LEGAL-BERT-FP adapts faster and better to specific sub-domains than to the full legal corpus.This pattern is especially observed for ECHR cases and US contracts.
- End-task Results: The optimal further-pre-training strategy varies across datasets, so test experiments select the best development variant for each end-task.The selected LEGAL-BERT-FP variant is based on development results.
- End-task Results: LEGAL-BERT variants almost always outperform tuned BERT-BASE across the three datasets.Figure 4 reports perplexities and end-task scores as averages over multiple runs, with minimum and maximum ranges.
- End-task Results: ECHR-CASES shows a 2.5% improvement on the multi-label task, compared with a 0.8% improvement on binary classification.The corresponding perplexity difference is a 1.1 drop.
- End-task Results: CONTRACTS-NER shows perplexity and F1 improvements, including 1.8% for contract header and 1.6% for dispute resolution.Lease details also improves by 1.1%.
- Model Efficiency: LEGAL-BERT-SMALL is comparable to LEGAL-BERT across most datasets while fitting in most modern GPU cards.A hierarchical ECHR-CASES model otherwise causes a 4× memory increase.
6 Conclusions and Future Work
The paper concludes that BERT adaptation strategies should be evaluated systematically in legal domains because the best strategy varies by domain and task. It releases LEGAL-BERT models, including a smaller competitive variant, and identifies further sub-domain evaluation as future work.
- Conclusions: The best strategy for transferring BERT to a new domain may be further pre-training or pre-training from scratch.The paper releases LEGAL-BERT models achieving state-of-the-art results in three end-tasks.
- Conclusions: Performance gains are stronger on challenging tasks where in-domain knowledge is more important.Examples include multi-label ECHR-CASES classification and contract header and lease details in CONTRACTS-NER.
- Conclusions: LEGAL-BERT-SMALL is three times smaller yet highly competitive, supporting adoption in low-resource test-beds.The paper also argues that expanded fine-tuning grid search should be adopted.
- Future Work: Future work will evaluate LEGAL-BERT on more legal datasets and tasks and examine LEGAL-BERT-SC and LEGAL-BERT-SMALL in specific sub-domains.EU legislation is given as an example sub-domain.
A Legal NLP datasets
The evaluation uses three legal NLP datasets spanning legislative classification, human-rights case classification, and named entity recognition in US contracts.
- EURLEX57K: EURLEX57K contains 57k EU legislative documents annotated with EUROVOC concepts.Documents average 727 words, with approximately five labels per document and 45k/6k/6k train-development-test splits.
- EURLEX57K: EURLEX57K includes many rare labels and supports a multi-label classification setting.The dataset is divided into training, development, and test documents.
- ECHR-CASES: ECHR-CASES contains approximately 11.5k European Court of Human Rights cases mapped to violated Convention articles.It supports binary violation classification and multi-label identification of violated articles.
- CONTRACTS-NER: CONTRACTS-NER contains approximately 2k annotated US contracts from EDGAR.Annotations cover contract elements organized into contract header, dispute resolution, and lease details.
B Implementation details and results on downstream tasks
The downstream experiments replicate established task-specific BERT architectures and evaluation measures across classification and sequence-tagging datasets.
- EURLEX57K: EURLEX57K uses BERT’s final [CLS] representation followed by a linear layer with L sigmoid activations.The same configuration is used for all LEGAL-BERT variations.
- ECHR-CASES: ECHR-CASES uses a hierarchical BERT that independently encodes case facts before self-attention produces a document representation.A linear softmax layer produces the final scores.
- CONTRACTS-NER: CONTRACTS-NER feeds original BERT token representations into a linear CRF layer.Experiments cover contract header, dispute resolution, and lease details.
- Evaluation: Reported evaluation measures follow the prior experiments being replicated.The replications correspond to Chalkidis et al. across the three downstream tasks.
C Efficiency comparison for various BERT-based models
Table 2 compares BERT-based models across batch sizes on a single 11GB NVIDIA-2080TI, emphasizing resource efficiency beyond total parameter count. LEGAL-BERT-SMALL is reported as faster than ALBERT variants despite having more parameters.
- Table 2 compares BERT-based models at different batch sizes using a single 11GB NVIDIA-2080TI.
- Resource efficiency depends mainly on hidden units, attention heads, and Transformer blocks rather than total parameters.
- Transformer memory bottlenecks arise mainly from wider hidden representations and the number of attention heads, rather than simply from stacked-layer count.
- LEGAL-BERT-SMALL has 3× and 2× the parameters of ALBERT and ALBERT-LARGE but faster training and inference times.
- Models that reduce these resource limitations may be adopted by researchers and practitioners with limited resources.