Source-linked AI summary
Legal Transformer Models May Not Always Help
Saibo Geng, Rémi Lebret, Karl Aberer
TL;DR
The paper asks when domain-adaptive pretraining is useful for legal NLP and evaluates it across tasks and training-data sizes, alongside adapter-based tuning. It finds DAPT most helpful in low-resource settings, reports comparable adapter performance to full-model tuning, and releases LegalRoBERTa.
Problem
The paper asks when costly domain-adaptive pretraining improves legal NLP, given that existing legal models can show only limited gains over general models.
Source: §1 · p. 1Method
The authors compare domain-pretrained language models across legal tasks and training-data sizes, benchmark adapters, and further pre-train RoBERTa on legal corpora.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The authors recommend domain-pretrained models for low-resource tasks and release LegalRoBERTa as a legal-domain RoBERTa model.
Source: §1 · p. 1, §9 · p. 6LegalRoBERTa pretraining was constrained by the available legal-corpus size and seemingly insufficient pretraining steps; EU-legislation models used identical hyperparameters without repeated random-seed experiments.
Source: §8 · pp. 5–6
Abstract
from arXiv · showhide
Deep learning-based Natural Language Processing methods, especially transformers, have achieved impressive performance in the last few years. Applying those state-of-the-art NLP methods to legal activities to automate or simplify some simple work is of great value. This work investigates the value of domain adaptive pre-training and language adapters in legal NLP tasks. By comparing the performance of language models with domain adaptive pre-training on different tasks and different dataset splits, we show that domain adaptive pre-training is only helpful with low-resource downstream tasks, thus far from being a panacea. We also benchmark the performance of adapters in a typical legal NLP task and show that they can yield similar performance to full model tuning with much smaller training costs. As an additional result, we release LegalRoBERTa, a RoBERTa model further pre-trained on legal corpora.
1 Introduction
The introduction frames legal NLP as a longstanding effort to automate legal work, then asks when costly domain-adaptive pretraining (DAPT) is worthwhile. It reports stronger DAPT benefits for low-resource tasks and introduces adapters and LegalRoBERTa as additional contributions.
1 Introduction
- Legal NLP has a long history of applying language-processing systems to activities such as contract review and legal advice.The introduction traces early online legal search systems to the 1960s and 1970s.Source: §1 · p. 1
- The paper presents LegalRoBERTa, a RoBERTa model further pre-trained on legal corpora.The introduction situates it alongside existing legal language models and prior work on DAPT and TAPT.Source: §1 · p. 1
- Because unsupervised pretraining can take several days even with good computing resources, the authors question whether it is worthwhile when gains are small.They cite prior work reporting only slightly better legalBERT performance than BERT on three legal NLP tasks.Source: §1 · p. 1
- DAPT produced more benefit when downstream tasks had little training data, while legal models showed limited boosts on rich-resource text classification.The authors tested one rich-resource and one low-resource task, including versions with different training-data sizes.Source: §1 · p. 2
- Adapters matched full-model fine-tuning in performance while requiring less disk space and offering a more efficient fine-tuning approach.The introduction also describes adapters as faster to train.Source: §1 · p. 2
2 Contributions
The contributions combine a legal-domain RoBERTa model, a retrieval adaptation of a legal summarization dataset, an empirical account of when DAPT helps, and an adapter evaluation for legal text classification.
2 Contributions
- LegalRoBERTa was pre-trained on less legal text than legalBERT yet achieved similar performance.Source: §2 · p. 2
- The authors adapted an existing legal summarization dataset for a legal text retrieval task.The dataset is Galgani and Hoffmann (2010).Source: §2 · p. 2
- Current open-source legal language models brought marginal benefit or no improvement on a rich-resource task, while DAPT helped on low-resource tasks.The authors also observed DAPT benefits when they deliberately reduced training data.Source: §2 · p. 2
- Adapters produced results comparable to full-model fine-tuning on a legal text classification task.Source: §2 · p. 2
3 Related Work
The related-work section positions the study as examining DAPT's value in legal NLP and extending legal-domain adaptation from BERT toward RoBERTa. It notes that concurrent work reported a similar conclusion about DAPT and downstream tasks.
3 Related Work
- The study addresses the gap between BERT and RoBERTa in legal NLP and investigates whether DAPT helps, noting concurrent work reached a similar conclusion about DAPT and downstream tasks.The authors say they were unaware of that work until their own study was mostly finished.Source: §3 · p. 2
4 Language Model Pre-training: LegalRoBERTa
The section describes the legal text collected for LegalRoBERTa and its domain-adaptive pre-training setup. It also reports training outcomes and constraints on the pre-training configuration.
4.1 Legal Corpora Description
- Around 5 GB of clean legal text was collected for domain-adaptive pre-training, despite limited public resources and some documents being PDFs.The authors note that legal documents may contain sensitive information, limiting what institutions release publicly.Source: §4.1 · p. 2
4.2 Comparison with Other corpora
- The legal corpus was significantly smaller than corpora used in other domain-adaptive pre-training experiments.Source: §4.2 · p. 2, Table 2 · p. 2
4.3 Pre-training Details
- Pre-training RoBERTa-BASE continued to 446k steps, beyond Devlin et al.'s suggested 100k, following advice that prolonged in-domain pre-training may help downstream fine-tuning.Source: §4.3 · p. 3
- The legal-corpus loss fell from 1.850 to 0.880, and perplexity after domain-adaptive pre-training was 2.2735.Source: §4.3 · p. 3
- Limited GPU memory constrained the batch size relative to similar domain-adaptive pre-training runs, and the authors suggest further legal-corpus pre-training may be beneficial.Source: §4.3 · p. 3
- RoBERTa-BASE had 1M pre-training steps on generic corpora, which the authors describe as skewing it toward generic language.The passage points to next-token prediction examples comparing LegalRoBERTa with RoBERTa and LegalBERT.Source: Table 10.1 · p. 3
5 Downstream Legal NLP Tasks and Model Testing
The section evaluates legal language models on EU-law classification and Australian case catchphrase retrieval. Domain-pretrained models show a clearer advantage in the low-resource retrieval task than in rich-resource classification.
5 Downstream Legal NLP Tasks and Model Testing
- The evaluation covers multi-label classification on EUR-LEX and information retrieval on Australian legal cases.The tested models included legal models, original BERT and RoBERTa models, and randomly initialized RoBERTa.Source: §5 · p. 3, Table 3 · p. 3
Experimental setup
- EUR-LEX contains 57K legislative documents and 4,271 labels divided into frequent, few-shot, and zero-shot groups by training-document counts.The groups contain 746 frequent, 3,362 few-shot, and 163 zero-shot labels.Source: Experimental setup · p. 3, Table 5 · p. 4
- The Australian dataset has 3,886 Federal Court cases; the study recasts catchphrase summarization data as retrieval because BERT is not well suited to text generation.The retrieval task uses case sentences to find the correct catchphrases.Source: Experimental setup · p. 4, Table 7 · p. 5
- Catchphrase retrieval adapts image-caption embedding ranking to case-description/catchphrase pairs, using a shared representation space and triplet loss.It has one ground-truth catchphrase per case and only one-tenth the training data of the classification task.Source: Experimental setup · p. 4
- Retrieval performance is measured with Recall@K, mean rank, and median rank on a 389-case test set; lower mean and median ranks are better.The authors note that retrieval metrics depend on test-set size.Source: Experimental setup · p. 4
- In catchphrase retrieval, all domain-pretrained models beat their corresponding originals, while EU-law classification showed only a slight advantage for Legal-BERT over BERT.The classification comparisons were run once with a fixed random seed, so the slight margin could reflect statistical error.Source: Experimental setup · pp. 3, 5, Table 4 · p. 4, Table 8 · p. 5
6 Varying Downstream task data size
The section tests whether domain pretraining’s effect changes with downstream training-set size. Its reported benefit is significant with scarce data and negligible when data are sufficient.
6 Varying Downstream task data size
- The experiments vary LMTC training-data size to compare BERT with LegalBERT.The comparison is reported in Table 6.Source: §6 · p. 5
- Domain pretraining’s improvement is tightly related to downstream training-data size, with significant benefit when data are scarce and negligible benefit when data are sufficient.The comparison varies LMTC training-data size for BERT and LegalBERT.Source: §6 · p. 5
7 Adapters in legal text classification
Adapters add a task-specific module while keeping the language model fixed, and they match full-model fine-tuning on performance but are not faster to train.
7 Adapters in legal text classification
- The authors motivate adapters by the time and storage demands of fine-tuning a full language model separately for multiple tasks.Source: §7 · p. 5
- Training updates only the added adapter parameters, leaving the language model unchanged, and each task needs only its adapter saved.This reduces per-task storage compared with saving a full fine-tuned model.Source: §7 · p. 5
- Adapters matched full-model fine-tuning in performance, but training was not faster because computation remained similar.The forward pass and back-propagation still required a similar amount of calculation.Source: §7 · p. 5
8 Limitations and Future Work
The authors identify constraints on LegalRoBERTa pretraining and LMTC evaluation, and suggest further testing for legal case retrieval.
8 Limitations and Future Work
- Limited legal-corpus availability constrained LegalRoBERTa pretraining; the authors suggest collecting more data, including through automatic scraping.Source: §8 · p. 5
- Pretraining steps appeared insufficient, while EU-legislation LMTC used identical hyperparameters because time and computing resources were limited.The authors suggest grid search and repeated runs with different random seeds.Source: §8 · p. 6
- Paired statistical testing could determine whether domain-pretrained models significantly outperform original models on legal case retrieval.Source: §8 · p. 6
9 Conclusion
The conclusion finds that domain pretraining helps on low-resource tasks but not significantly on rich-resource tasks, and recommends its use accordingly.
9 Conclusion
- Existing legal transformer models showed no significant improvement on a rich-resource task but considerable improvement on low-resource tasks or reduced training sets.Source: §9 · p. 6
- The authors recommend domain-pretrained language models only for low-resource tasks.Source: §9 · p. 6
- The paper reports that adapters are suitable for legal NLP tasks and releases Legal-RoBERTa, a RoBERTa model adapted to the legal domain.Source: §9 · p. 6
Appendix
The appendix provides related-work pretraining details, LegalRoBERTa prediction examples, a catchphrase-retrieval illustration, an LMTC hyperparameter, and results for different training-data sizes.
10.1 Domain Adaptive Pre-training Details of Related Work
- The appendix table presents domain-adaptive pretraining details for related work.Source: §10.1 · p. 8
10.2 Examples of Next-Token-Prediction Results of LegalRoBERTa
- The cited material presents two next-token-prediction examples: Table 10 is labeled “Example 1” and contains a masked agreement sentence, while Table 11 is labeled “Example 2” and contains a masked custody-treatment sentence.Source: Table 10 · p. 8, §10.2 · p. 8, Table 11 · p. 8
10.3 Model Architecture in Catchphrase Retrieval task
- The appendix illustrates automatic catchphrase retrieval from legal court case documents.Source: §10.3 · p. 8
10.4 Hyper-parameters in LMTC task
- The LMTC hyperparameter list specifies a maximum sequence size of 216.Source: §10.4 · p. 9
10.5 Results on LMTC with different sizes of training data
- The appendix identifies a table of BERT and LegalBERT LMTC results across different training-data sizes.Source: §10.5 · p. 9