Source-linked AI summary
Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents
Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu, Maosong Sun
TL;DR
Mainstream PLMs struggle with Chinese legal documents that contain thousands of tokens, limiting legal long-document understanding. The paper releases Lawformer, a Longformer-based pretrained model, and evaluates it across several LegalAI tasks, reporting strong performance on long-document tasks while identifying legal-knowledge integration and generation as future directions.
Problem
Mainstream PLMs cannot adequately process legal documents with thousands of tokens because their sequence capacity is limited.
Method
The paper pretrains Lawformer, a Longformer-based Chinese legal language model, on large-scale legal case documents and evaluates it across typical LegalAI tasks.
Results
Lawformer achieves strong performance on LegalAI tasks requiring long documents and comparable results with legal-corpus RoBERTa on short-input tasks.
Takeaways & Limitations
Lawformer provides a released Chinese legal pretrained model for understanding legal documents containing thousands of tokens.
Takeaways & Limitations
The authors identify legal knowledge augmentation as necessary for many LegalAI tasks and leave it for future work.
Abstract
from arXiv · showhide
Legal artificial intelligence (LegalAI) aims to benefit legal systems with the technology of artificial intelligence, especially natural language processing (NLP). Recently, inspired by the success of pre-trained language models (PLMs) in the generic domain, many LegalAI researchers devote their effort to apply PLMs to legal tasks. However, utilizing PLMs to address legal tasks is still challenging, as the legal documents usually consist of thousands of tokens, which is far longer than the length that mainstream PLMs can process. In this paper, we release the Longformer-based pre-trained language model, named as Lawformer, for Chinese legal long documents understanding. We evaluate Lawformer on a variety of LegalAI tasks, including judgment prediction, similar case retrieval, legal reading comprehension, and legal question answering. The experimental results demonstrate that our model can achieve promising improvement on tasks with long documents as inputs.
1 Introduction
LegalAI applies artificial intelligence to legal tasks, but mainstream PLMs struggle with legal documents that exceed their input limits. Lawformer addresses this gap with a Longformer-based Chinese legal model and evaluates it across diverse LegalAI tasks.
- LegalAI applies artificial intelligence, especially NLP, to legal tasks and has attracted attention from researchers and professionals.Its stated aims include improving legal practitioners’ work efficiency and providing timely aid to people unfamiliar with legal knowledge.
- 1260.2 tokens is the average length of criminal and civil case fact descriptions, exceeding the maximum length mainstream PLMs can handle.Limited long-sequence capacity prevents these PLMs from satisfactorily representing legal documents.
- Lawformer is a Longformer-based Chinese legal language model pretrained on large-scale legal case documents and able to encode thousands of tokens.It combines local sliding-window attention with global task-motivated full attention to capture long-distance dependency.
- The paper evaluates Lawformer on legal judgment prediction, similar case retrieval, legal reading comprehension, and legal question answering.These tasks require domain understanding, analysis of complicated case scenarios, and interpretation of legal provisions.
- New criminal and civil legal judgment prediction datasets are constructed, while pre-existing datasets support the other evaluated tasks.The authors report that existing judgment-prediction datasets differ from real-world distributions and omit civil cases.
- Lawformer achieves strong performance on long-document LegalAI tasks and comparable results with legal-corpus RoBERTa on short-input tasks.The reported evaluation covers various LegalAI tasks requiring long-document processing.
2 Related Work
Related work covers LegalAI applications, domain-adaptive pretraining, and methods for reducing the cost of long-document processing. Lawformer adopts Longformer-style attention to address the limitations of full self-attention.
- LegalAI research spans legal charge prediction, law article retrieval, court view generation, reading comprehension, question answering, and case retrieval.Prior work includes both handcrafted or statistical methods and deep-learning approaches, including legal-corpus pretraining.
- Domain-Adaptive Pre-training: PLMs are pretrained on large unlabelled corpora and support downstream NLP tasks, motivating domain-adaptive pretraining across biomedical, clinical, scientific, and legal domains.Domain-adaptive approaches further pretrain BERT on domain-specific texts.
- Long-Document Pre-training: Lawformer combines three attention mechanisms, using local windows, dilated windows, and global attention to model long documents.The example uses sliding-window and dilated-window sizes of 2, a gap of 1, and global attention for [CLS].
- Long-Document Pre-training: Traditional transformer PLMs are limited on long documents because full self-attention has high computational complexity.Prior approaches include left-to-right autoregressive objectives and sliding-window-based self-attention.
3 Our Approach
Lawformer combines multiple attention patterns with legal-domain pre-training to process Chinese legal documents containing thousands of tokens. It is evaluated across legal tasks using task-specific global attention selections.
- Attention Mechanisms: Longformer combines sliding window, dilated sliding window, and global attention mechanisms to encode long sequences.These attention patterns replace standard full self-attention.
- Attention Mechanisms: Sliding window attention restricts each token to surrounding tokens, while dilation introduces gaps to extend the reachable context.Different attention heads can use different dilation gaps.
- Attention Mechanisms: Global attention lets selected tokens attend to the whole sequence for task-specific representations.Questions receive global attention in question answering, while [CLS] receives it for text classification.
- Attention Mechanisms: The combined attention mechanisms process long sequences with linear complexity.
- Data Processing: Lawformer is pre-trained on Chinese criminal and civil case documents collected from China Judgment Online after filtering for fact descriptions longer than 50 tokens.Documents are divided into party information, fact description, court views, and judgment results before pre-training.
- Pre-training Details: Pre-training continues from RoBERTa-wwm-ext with an MLM objective using sequence length 4,096 and 200,000 training steps.Different documents are concatenated to use the full input length, and training uses 8 × 32G NVIDIA V100 GPUs.
4 Experiments
The experiments evaluate Lawformer across judgment prediction, case retrieval, reading comprehension, and question answering, using both new and existing datasets. Lawformer generally performs strongly, especially when tasks require processing long legal documents, though long-case retrieval and complex reasoning remain challenging.
- Legal Judgment Prediction: CAIL-Long contains 1,129,053 criminal cases and 1,099,605 civil cases with real-world length distributions.Criminal cases include charges, relevant laws, and penalty terms; civil cases include causes of action and relevant laws.
- Legal Judgment Prediction: Lawformer achieves the best micro-F1 and macro-F1 performance among four models on both criminal and civil judgment prediction.Its macro-F1 improvement suggests better handling of highly imbalanced labels, although overall judgment prediction remains unsatisfactory.
- Legal Case Retrieval: LeCaRD evaluates retrieval using 107 query cases and 10,716 candidate cases, with candidate cases averaging 6,319.14 tokens.Models judge whether each candidate case is relevant to a query case using binary classification.
- Legal Case Retrieval: 6.59 points: Lawformer improves mean average precision over baseline models on LeCaRD case retrieval.Baseline models cannot read complete documents, while Lawformer processes sequences with thousands of tokens; however, candidate cases average 6,319.14 tokens.
- Legal Reading Comprehension: L-RoBERTa and Lawformer outperform BERT and RoBERTa on CJRC, while L-RoBERTa performs comparably to Lawformer after long documents are filtered out.CJRC contains 9,532 question-answer pairs, and its documents average 441.04 tokens.
- Legal Question Answering: Lawformer and L-RoBERTa achieve comparable question-answering results, but all models struggle with the complex reasoning required by the task.The results indicate that enhancing legal knowledge and logical reasoning remains challenging.
5 Conclusion and Future Work
Lawformer is introduced as a Longformer-based model pretrained on tens of millions of Chinese legal case documents and evaluated across several LegalAI tasks. Although it improves performance on long-document tasks, the authors identify continuing challenges and propose legal-knowledge augmentation and generative legal models as future work.
- Lawformer is pretrained on tens of millions of criminal and civil case documents and evaluated on judgment prediction, similar case retrieval, reading comprehension, and question answering.
- The experimental results show significant performance improvement on tasks with long documents as inputs.
- The authors report that challenges in legal document understanding still exist despite Lawformer’s performance improvement.
- Future work will integrate legal knowledge into legal pre-trained language models because enhancing models with legal knowledge is necessary for many LegalAI tasks.
- The authors also plan to pretrain generative legal models, which could help legal professionals handle heavy and redundant writing work more efficiently.