Source-linked AI summary
Understand Legal Documents with Contextualized Large Language Models
Xin Jin, Yuchen Wang
TL;DR
Large legal caseloads require substantial human effort to process documents. The paper applies context-sensitive models to rhetorical-role prediction and legal NER, reporting better performance than baselines, including Legal-LUKE 15.0% better than our baseline BERT-CRF in F1 score.
Problem
Growing caseloads demand substantial effort to process legal documents, while variable, noisy legal language complicates rhetorical-role classification and entity recognition, motivating automated support for judges.
Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 3Method
The authors model rhetorical roles as sequential sentence classification with Legal-BERT-HSLN capturing intra- and inter-sentence context, and use contextualized, entity-aware Legal-LUKE for legal NER.
Source: §1 · pp. 1–2, §3.1 · p. 3Results
The proposed models outperform baselines across tasks; Legal-LUKE is 15.0% better than our baseline BERT-CRF in F1 score.
Source: §1 · p. 2Takeaways & Limitations
The paper frames automated understanding and highlighting of key information and context as a way to assist judges and other users of legal documents.
Source: §1 · p. 1
Abstract
from arXiv · showhide
The growth of pending legal cases in populous countries, such as India, has become a major issue. Developing effective techniques to process and understand legal documents is extremely useful in resolving this problem. In this paper, we present our systems for SemEval-2023 Task 6: understanding legal texts (Modi et al., 2023). Specifically, we first develop the Legal-BERT-HSLN model that considers the comprehensive context information in both intra- and inter-sentence levels to predict rhetorical roles (subtask A) and then train a Legal-LUKE model, which is legal-contextualized and entity-aware, to recognize legal entities (subtask B). Our evaluations demonstrate that our designed models are more accurate than baselines, e.g., with an up to 15.0% better F1 score in subtask B. We achieved notable performance in the task leaderboard, e.g., 0.834 micro F1 score, and ranked No.5 out of 27 teams in subtask A.
1 Introduction
Growing legal caseloads and long, noisy, terminology-heavy documents motivate automated legal-document understanding. The paper proposes contextualized models for rhetorical-role prediction and legal named entity recognition, reporting better baseline performance and a fifth-place leaderboard result.
1 Introduction
- Legal caseload growth motivates automated methods to help people process and understand relevant information in long legal documents.The introduction cites more than 47 million pending cases in India as an example.Source: §1 · p. 1
- Legal documents challenge automated processing with technical and domain-specific terminology, extreme length, typos, and unstructured text.The introduction notes that long documents challenge dependency-based methods and RNNs in extracting context.Source: §1 · p. 1
- Rhetorical-role prediction structures noisy documents into coherent units, while named entity recognition identifies key legal entities for judges and later automation.The paper presents both tasks as key steps in legal-document processing and understanding.Source: §1 · p. 1
- Legal-BERT-HSLN ranked No. 5 of 27 teams with a 0.8343 micro F1 score on rhetorical-role prediction.The introduction also reports Legal-LUKE as 15.0% better than baseline BERT-CRF in F1 score.Source: §1 · p. 2
- The paper formalizes rhetorical-role prediction as sequential sentence classification and proposes Legal-BERT-HSLN for comprehensive sentence semantics and Legal-LUKE for contextualized, entity-aware legal NER.The introduction reports that evaluations showed better performance than baselines.Source: §1 · pp. 1–2
2 Background
The background introduces sequential sentence classification as context-aware classification across sentence sequences and describes legal NER as a means of extracting legally relevant entities for document analysis.
2.1 Sequential Sentence Classification
- Sequential sentence classification labels a sequence of sentences using sentence semantics and context, where a sentence’s classification may depend on surrounding sentences.The passage notes that classification can occur at sentence or document level and is used in tasks such as summarization and topic classification.Source: §2.1 · p. 2
2.2 Legal Named Entity Recognition
- Legal NER detects and labels legally relevant entities in unstructured reports, supporting information mining and document retrieval or mapping to taxonomies and cases.The passage also highlights entity recognition for anatomies, findings, and anatomical locations in legal reports containing complex medical terminology.Source: §2.2 · p. 2
3 System Overview
The system addresses rhetorical-role classification as sequential sentence classification and legal NER with distinct context-aware architectures. Legal-BERT-HSLN models intra- and inter-sentence context, while Legal-LUKE combines legal-contextualized word and entity representations.
3.1 Rhetorical Role Classification
- Rhetorical-role classification is challenging because models must assign predefined categories while capturing long-term dependencies between sentences.The passage identifies dependency capture as essential to determining overall textual meaning and context.Source: §3.1 · pp. 2–3
- Legal-BERT-HSLN models rhetorical roles with Legal-BERT token embeddings, within-sentence Bi-LSTM and attention, surrounding-sentence context, and a linear-plus-CRF output layer.The architecture is presented in Figure 1.Source: §3.1 · p. 3, Figure 1 · p. 3
3.2 Legal Named Entity Recognition
- Legal NER faces noisy documents, morphological variation, misspellings, and out-of-vocabulary words that complicate entity recognition..Source: §3.2 · p. 3
- Legal-LUKE uses preprocessing and contextualized, entity-aware representations to address legal NER; its encoder consumes words and entities and produces legal-contextualized representations.The model builds embeddings from token, type, and position embeddings; entity positions are associated with corresponding word tokens.Source: §3.2 · pp. 3–4
4 Evaluations
The evaluations compare contextualized legal models with baselines on rhetorical-role classification and legal named entity recognition. Legal-BERT-HSLN ranked fifth in the rhetorical-role test leaderboard, while Legal-LUKE outperformed NER baselines on validation data.
4.1.3 Test Environment
- Legal-LUKE training moved to a second server after out-of-memory issues on the initial evaluation machine.The original setup had four GTX 1080 Ti GPUs; the second server used an RTX A5000.Source: §4.1.3 · p. 5
4.1 Experimental Setup
- Rhetorical-role classification assigns one of 13 labels per sentence; training comprised 247 documents and 30k sentences, validation comprised 30 documents and 3k sentences, and all evaluations used validation.Subtask A baselines included BERT-Base, BERT-Mean, BERT-Regularization, and BERT-Augmentation. Subtask B baselines included BERT-CRF, BERT-Span, XLM-roBERTa-CRF, and mLUKE.Source: §4.1.1 · p. 4, §4.1.2 · pp. 4–5, §4.2 · p. 5
4.2 Results of Rhetorical Role Classification
- The independent-sentence BERT baseline scored 0.631 micro F1, while short noisy sentences and training-label bias were identified as dataset concerns.The authors note that the overall sentence-length pattern resembled a Poisson distribution, with significant noise below 20; they suggest validating label bias in future work.Source: §4.2 · p. 5, Figure 3 · p. 5, Figure 4 · p. 6
- 0.828 micro F1 was the best validation score for Legal-BERT-HSLN, using batch size 16, maximum sentence length 32, and learning rate 1 × 10−4.PRE_NOT_RELIED accuracy remained 0, matching the reported state-of-the-art result; the authors suggest investigating this outlier.Source: §4.2 · p. 6, Figure 5 · p. 6, Figure 6 · p. 6
- 0.8343 micro F1 placed Legal-BEST-HSLN No. 5 of 27 teams on the rhetorical-role test set.Test performance exceeded all reported validation performance; the authors suggest a distribution shift among the training, validation, and test sets.Source: §4.2 · p. 6
4.3 Results of Legal Named Entity Recognition
- Up to 14.3% better micro F1 on validation data was achieved by Legal-LUKE than by the NER baselines.Table 2 presents overall validation performance for legal named entity recognition.Source: §4.3 · p. 6, Table 2 · p. 7
- BERT-Span outperformed BERT-CRF, while XLM-roBERTa-CRF achieved 11.4% better F1 than BERT-CRF; all models reached peak performance early.The models shared decoder modules in the encoder comparison, and Figure 7 shows XLM-roBERTa-CRF validation performance across epochs.Source: §4.3 · pp. 6–7, Figure 7 · p. 7
- 0.667 F1 placed Legal-LUKE No. 13 of 17 teams on the NER test set, well below its validation performance.The authors hypothesize that organizer preprocessing produced different token indices and label shifts, potentially because their processing removed empty lines.Source: §4.3 · p. 7
5 Conclusion
The paper uses context-sensitive language models for legal documents, designing Legal-BERT-HSLN for rhetorical-role classification and Legal-LUKE for legal named-entity recognition. The authors report better performance than baselines and a top-five leaderboard result for rhetorical-role classification.
5 Conclusion
- The proposed models outperform baselines and place among the top five teams in rhetorical-role classification.The conclusion reports this overall outcome without giving metric values.Source: §5 · p. 7
- Legal-BERT-HSLN models hierarchical context to address rhetorical-role prediction as sequential sentence classification.Source: §5 · p. 7
- Legal-LUKE applies contextualized, entity-aware representations to legal named-entity recognition.Source: §5 · p. 7