Source-linked AI summary
Identification of Rhetorical Roles of Sentences in Indian Legal Judgments
Paheli Bhattacharya, Shounak Paul, Kripabandhu Ghosh, Saptarshi Ghosh, Adam Wyner
TL;DR
Rhetorical roles in legal judgments matter for downstream tasks, but interleaving themes and subjective labels make them challenging to identify. The paper studies annotations on Supreme Court of India judgments and evaluates neural models without handcrafted features. Those models outperform handcrafted-feature baselines, while predictions also reflect human difficulty distinguishing some label pairs.
Problem
Legal judgments are poorly structured, and rhetorical roles can be subjective, making identification challenging despite its relevance to tasks such as summarization and legal search.
Source: §1 · p. 1Method
The paper analyzes human annotations of legal judgments and tests Hierarchical BiLSTM and Hierarchical BiLSTM-CRF models for rhetorical-role classification.
Source: §1 · p. 2Results
The Hierarchical BiLSTM-CRF achieves a Macro F-score in the range [0.8 −0.9] and neural models outperform handcrafted-feature baselines.
Source: §1 · p. 2, §6.2 · p. 8Takeaways & Limitations
Neural models identify rhetorical roles better than handcrafted-feature methods in the evaluated legal documents, while model predictions reflect human confusion around subjective label pairs.
Source: §7 · p. 9, §6.3 · p. 9Neural models avoid costly feature hand-crafting but make it difficult to understand why a sentence receives a particular role.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Automatically understanding the rhetorical roles of sentences in a legal case judgement is an important problem to solve, since it can help in several downstream tasks like summarization of legal judgments, legal search, and so on. The task is challenging since legal case documents are usually not well-structured, and these rhetorical roles may be subjective (as evident from variation of opinions between legal experts). In this paper, we address this task for judgments from the Supreme Court of India. We label sentences in 50 documents using multiple human annotators, and perform an extensive analysis of the human-assigned labels. We also attempt automatic identification of the rhetorical roles of sentences. While prior approaches towards this task used Conditional Random Fields over manually handcrafted features, we explore the use of deep neural models which do not require hand-crafting of features. Experiments show that neural models perform much better in this task than baseline methods which use handcrafted features.
1. Introduction
Legal judgments interleave themes, making rhetorical roles difficult for both automated systems and human experts to distinguish. This paper combines an annotation study with neural models that outperform handcrafted-feature baselines.
1. Introduction
- Legal documents are poorly structured, and themes such as a decision’s rationale, precedents, and statutes can interleave, complicating role identification.The labels can also be subjective, with legal experts finding some pairs difficult to distinguish.Source: §1 · pp. 1–2
- Prior automation relied on handcrafted features, some dependent on costly legal expertise or specific to a few domains, with cross-domain feature coverage unexplored.The paper explores neural models that learn features without hand-engineering.Source: §1 · p. 2
- Neural models outperform handcrafted-feature baselines when classifying seven rhetorical labels across documents from five legal domains.The Hierarchical BiLSTM-CRF achieves a Macro F-score in the range [0.8 −0.9].Source: §1 · p. 2
- The paper brings together an extensive annotation study and deep-learning models for automating rhetorical-role identification in legal documents.Its annotation analysis finds that even legal experts have difficulty distinguishing some label pairs, indicating inherent subjectivity.Source: §1 · p. 2
2. Related Work
Prior work covered annotation, automation, and legal-domain deep learning, but rhetorical-role automation had not yet applied deep-learning methods. This paper addresses that gap without handcrafted features.
2. Related Work
- Related annotation research includes the 504-sentence TEMIS corpus and work on gold-standard curation and annotator agreement.Other studies also analyze annotation while automating semantic labeling.Source: §2 · p. 2
- Earlier work includes CRF-based role identification, court-document segmentation, factual versus non-factual classification, and comparisons of rule-based and machine-learning approaches.The cited studies include Saravanan et al., segmentation work, factuality classification, and Walker et al..Source: §2 · p. 3
- Prior automated rhetorical-role identification in the legal domain almost always used handcrafted features, whereas this paper applies deep learning without them.Deep learning had been used for other legal tasks, but not, to the authors’ knowledge, for this task.Source: §2 · p. 3
3. Dataset
The experiments use 50 Supreme Court of India judgments sampled proportionally from the five most frequent legal domains in a larger crawl.
3. Dataset
- The dataset contains 50 Supreme Court of India judgments from five domains, sampled proportionally to domain frequency from a crawl of 53,210 documents.The experiments are all performed on these 50 documents.Source: §3 · p. 3
- The sample includes 16 Criminal, 10 Land and property, 9 Constitutional, 8 Labour and Industrial, and 7 Intellectual Property Rights documents.Source: §3 · p. 3
4. Annotation Details
The annotation section sets out seven rhetorical roles, an iterative process involving three senior law students, and an analysis of inter-annotator agreement.
4. Annotation Details
- The annotation study covers the selected rhetorical roles, annotation procedure, and inter-annotator agreement analysis.Source: §4 · p. 3
4.2. Annotation Process
- Three senior law students annotated using a manual developed with them, independently labeling initial documents before joint discussion and iterative refinement.The process was repeated iteratively across the 50 documents.Source: §4.1 · p. 4, §4.2 · p. 4
4.1. Annotation Labels / Rhetorical Roles
- Seven roles cover case facts, lower-court rulings, arguments, statutes, precedents, the decision’s ratio, and the present court’s ruling.Facts describe the case chronology; the present-court ruling is its ultimate decision or conclusion.Source: §4.1 · p. 4
4.3. Analysis of Inter-Annotator Agreement and Curation of Gold Standard
- The study computes inter-annotator agreement as part of analyzing agreement and curating a gold standard.Source: §4.3 · p. 4
IAA measure
The study averages pairwise precision, recall, and F-scores across annotator pairs. GATE distinguishes exact matches, partial span matches, and label disagreements when calculating agreement.
IAA measure
- Pairwise IAA is computed for each annotator pair and then averaged across the three pairs.The pairs are (A1, A2), (A2, A3), and (A1, A3).Source: IAA measure · p. 4
- GATE counts identical spans with identical labels as correct, same-label spans that differ as partial, and different labels as missing or spurious.Source: IAA measure · pp. 4–5
- Table 1 presents the three annotators’ average inter-annotator agreement as F-scores measured by GATE.Source: Table 1 · p. 5
- GATE reports strict, lenient, and average measures by treating partial matches as incorrect, correct, or midway, respectively.The F-score uses β=1, weighting precision and recall equally.Source: IAA measure · p. 5
Analysis of F-scores
The reported IAA varies by label and scoring strictness, while sentence-level comparisons reveal disagreement around particular label pairs. Agreement is reported as uniform across the five legal domains.
Analysis of F-scores
- Strict scores are lower partly because annotators sometimes marked sentence-final punctuation differently, which lenient scoring disregards.The authors describe cases where one annotator included the full stop and another omitted it.Source: Analysis of F-scores · p. 5
- Lenient IAA F-scores exceed 0.85 for every label, while strict scores are higher for Statute, Ruling by Lower Court, and Facts than for Precedent and Ruling by Present Court.Source: Analysis of F-scores · p. 5
- Sentence-level agreement is generally high, but disagreements cluster around Ratio with FAC, PRE, RLC, and RPC, and between ARG and FAC.The paper relates these ambiguities to connections between reasons for judgment, facts, prior judgments, lower-court rulings, and arguments.Source: Analysis of F-scores · p. 6
- Inter-annotator agreement is reported as uniform across the five legal domains.Source: Analysis of F-scores · p. 6
- The gold-standard label for each sentence is assigned by majority vote among the three annotators.Source: Analysis of F-scores · p. 6
5. Methods for automatically identifying rhetorical roles
The task is treated as sentence-level sequence labeling, with handcrafted-feature CRFs as baselines and hierarchical BiLSTMs as neural alternatives. The neural setup also adds a CRF layer to model label dependencies.
5. Methods for automatically identifying rhetorical roles
- The task is seven-class sequence labeling over 9,380 SpaCy-split sentences from 50 documents; sentence-splitting accuracy was close to 90%.Source: §5 · p. 6
- The handcrafted-feature baselines use CRFs to model sentence-label sequences with emission and transition scores.The three variants use features from, from, or their combination; features include POS tags, layout, cue phrases, and named entities.Source: §5 · pp. 6–7
- The hierarchical BiLSTM extracts sentence features from sentence embeddings initialized either randomly or with sent2vec embeddings from 53K court documents.Source: §5 · p. 7
- A CRF layer is added above the hierarchical BiLSTM because its probability scores do not account for label dependencies.The CRF receives feature vectors from the top-level BiLSTM; both random and pre-trained sentence embeddings are tested.Source: §5 · p. 7
6. Results and Analysis
Neural models outperform handcrafted-feature baselines, and the analysis examines embedding and CRF choices, performance across labels and legal domains, and agreement with human annotators.
6.1. Experimental setup and evaluation metrics
- Five-fold cross-validation uses 40 documents for training and 10 for testing per fold; performance is averaged across folds and measured with macro-averaged Precision, Recall, and F-score.Predictions are counted as correct when they match the annotators’ majority label; macro-averaging computes each metric per class before averaging.Source: §6.1 · p. 7
6.2. Comparing performances of different models
- Neural models substantially outperform handcrafted-feature baselines, indicating better learned latent features.Table 3 reports macro Precision, Recall, and F-score for the baseline and neural methods.Source: Table 3 · p. 8, §6.2 · p. 8
- Pretrained legal-corpus embeddings substantially improve both neural models, while CRF adds little over pretrained Hier-BiLSTM given long documents and few training sequences.Documents average 200 sentences, limiting the CRF’s ability to learn transition scores.Source: §6.2 · p. 8
6.3. Detailed analysis of the best performing model (Hier-BiLSTM-CRF)
- Hier-BiLSTM-CRF predicts Ratio and Ruling by Present Court well, but performs less well on Arguments (F-score of 0.5924), whose sentences are interleaved and relatively rare.Ratio accounts for 38.63% of corpus sentences, while Ruling by Present Court accounts for 2.79% and tends to occur near document ends.Source: §6.3 · pp. 8–9, Table 4 · p. 8
- Across legal domains, F-scores are 0.82–0.86 except in Labour & Industrial law, where lower performance aligns with lower inter-annotator agreement.Table 4 reports per-label and per-domain F-scores, with averages in its final row and column.Source: Table 4 · p. 8, §6.3 · p. 9
- Model–human disagreements cluster on Arguments/Facts and Ratio/Fact/Precedence pairs, which also have low human inter-annotator agreement and suggest subjectivity in these roles.Table 5 is an agreement matrix comparing model labels with the annotators’ majority labels.Source: §6.3 · p. 9, Table 5 · p. 9
7. Conclusion and Future Work
The paper reports improved rhetorical-role identification with deep learning alongside an extensive annotation and agreement analysis. It also notes a transparency trade-off and plans to test generalization across jurisdictions.
7. Conclusion and Future Work
- Deep learning models identify legal-document rhetorical roles better than handcrafted-feature methods, alongside analysis of human and model–annotator agreement.The paper presents the annotation study and automation models together as its contribution.Source: §7 · p. 9
- Neural models avoid hand-crafted features and expensive legal expertise, but trade explainability and transparency for that advantage.The paper frames their use as appropriate when good performance is assumed to matter more than transparency.Source: §7 · pp. 9–10
- Future work will test whether deep learning models generalize across jurisdictions using legal documents from other countries.Source: §7 · p. 10