Source-linked AI summary
ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation
Vijit Malik, Rishabh Sanjay, Shubham Kumar Nigam, Kripa Ghosh, Shouvik Kumar Guha, Arnab Bhattacharya, Ashutosh Modi
TL;DR
The paper addresses the need for explainable systems that assist judges with court-outcome prediction. It introduces the ILDC corpus and CJPE task, evaluates prediction and explanation models, and reports a 78% F1 best prediction result versus 94% average annotator accuracy. The analysis also finds differences between algorithmic and expert explanations.
Problem
Legal judgment prediction must handle long, noisy, domain-specific documents and provide explanations that make automated suggestions understandable in the legal process.
Method
The paper introduces ILDC, defines CJPE prediction and explanation tasks, and evaluates models using decisions and expert explanations from a separate test set.
Results
78% F1 was achieved by the best case-decision model, compared with 94% average annotator accuracy.
Takeaways & Limitations
The corpus and task expose computational challenges in legal documents and differences between machine-generated and expert explanations.
Takeaways & Limitations
Explanation annotations cover only a small portion of the test set, and legal proceedings can contain intricacies that confuse even legal experts.
Abstract
from arXiv · showhide
An automated system that could assist a judge in predicting the outcome of a case would help expedite the judicial process. For such a system to be practically useful, predictions by the system should be explainable. To promote research in developing such a system, we introduce ILDC (Indian Legal Documents Corpus). ILDC is a large corpus of 35k Indian Supreme Court cases annotated with original court decisions. A portion of the corpus (a separate test set) is annotated with gold standard explanations by legal experts. Based on ILDC, we propose the task of Court Judgment Prediction and Explanation (CJPE). The task requires an automated system to predict an explainable outcome of a case. We experiment with a battery of baseline models for case predictions and propose a hierarchical occlusion based model for explainability. Our best prediction model has an accuracy of 78% versus 94% for human legal experts, pointing towards the complexity of the prediction task. The analysis of explanations by the proposed algorithm reveals a significant difference in the point of view of the algorithm and legal experts for explaining the judgments, pointing towards scope for future research.
1 Introduction
The paper introduces ILDC and the CJPE task to support explainable court judgment prediction, addressing the complexity of long, noisy legal documents and difficult reasoning.
- India’s legal-case backlog motivates systems that assist judges by suggesting case outcomes to expedite the judicial process.
- Automated legal decision systems must explain both the predicted outcome and how the decision was reached.
- ILDC contains Indian Supreme Court proceedings annotated with original decisions, while a separate test set includes expert judgment explanations.
- CJPE requires predicting whether an appeal is allowed or dismissed and identifying case sentences or phrases that justify the decision.
- The corpus and task address unstructured, verbose, noisy legal documents, domain-specific language, and the challenge of extracting facts and arguments for explanation.
- The authors position ILDC as a resource for augmenting judges rather than replacing them, while noting that unforeseen social implications require further research.
2 Related Work
ILDC differs from earlier legal judgment-prediction resources by combining unstructured Indian Supreme Court documents with explanation evaluation and broad case coverage.
- Unlike typical LJP datasets, CJPE evaluates explanations using a separately annotated gold-explanation test set.
- CJPE uses unannotated, unstructured case documents rather than explicitly supplied facts, making the setting more realistic and challenging.
- The task considers rulings, arguments, statutes, precedents, and decision rationales because bare facts do not fully form a judgment premise.
- ILDC is described as the largest Indian legal corpus, containing 34,816 documents from generic Supreme Court cases rather than one case class.
- Existing resources include Chinese criminal-case data, European human-rights cases, and other datasets focused mainly on legal judgment prediction.
3 Indian Legal Document Corpus
ILDC is an English-language corpus of Indian Supreme Court proceedings labeled with original decisions, with expert explanation annotations for a separate test set. Its construction addresses noisy, heterogeneous documents while limiting the corpus to Supreme Court cases and providing randomly divided, decision-balanced evaluation sets.
- Corpus scope: ILDC contains Supreme Court of India case proceedings labeled with the judges’ original decisions, while a separate test set also includes expert explanations.The documents concern decisions between appellants or petitioners and respondents, based on case facts, arguments, statutes, precedents, and lower-court rulings.
- Corpus scope: The corpus was built from publicly available Supreme Court proceedings covering 1947 through April 2020.
- Corpus construction: Preprocessing removes noisy text and metadata, deletes decision-ending sections as prediction targets, and extracts the actual decision labels from those sections.The proceedings contain spelling mistakes, varied formats and sizes, and decisions typically appear near the document’s end.
- Corpus construction: ILDC separates single-petition and multi-petition cases, assigning a multi-petition document the accepted label if at least one appeal was accepted.Validation and test sets are shared by the ILDCmulti and ILDCsingle versions.
- Evaluation design: The corpus is randomly divided into train, validation, and test sets, with validation and test balanced by decision rather than split temporally.
- Annotation and safeguards: Expert explanations are available for only a small portion of the test set because annotation is specialized, time-consuming, and laborious.The authors plan to expand explanation and lower-court annotations, while judge and party names were anonymized to mitigate bias.
4 Annotation Analysis
Annotation analysis shows high agreement on judgment predictions but substantial variation in which textual reasons experts select as explanations. The differences reflect subjectivity, linguistic interpretation, and distinct preferences for substantive, procedural, factual, or minimal reasoning.
- Case judgment accuracy: Annotators’ judgment agreements are high but not perfect, suggesting subjectivity in legal decision-making and differences in linguistic interpretation.The analysis compares annotator judgments with original Supreme Court decisions and examines disagreement through detailed case studies.
- Judgment agreement: 94.6% is the highest pairwise agreement between Experts 1–3 and 3–5, while Fleiss’ kappa is 0.820 across all five annotators.
- Explanation agreement: Explanation agreement is evaluated with ROUGE, BLEU, METEOR, Jaccard similarity, and overlap metrics because textual annotation lacks standard agreement measures.ROUGE-L is shown in Figure 1, while the authors report similar trends for the other metrics.
- Explanation patterns: Experts 1, 3, and 4 often use holistic reasoning covering substantive and procedural aspects, whereas Experts 2 and 5 often use bare-minimum reasoning.Differences also concern whether factual sentences are considered.
- Task complexity: Legal cases can confuse even experts because long, verbose proceedings contain intricate linguistic references that affect how the current appeal is interpreted.The paper illustrates this with a case where a sentence distinguishes acceptance of a previous appeal from rejection of the present appeal.
5 CJPE Task
The CJPE task combines court-decision prediction with explanations identifying sentences that justify the predicted outcome. Experiments compare models for long, specialized legal documents and evaluate hierarchical occlusion explanations against expert annotations.
- Task definition: CJPE predicts whether an appeal is accepted or dismissed and identifies case-description sentences supporting that decision.The task has separate prediction and explanation subtasks, with explanations produced from the case proceeding and predicted decision.
- Task definition: Long documents, specialized vocabulary, noisy formatting, and legal reasoning make Indian Supreme Court case prediction computationally challenging.The authors report that standard feature-based and pretrained approaches were no better than a random classifier in their setting.
- Case decision prediction: 64% F1 was the best result among sequential models, achieved by GloVe embeddings with BiGRU and attention.Sequential models used document text at word, sentence, or chunk levels.
- Case decision prediction: 78% F1 was achieved by XLNet with BiGRU, while hierarchical models generally outperformed transformer models.The best model used a hierarchical transformer architecture, with the last 512 tokens giving the best transformer performance.
- Case decision explanation: The hierarchical occlusion method masks chunks and then sentences, using output changes to score their contribution and selecting top-k sentences.Chunk scores compare masked and unmasked probabilities; sentence scores compare normalized changes in transformer logits, with approximately 40% selected per chunk.
- Model explainability versus annotators: 0.8337 was the highest Overlap-Min value with Expert 4, whereas the highest ROUGE-L value was 0.4445 with Expert 4.Other explanation metrics ranged from low to medium overlap, indicating disagreement between machine and legal-expert explanations.
6 Conclusion
The paper contributes ILDC and the CJPE task as resources for explainable legal judgment prediction. Corpus analysis and modeling results show that legal documents create substantial computational challenges.
- Conclusion: ILDC contains case decisions and separate-test-set explanations, providing a resource for Legal NLP research on explainable judgment prediction.The corpus is derived from Supreme Court of India proceedings and includes explanation annotations for a separate test set.
- Conclusion: Corpus analysis and modeling results demonstrate the computational complexity of legal documents.The conclusion presents this complexity as motivation for continued research in the area.
Ethical Concerns
The system is intended to support legal professionals rather than replace them, with design choices aimed at limiting reliance on personal identities and biases. Its scope is restricted to Supreme Court of India cases.
- Data governance: The source proceedings were scraped from Indian Kanoon, which permits scraping, and annotators participated voluntarily.The paper states that no copyrights were infringed during data collection.
- Use and oversight: The system is designed to provide information for legal research and decision-making, not to replace legal professionals.The paper frames this purpose as avoiding decisions about human legal rights and obligations by non-human intelligence.
- Bias considerations: Cases were selected randomly and identities were anonymized to minimize or eliminate possible prediction bias.The selection avoided focusing on law type, parties’ backgrounds, or judges’ identities.
- Scope: The corpus uses only Supreme Court of India cases because these decisions emphasize legal issues and bind subordinate courts.This forum choice is also presented as supporting broader application to other cases.
A Annotations and Case studies: Agreement in Judgment Prediction for Annotators
Expert annotations sometimes diverged from original court decisions, with disagreements linked to multiple issues, legal misinterpretation, linguistic ambiguity, or confusion between appeals and review petitions.
- Agreement in Judgment Prediction for Annotators: Experts disagreed with the original decision in cases involving multiple issues and separate appellant arguments.The passage attributes one discrepancy to the case’s structure and several issues under consideration.
- Agreement in Judgment Prediction for Annotators: Experts sometimes diverged after misconstruing legal positions or relying on a precedent the Supreme Court considered irrelevant.This explanation specifically accounts for Expert 2’s disagreement.
- Agreement in Judgment Prediction for Annotators: Experts 2 and 5 accepted an appeal that the court had rejected because Sentence 17 obscured the distinction between an appeal and a review petition.The court was affirming an earlier appeal acceptance while rejecting the later review application.
B Models Details
The experiments used the hyperparameter settings summarized in Table 6 and ran on Google Colab with a single Tesla P100-PCIE-16GB GPU.
- Models Details: Table 6 summarizes hyperparameter settings for all models.The passage provides no individual parameter values.
- Models Details: All experiments were run on Google Colab.
- Models Details: The experiments used Colab’s default single Tesla P100-PCIE-16GB GPU.
B.1 Case Prediction Model Details
The study compares classical, sequential, hierarchical, and transformer-based models for case prediction, with performance shaped by document length, input location, and model architecture.
- Classical Models: Classical models used Sent2Vec sentence embeddings, Doc2Vec document embeddings, and extractive summarization for long legal proceedings.The embeddings were trained on ILDCmulti because the data are domain-specific.
- Classical Models: TF-IDF features performed no better than a random classifier, which the authors attribute to document length and weak long-term dependency capture.
- Classical Models: Doc2Vec+LR was competitive with sequential models, while a simple word-embedding model performed similarly to HAN.
- Sequential Models: 64% F1 was achieved by GloVe embeddings with BiGRU and attention, outperforming catchphrase-selected sentences at 61% F1.
- Transformer Models: Transformer experiments used BERT, DistilBERT, RoBERTa, and XLNet, while Longformer and Reformer were excluded because of high compute requirements.
- Transformer Models: The best transformer input used the last 512 tokens with BERT Base, and later document sections generally improved prediction performance.
- Transformer Models: RoBERTa and XLNet outperformed BERT, while BERT outperformed DistilBERT in the reported transformer comparisons.
- Hierarchical Models: Hierarchical models generally outperformed transformer models, but adding attention to BiGRU did not significantly improve performance; CNN provided competitive performance.
B.2 Explanability Models and Results Details
The explainability model uses hierarchical occlusion to identify influential chunks and sentences, then compares occlusion-based importance with attention-based visualizations across test documents.
- Occlusion-based Explanations: Chunk explainability is computed by masking each chunk embedding and comparing the masked probability with the original model probability.Positive-scoring chunks are passed to the transformer stage for sentence-level explanation extraction.
- Occlusion-based Explanations: Sentence explanation scores compare masked and original transformer logits, normalized by sentence length, and select the top ∼40% of sentences per chunk.
- Visualization Results: Attention weights concentrate on the last chunks, whereas occlusion scores often also rank the last chunk highly without being optimized for that behavior.
- Visualization Results: The visualizations support the hypothesis that relevant syntactic and semantic information lies toward the end of case documents.