Source-linked AI summary
CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction
Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, Jianfeng Xu
TL;DR
Legal judgment prediction lacked a publicly accessible, high-quality large-scale dataset for inferring judgments from case facts. The paper introduces CAIL2018, a large criminal-case dataset with detailed judgment annotations and evaluates conventional text-classification baselines. Results show that legal judgment prediction remains challenging, despite considerable accuracy on charges and relevant law articles.
Problem
Legal judgment prediction lacked a publicly accessible, high-quality dataset for predicting judgments from case fact descriptions.
Method
The paper constructs CAIL2018 by extracting law articles, charges, and prison terms from Chinese criminal judgment documents and evaluates conventional text-classification baselines.
Results
Experiments achieve considerable accuracy on charge and relevant-law-article prediction, while macro-precision and macro-recall show that LJP remains challenging.
Takeaways & Limitations
CAIL2018 provides a large, publicly available benchmark with detailed annotations for further legal judgment prediction research.
Takeaways & Limitations
The dataset has severe category imbalance: the top 10 charges cover 79.0% of cases, while the bottom 10 cover 0.12%.
Abstract
from arXiv · showhide
In this paper, we introduce the \textbf{C}hinese \textbf{AI} and \textbf{L}aw challenge dataset (CAIL2018), the first large-scale Chinese legal dataset for judgment prediction. \dataset contains more than $2.6$ million criminal cases published by the Supreme People's Court of China, which are several times larger than other datasets in existing works on judgment prediction. Moreover, the annotations of judgment results are more detailed and rich. It consists of applicable law articles, charges, and prison terms, which are expected to be inferred according to the fact descriptions of cases. For comparison, we implement several conventional text classification baselines for judgment prediction and experimental results show that it is still a challenge for current models to predict the judgment results of legal cases, especially on prison terms. To help the researchers make improvements on legal judgment prediction, both \dataset and baselines will be released after the CAIL competition\footnote{http://cail.cipsc.org.cn/}.
1 Introduction
Legal judgment prediction aims to infer case judgments from fact descriptions, but earlier work was constrained by limited and narrow datasets. CAIL2018 addresses the lack of a publicly accessible, high-quality large-scale dataset with richer judgment annotations.
- Legal judgment prediction aims to predict legal-case judgments from their fact descriptions.
- Earlier studies often analyzed judgments statistically on small case collections rather than directly predicting them.
- Text-classification approaches extracted features from case descriptions but remained restricted to particular case types and faced generalization issues.
- CAIL2018 was introduced because no publicly accessible, high-quality dataset for legal judgment prediction was available.
- CAIL2018 contains more than 2.6 million criminal cases published by the Supreme People’s Court of China.
- Each case includes fact descriptions and judgment results annotated with applicable law articles, charges, and prison terms.
2 Dataset Construction
CAIL2018 is constructed by filtering Chinese criminal judgment documents and extracting structured judgment labels from their text. The resulting dataset contains 2,676,075 cases but remains highly imbalanced across charges and law articles.
- The dataset is constructed from 5,730,302 criminal documents collected from China Judgments Online, retaining judgment documents for LJP training.
- Fact descriptions serve as model inputs, while applicable law articles, charges, and prison terms are extracted from referee results using regular expressions.
- Cases with multiple defendants are removed to reduce LJP difficulty.
- Low-frequency charges and law articles occurring fewer than 30 times are filtered out, along with nonspecific top law articles and charges.
- 2,676,075 criminal cases remain after preprocessing, with 183 criminal law articles and 202 charges.
- The top 10 charges cover 79.0% of cases, whereas the bottom 10 cover only 0.12%, making low-frequency labels difficult to predict.
3 Experiments
The study evaluates conventional text-classification baselines for predicting law articles, charges, and prison terms. Accuracy is considerable for charges and law articles, but macro-precision and macro-recall indicate that LJP remains challenging under data scarcity and imbalance.
- The experiments evaluate baselines on three subtasks: predicting law articles, charges, and prison terms.
- Baselines: TFIDF+SVM combines TFIDF word features with a linear-kernel Support Vector Machine classifier.
- Baselines: FastText performs text classification using N-grams and hierarchical softmax.
- Baselines: CNN encodes fact descriptions with multiple convolutional filters for text classification.
- Implementation: All methods use randomly selected training and test cases, Chinese word segmentation, and 200-dimensional word embeddings for neural models.
- Evaluation: The models are evaluated with accuracy, macro-precision, and macro-recall on the test set.
- Results: Considerable accuracy is achieved for charge and relevant-law-article prediction, while macro-precision and macro-recall show that LJP remains a major challenge.The paper attributes this challenge to insufficient training data and class imbalance.
4 Conclusion
The paper releases CAIL2018 as the largest publicly available legal judgment prediction dataset, with detailed annotations aligned with real-world scenarios. Experiments show that legal judgment prediction remains challenging and offers substantial room for improvement.
- CAIL2018 is presented as the largest publicly available legal judgment prediction dataset, with detailed annotations consistent with real-world scenarios.
- Experimental results demonstrate that legal judgment prediction remains challenging and leaves room for further improvement.