A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges

1. Introduction

• Automated LJP is motivated by the workload and specialized demands on legal practitioners; recent neural and pretrained models have advanced the task, which accounts for around 65% of legal-domain papers at major NLP conferences.In Louisiana, each attorney handles up to 50 cases per day, leaving 1–5 minutes per case for preparation.Source: §1 · p. 1
• Most researchers focus on a few popular LJP datasets, and existing task classifications are not thorough or applicable across legal systems and domains.The survey presents 31 publicly available datasets in six languages and popular state-of-the-art models.Source: §1 · p. 2
• The proposed dataset taxonomy classifies LJP along three attributes: task type, legal system, and law domain.It is based on the outcomes of LJP tasks.Source: §1 · p. 2
• The survey covers 31 LJP datasets in six languages, six pre-training datasets in three languages, and 14 evaluation metrics grouped into four categories.It also discusses metric usage across existing LJP tasks.Source: §1 · p. 2
• The survey reviews 12 legal-domain pretrained models in three languages, three research directions, and state-of-the-art results for eight representative datasets.The directions include multi-task, interpretable, and few-shot learning.Source: §1 · p. 3
• The survey identifies inadequate facts, complex legal reasoning, and interpretability as major challenges for existing LJP datasets and methods.The paper notes that current models capture dataset statistics but still struggle to reason over complex evidence and logical dependencies.Source: §1 · p. 3

2.1. Background

• Litigation proceeds through claim collection, trial debate, and sentencing, with common- and civil-law systems differing in prosecution, evidence, fact-finding, and use of precedent or legislation.The taxonomy distinguishes common-law fact verdicts and sentences from civil-law justification of judgments and judgment results.Source: §2.1 · p. 4, §2.2 · p. 5

2.2. Taxonomy

• The proposed taxonomy uses task type, legal system, and law domain because the familiar article-charge-penalty classification misses tasks such as pro-or-con decisions.The paper analyzes 31 datasets using these dimensions.Source: §2.2 · p. 4
• Tasks are divided into main outcomes that determine judgments and auxiliary tasks whose outcomes can latently improve judgment prediction, across criminal, civil, and administrative domains.Examples of auxiliary tasks include fact-snippet extraction and attributes of confusing charges.Source: §2.2 · pp. 4–5

2.2.1. Types of LJP Tasks

• Article recommendation is among the hottest LJP tasks, with datasets covering binary or multi-label violations, optimal article sets, and fact-to-article correspondence.The paper groups article recommendation by many-vs-one, many-vs-many, and one-vs-many relationships.Source: §2.2.1 · pp. 5–6
• Charge prediction includes case-charge correspondence, common-versus-few-shot evaluation, and logical knowledge such as dependencies among articles, charges, and prison terms.Some datasets restrict cases to one defendant or set minimum charge frequencies.Source: §2.2.1 · pp. 6–7
• Prison-term prediction datasets use relevant charges to predict terms, while the paper notes that the available datasets are inadequate.CPTP and TOPJUDGE datasets are cited as examples.Source: §2.2.1 · p. 8
• Court-view generation is scarce because it requires complex reasoning over facts and law; examples condition generation on charge labels or pleas-aware fact representations.Court-View-Gen targets criminal cases, while AC-NLG targets civil cases.Source: §2.2.1 · p. 8
• Plea-judgment tasks predict rulings on plaintiffs’ pleas from judge-summarized facts, courtroom debate facts, or case materials with plea judgments masked.Examples include Auto-Judge, ILDC, and LJP-MSJudge.Source: §2.2.1 · pp. 8–9

3. Datasets

• Dataset construction uses automated extraction of metadata, input-output samples, or facts, alongside legal-expert annotation of rationale sentences or legal knowledge.The expert-annotated knowledge may be absent from the judgment documents.Source: §3 · p. 9
• The paper identifies dataset availability and potentiality as a significant challenge for LJP..Source: §3 · p. 9
• Public LJP datasets are categorized as task-specific, containing one subtask, or multi-task, containing multiple subtasks..Source: §3 · p. 10

3.1. Single-Task Datasets

• Task-specific datasets are grouped according to the outcome their single LJP task predicts..Source: §3.1 · p. 10

3.1.1. Article Prediction Datasets

• Article-prediction datasets include violation detection and identification, with coverage tending to be less related to civil and administrative cases.Examples range from 584 ECHR cases in echr to 11,478 cases in ECHR-CASES; MLMN annotates fact-article correspondences in 1,189 Chinese criminal judgments.Source: §3.1.1 · p. 10

3.1.2. Charge Prediction Datasets

• Charge-prediction datasets largely focus on Chinese cases; FLA has 60,000 cases and retains single-defendant cases, labeling charges appearing over 80 times as positive.CAIL2018 contains 2,676,075 criminal cases, while MAMD includes multi-defendant cases.Source: §3.1.2 · p. 10

3.1.3. Prison Term Prediction Datasets

• Prison-term prediction datasets are mainly Chinese and rarely English; CPTP contains 238,749 criminal cases, 1,240 prison-term values, and 157 charge types.MLMN also divides judgment results into five penalty classes.Source: §3.1.3 · p. 11

3.1.4. Plea Judgment prediction Datasets

• Plea datasets mainly focus on SPC and Swiss court cases, alongside French, Indian, UK, and US examples; SwissJudgmentPrediction contains 85K multilingual cases.Auto-Judge has 100,000 divorce cases, while LJP-MSJudge contains 70,482 private-lending cases and courtroom debates.Source: §3.1.4 · pp. 11–12

3.1.5. Court view Generation Datasets

• Court-view generation datasets mainly focus on Chinese rather than English; Court-View-Gen contains 171,981 single-defendant, single-charge cases for generating views from charge labels.The task generates judges’ rationales in court views.Source: §3.1.5 · p. 12

3.2. Multi-Task Datasets

• Multi-task datasets contain multiple detailed subtasks intended to support improvements in legal judgment prediction..Source: §3.2 · p. 12
• QAjudge-CJO, QAjudge-PKU, and QAjudge-CAIL pair facts with applicable articles, charges, and penalty terms, filtering out several complex or infrequent cases.The datasets exclude multiple defendants, multiple charges, infrequent charges, and articles appearing fewer than 100 times.Source: §3.2 · p. 12
• CAIL2018 contains 2,676,075 criminal cases, while CAIL-Long contains 1,129,053 criminal and 1,099,605 civil cases with different annotations by case type.CAIL-Long criminal cases include charges, laws, and penalty terms; civil cases include causes of action and laws.Source: §3.2 · p. 12
• AC-NLG annotates 66,904 civil cases with plaintiff claims, facts, court views, judgments, and rationales for joint judgment-and-rationale generation.The paper links joint generation of judgments and rationales with improved interpretability.Source: §3.2 · p. 12

3.3. Datasets for Pre-trained Language Model

• Public sources for unlabelled legal pretraining corpora include CJO, HUDOC ECHR, CourtListener, and Légifrance; Case Law Access Project access is by request.Source: §3.3 · p. 12
• Legal-domain PLM datasets have been scraped from public legal resources to study adaptation of pretrained language models to legal tasks.Source: §3.3 · p. 12
• Legal pretraining resources include English corpora from legislation, cases, and contracts, Chinese case corpora, and raw French legal text.Examples include 12GB of English legal text, 84GB of Chinese legal cases, and 6.3GB of French text.Source: §3.3 · pp. 12–13
• Existing legal PLM datasets tend to focus on a single-language corpus and give less consideration to multilingual corpora.Source: §3.3 · p. 13

4. Evaluation Metrics

• LJP outputs call for four metric categories: classification; classification with regression; classification with generation; and text generation.These categories respectively cover law articles, charges, plea judgments and element answers; prison terms; fact snippets; and generated court views.Source: §4 · p. 13

4.1. Evaluation of Text Classification

• Classification metrics use category-level true and false positives and negatives to derive precision, recall, and related measures.Macro measures average across categories, while micro measures aggregate across law cases; Table 5 lists metrics used in dataset papers.Source: §4.1 · pp. 13–14, Table 5 · p. 14

4.2. Evaluation of Text Generation

• Generation metrics assess candidate text against reference text using measures including Jaccard similarity, BLEU, ROUGE, Meteor, and BERT SCORE.Jaccard compares set intersection with union; BLEU measures modified n-gram precision with a brevity penalty, while ROUGE is recall-based and measures n-gram co-occurrence.Source: §4.2 · pp. 15–16

4.3. Evaluation of Text Regression

• Prison-term prediction over a continuous interval is treated as regression and evaluated with Log distance and S score.The metrics are applied to the difference between predicted and ground-truth prison terms.Source: §4.3 · p. 16

5. Methods

• Method choice is framed by dataset structure: multi-task learning uses dependencies among subtasks, pretrained models fine-tune task-specific datasets, and few-shot learning addresses imbalanced judgment results.Source: §5 · p. 17

5.1. Multi-Task Learning

• Multi-task methods use relationships among LJP tasks, including joint article and charge prediction, topological article-charge-penalty dependencies, and forward prediction with backward verification.Other approaches distinguish confusing charges using attributes or confusing law articles.Source: §5.1 · p. 17

5.2. Pre-trained Language Model

• Legal PLMs include English LEGAL-BERT variants, CaseLaw-BERT and LegalDB, Chinese Lawformer, and French JuriBERT; long-document approaches include HIER-BERT and Lawformer.Lawformer combines local sliding-window and global task-motivated attention to capture long-distance dependencies in Chinese legal documents with thousands of tokens.Source: §5.2 · pp. 17–18

5.3. Interpretable-Perspective Learning Framework

• Interpretability is divided into introspection, explaining how a model determines its output, and justification, producing sentences about how evidence is compatible with that output.Examples include reinforcement-learning rationales or question selection and answers, as well as fact-article correspondence and charge-based prison-term prediction.Source: §5.3 · p. 18

5.4. Few-Shot Learning Framework

• Few-shot methods address the focus of existing work on high-frequency outcomes, using charge attributes, focal-loss capsule modeling, or a counterfactual decoder for judgment imbalance.Source: §5.4 · p. 18

6.1. ECHR cases

• LEGAL-BERT-FP 100k ALL LEGAL widely outperforms other methods in ECHR-CASES binary violation classification, with LEGAL-BERT-FP 500k SUB-DOMAIN also performing excellently.The authors suggest further pre-training as a way to port BERT to a new domain.Source: §6.1 · p. 19

6.2.1. CAIL-Long Dataset

• Lawformer achieves the best performance on CAIL-Long criminal and civil cases, supporting its use for Chinese legal documents longer than 512 tokens.Lawformer is Longformer-based and uses legal-domain pretraining to process long documents.Source: §6.2.1 · pp. 19–20

6.2.2. CAIL-2018 Dataset

• On CAIL-2018, LADAN leads in accuracy, macro-precision, and macro-recall; MPBFN-WCA exceeds TOPJUDGE, while both perform poorly on MP, MR, and F1.The authors associate the latter weakness with predicting few-shot charges; Few-Shot charge prediction is close to LADAN, but prison-term performance is worse with predefined attributes.Source: §6.2.2 · p. 20

6.3. FSC, UKC, SCOTUS, and SCI cases

• Across FSC, UKC, SCOTUS, and SCI, XLNet with BiGRU leads on ILDC, TFIDF with LR leads among UKC classical models, and classical models perform excellently on FSC.Table 9 presents the current results for these court cases.Source: §6.3 · pp. 20–21, Table 9 · p. 20
• On ILDC, machine-extracted fact snippets are compared with expert annotations, pointing to efficient explainable models as a future LJP direction.The comparison is reported as exact matching results.Source: Table 10 · p. 20, §6.3 · p. 21

6.4. FSCS cases

• On FSCS, Majority performs best among baseline and BERT-based models in MiF; long BERT and hierarchical BERT outperform Italian results in MaF, considering the error margin.The authors identify class imbalance, dataset scale, and BERT variants as possible influences on performance.Source: §6.4 · p. 21, Table 11 · p. 21

6.5. TSC cases

• On TSCC, BiGRU with attention outperforms the non-neural SVM and Naive Bayes baselines.Source: §6.5 · p. 21, Table 12 · p. 21

6. Results & Observations

• Across ECHR-CASES, CAIL-Long, ILDC, and FSCS, further legal-corpus pretraining could improve judgment prediction over classical or generic-corpus models; CAIL-2018 results favor combining two MTL families.ILDC interpretability remains challenging, and FSCS results indicate that BERT-based methods without few-shot learning can underperform on imbalanced labels.Source: §6.5 · p. 21

Sophisticated LJP Datasets from real courtrooms

• Judge-summarized datasets such as CAIL2018 and CAIL-Long may neglect admissibility of evidence and facts in real court settings, risking weaker case-logic representation and prediction correctness.LJP-MSJudge is the only identified real-courtroom dataset, covering ten fact labels for 70,482 private-lending cases; the authors recommend datasets that state legitimate intent when proof rules or evidence texts are ambiguous.Source: §7 · p. 22

Complex legal logical reasoning

• Improving prison-term prediction remains urgent even though article and charge prediction accuracy exceeds 90%.The authors call for legal knowledge from varied views and machine reasoning in NLP.Source: §7 · p. 22

Adaptive interpretability

• Explaining black-box LJP models remains challenging because practical judgment logic is specialized to each law domain.XLNet with BiGRU outperforms classical and sequential models on ILDC, but that performance does not remove the domain-specific explanation challenge.Source: §7 · p. 22

• The survey recommends testing evidence admissibility, complex reasoning features, and discriminative task-specific features in LJP datasets and neural models.Source: §8 · p. 22