Source-linked AI summary
Neural Legal Judgment Prediction in English
Ilias Chalkidis, Ion Androutsopoulos, Nikolaos Aletras
TL;DR
English legal judgment prediction lacked neural-model evidence and a large raw-text dataset. The paper releases an ECHR dataset, evaluates neural models across three tasks, studies anonymization-based bias, and proposes HIER-BERT. Neural models outperform prior feature-based English methods, while HIER-BERT bypasses BERT’s length limitation and achieves the best results.
Problem
English legal judgment prediction had been studied with feature-based models, while neural approaches had focused on Chinese; broader English evidence matters for predicting court outcomes.
Method
The paper releases an approximately 11.5k-case ECHR dataset, evaluates neural models on three tasks, tests anonymization, and proposes hierarchical BERT for long cases.
Results
Neural models outperform the prior English SVM with bag-of-words across binary violation, multi-label violation, and case importance prediction.
Takeaways & Limitations
The dataset establishes English neural baselines for legal judgment prediction and supports analysis of model sensitivity to demographic information.
Takeaways & Limitations
The models provide no justifications that legal practitioners could trust, making valid justification an important priority for future work.
Abstract
from arXiv · showhide
Legal judgment prediction is the task of automatically predicting the outcome of a court case, given a text describing the case's facts. Previous work on using neural models for this task has focused on Chinese; only feature-based models (e.g., using bags of words and topics) have been considered in English. We release a new English legal judgment prediction dataset, containing cases from the European Court of Human Rights. We evaluate a broad variety of neural models on the new dataset, establishing strong baselines that surpass previous feature-based models in three tasks: (1) binary violation classification; (2) multi-label classification; (3) case importance prediction. We also explore if models are biased towards demographic information via data anonymization. As a side-product, we propose a hierarchical version of BERT, which bypasses BERT's length limitation.
1 Introduction
The paper introduces an English ECHR legal judgment prediction dataset and evaluates neural models across three prediction tasks. It also examines demographic-information bias and proposes a hierarchical BERT model for long case descriptions.
- The dataset contains approximately 11.5k ECHR cases with raw text, substantially expanding prior English resources based on features from approximately 600 cases.
- Neural models are evaluated on binary violation, multi-label violation, and case importance prediction.
- Neural models outperform the English SVM with bag-of-words baseline across all three tasks.
- Data anonymization is used to investigate whether legal predictive models rely on demographic or other factual information.
- HIER-BERT bypasses BERT’s length limitation and leads to the best results.
2 ECHR Dataset
The ECHR dataset contains approximately 11.5k cases with extracted facts, violation labels, and importance scores. Its temporal split, class balancing, and rare labels support evaluation of both prediction and few-shot learning.
- The dataset contains approximately 11.5k cases from the ECHR public database.
- Each case provides facts extracted from its description, violated Convention articles when applicable, and an ECHR-assigned importance score.
- Training and development cases span 1959–2013, while test cases span 2014–2018.
- The training and development sets are balanced by violation status, whereas 66% of test cases contain violations.
- Forty-five of 66 labels are absent from training, and another 11 occur in fewer than 50 cases, creating a few-shot learning testbed.
3 Legal Prediction Tasks
The paper defines prediction tasks for violation presence, specific violated articles, and case importance using ECHR annotations. These tasks include binary, multi-label, and regression formulations.
- Binary violation prediction classifies a case as positive when any human-rights article or protocol was violated, and negative otherwise.
- Multi-label violation prediction assigns the specific violated articles or protocols among 66 possible labels, with no labels for non-violations.
- Case importance is predicted by regression on a scale from 1 for key cases to 4 for unimportant cases.
- The importance scores indicate a case’s contribution to the development of case law and help identify pivotal cases.
4 Neural Models
The evaluated models represent cases with recurrent networks, hierarchical attention, label-wise attention, or BERT. HIER-BERT applies BERT hierarchically to facts and cases to avoid truncating long descriptions.
- BiGRU-Att concatenates case facts into one word sequence and uses self-attention to compute a case embedding.
- HAN encodes each fact with a BiGRU and self-attention, then encodes the resulting fact embeddings with a second BiGRU and self-attention.
- LWAN uses one attention mechanism and case embedding per possible label, making it specific to multi-label violation prediction.
- BERT is fine-tuned with a task-specific output layer but truncates case descriptions because it processes up to 512 wordpieces.
- HIER-BERT processes words within each fact using BERT, then attends over fact embeddings to form a case embedding.
5 Experiments
Experiments compare neural and feature-based baselines across binary violation, multi-label violation, case-importance, attention, and anonymization settings. Hierarchical models perform strongly, but results also expose sensitivity to document length, rare labels, and demographic masking.
- Experimental Setup: Five runs use pretrained GLOVE embeddings, task-specific losses, random hyperparameter search, and mean scores with standard deviations.The setup uses categorical cross-entropy for classification and mean absolute error for regression.
- Binary Violation Results: 79.5 F1: BIGRU-ATT surpasses Aletras et al. (2016), while HIER-BERT reaches 80.5 versus 79.5 and has lower run variability, 0.2% versus 2.7%.BERT performs poorly when descriptions are truncated, whereas HIER-BERT uses the full case.
- Models Biases: Anonymization leaves most neural performance comparable, but HIER-BERT worsens by 2%, suggesting greater sensitivity to demographic information.The authors speculate that trainable BERT wordpiece embeddings may overfit vocabulary containing demographic information.
- Multi-label Violation Results: 60.0 micro-F1: HIER-BERT outperforms BIGRU-ATT at 57.6 and is comparable with HAN at 59.9, while few-label performance remains difficult.The 11 few labels occur in only 1.25% of documents, whereas four frequent articles occur in approximately 42%.
6 Related Work
Prior English legal judgment prediction used feature-based linear models, while neural approaches had mainly been studied for Chinese legal tasks. The paper situates its English neural evaluation against this limited prior work.
- English legal judgment prediction previously used linear models with bag-of-words and topic features.
- Neural legal judgment prediction models had been considered mainly in Chinese rather than English.
- Chinese work used hierarchical attention networks to encode case facts and selected relevant law articles for criminal-charge prediction.
- Chinese criminal-charge studies differed in how they handled rare labels, with one filtering charges appearing fewer than 80 times and another using multi-task learning.
7 Limitations and Future Work
The neural models outperform earlier feature-based systems but do not provide trusted justifications for their predictions. The authors therefore identify interpretability, broader resources, courts, languages, and modeling settings as future directions.
- Neural models outperform previous feature-based models but provide no justification that legal practitioners could trust.
- Attention scores indicate which text parts affect predictions, but they are not valid legal justifications.
- Future work will examine additional legal resources, other courts and languages, and adaptations of Chinese legal models with multi-task learning.