Source-linked AI summary
Predicting the Law Area and Decisions of French Supreme Court Cases
Octavia-Maria Sulea, Marcos Zampieri, Mihaela Vela, Josef van Genabith
TL;DR
The paper addresses under-explored prediction of court rulings on French data and the absence of a temporal dimension in prior ruling-prediction work. It applies text classification to masked French Supreme Court rulings and reports high f1 scores for predicting law area and rulings, while date estimation is more difficult.
Problem
Method
The paper applies text classification to a French Supreme Court ruling collection, masking target mentions to simulate prediction from draft case descriptions.
Source: §3 · p. 2Results
A linear SVM trained on bag-of-words obtains high f1 scores for predicting law area and case ruling, while estimating case dates is more difficult.
Source: §6 · p. 6Takeaways & Limitations
The work is proof of concept for assistive technology that can help law professionals obtain guidance and orientation from large corpora of prior rulings.
Source: §6 · p. 6Date estimation is difficult, possibly because the dataset is imbalanced or because judicial language has changed little over time.
Source: §6 · p. 6
Abstract
from arXiv · showhide
In this paper, we investigate the application of text classification methods to predict the law area and the decision of cases judged by the French Supreme Court. We also investigate the influence of the time period in which a ruling was made over the textual form of the case description and the extent to which it is necessary to mask the judge's motivation for a ruling to emulate a real-world test scenario. We report results of 96% f1 score in predicting a case ruling, 90% f1 score in predicting the law area of a case, and 75.9% f1 score in estimating the time span when a ruling has been issued using a linear Support Vector Machine (SVM) classifier trained on lexical features.
1 Introduction
The paper applies text classification to French Supreme Court documents to predict case rulings, law areas, and ruling dates. It addresses an under-explored application of classification to court-ruling prediction and uses a corpus spanning from the 1800s to the present.
1 Introduction
- Text classification is applied to predict a case’s ruling, law area, and the date a ruling was issued.The study uses bag-of-words features and linear SVM classifiers on French Supreme Court rulings.Source: §1 · p. 1
- The corpus contains over 126,000 French Supreme Court documents spanning from the 1800s to the present day.Source: §1 · p. 1
- Predicting court rulings with text classification remains under-explored in legal NLP, despite prior work on tasks such as legal-text summarization.The cited ECHR decision-prediction study is one of the few examples identified by the authors.Source: §1 · p. 1
2 Related Work
Prior legal-text classification research addressed tasks including argument identification, domain classification, and summarization, while court-ruling prediction work was limited. The paper addresses gaps in French ruling prediction and temporal classification, using masking to emulate predictions from draft case descriptions.
2 Related Work
- Earlier legal-text classification covered argument identification, domain classification, summarization, and categorization of legal sentences and concepts.Palau and Moens (2009) report 80% accuracy for classifying argumentative versus non-argumentative sentences; Boella et al. (2011) report f1 measures of 76% and 97.5% for domain-related and specific-domain classification.Source: §2 · pp. 1–2
- Prior court-ruling prediction studies cited by the authors used US Supreme Court and ECHR data, reporting 70% and 78% accuracy, respectively.Source: §2 · p. 2
- To the authors’ knowledge, prior court-ruling prediction work had not used French data or incorporated a temporal dimension.Source: §2 · p. 2
- The paper masks mentions of target predictions to simulate classifying draft descriptions that do not state those targets.The authors apply this masking to all experiments.Source: §2 · p. 2
3 Data
The study uses a large collection of French Supreme Court rulings and defines prediction tasks for law area, ruling, and ruling date. It prepares labels and masks target-related text to approximate predictions from draft case descriptions.
3 Data
- The cleaned corpus contains 126,865 unique rulings with case descriptions and labels for law area, ruling date, case ruling, and cited laws.The source collection contains 131,830 XML documents; duplicate and incomplete entries were excluded.Source: §3 · p. 2
3.1 Tasks and Labels
- The three tasks predict law area, case ruling, and when a case description and ruling were issued.Source: §3.1 · p. 2
- Law-area prediction retains labels with over 200 examples, leaving eight classes.The table presents their distribution across case descriptions.Source: §3.1 · p. 2, Table 1 · p. 3
- Ruling prediction uses a first-word setup with six labels and a separate multi-word setup retaining labels with over 200 examples.In the multi-word setup, valid decisions such as non-lieu and qpc are excluded when their individual labels lack enough examples.Source: §3.1 · p. 3, Table 2 · p. 3
- Temporal labels use seven broad classes grouping all cases before 1960 together, with a finer alternative of 14 classes.The finer setup merges earlier periods into intervals including 1830–1840, 1850–1860, 1870–1880, and 1890–1910.Source: §3.1 · pp. 3–4, Table 3 · p. 4
3.2 Masking and Feature Selection
- For draft-description simulation, the authors remove target-label words, including nominal and verbal forms of ruling terms, from case descriptions.For law area, all words in the label are removed; for ruling prediction, feature inspection prompted removal of verbal forms as well as the ruling word itself.Source: §3.2 · p. 4
- The authors inspect the highest-ranking word bigrams and trigrams to assess whether selected features identify case facts.The reported examples include noun phrases, preposition-and-verb or noun combinations, and infinitival constructions.Source: §3.2 · p. 4
- For temporal prediction, all digits are removed, which can also remove digits referring to cited laws.Source: §3.2 · p. 4
4 Computational Approach
The computational approach uses linear SVM text classification with bag-of-words features, adding type-token ratio for temporal classification. Experiments use stratified cross-validation and a random baseline.
4 Computational Approach
- The classifiers use a scikit-learn implementation of LIBLINEAR SVM with unigram and bigram frequency features.CountVectorizer extracts the features from case descriptions.Source: §4 · pp. 4–5
- Type-token ratio is added to bag-of-words features for temporal classification because bag-of-words performed less well on that task.Source: §4 · p. 5
- Experiments use stratified 10-fold cross-validation, a distribution-aware random baseline, and average precision, recall, and f1 scores.The linear SVM uses C = 0.1 in all experiments.Source: §4 · p. 5
5 Results
The results cover law-area prediction, court-ruling prediction, and temporal classification under the paper’s masking regimes. Ruling and law-area prediction outperform random baselines, while temporal prediction is more difficult.
5.1 Law Area
- Law-area classification on 8 classes achieved high average precision, recall, and F1, well above the random baseline.Table 4 reports the 8-class Linear SVM task.Source: §5.1 · p. 5, Table 4 · p. 5
5.2 Court Ruling
- Although average scores show an apparent 6-percentage-point decrease when trained on the dataset with more classes, the method–random-baseline difference actually grows by 4 percentage points from the first-word setup.Source: §5.2 · p. 5
5.3 Temporal Text Classification
- 43% F1 versus 19% for random: type-token ratios alone did modestly better than random but did not improve BOW when combined.Temporal prediction used a linear SVM with unigram and bigram features in 7- and 14-class settings.Source: §5.3 · pp. 5–6
6 Conclusions and Future Work
The conclusion reports high BOW-based F1 scores for predicting law area and rulings, but says date prediction was harder. It also identifies formulaic ruling expressions and realistic masking as directions for further investigation.
6 Conclusions and Future Work
- A linear SVM trained on BOW achieved high F1 scores for predicting a case’s law area and ruling from its description.The conclusion characterizes date estimation using BOW and type-token ratio as more difficult, possibly because of class imbalance.Source: §6 · p. 6
- Date estimation was harder to learn, possibly because of minority-class scarcity or little change in judges’ language over time.The possible stability of judicial language is presented as an observation worth further investigation.Source: §6 · p. 6
- The most salient ruling-prediction bigrams and trigrams were formulaic expressions associated with rulings, not case-specific facts.Future work would test whether the most effective sentences for prediction are factual.Source: §6 · p. 6
- The authors frame the work as proof of concept for assistive guidance and orientation from large corpora of previous rulings.They propose further masking to simulate predictions from draft descriptions that omit explicit target information.Source: §6 · p. 6
- Future work would explore combining other features and using ensembles and meta-classifiers to improve performance.The paper notes that these approaches have achieved high performance in other text-classification tasks.Source: §6 · p. 6