Source-linked AI summary
Exploring the Use of Text Classification in the Legal Domain
Octavia-Maria Sulea, Marcos Zampieri, Shervin Malmasi, Mihaela Vela, Liviu P. Dinu, Josef van Genabith
TL;DR
The paper asks whether text classification can support legal professionals by predicting case outcomes and categories from realistic draft case descriptions. It masks target information in French Supreme Court rulings and applies an SVM ensemble system across law-area, ruling, and temporal tasks. The ensemble outperforms the prior method across the paper’s evaluated tasks, while temporal predictions remain estimates because time-span definitions are arbitrary and supervised methods do not capture time linearity.
Problem
The paper addresses whether machine-learning text classifiers can support legal professionals by predicting decisions from draft case descriptions in an under-explored legal domain.
Method
The study masks target references in French Supreme Court case descriptions and applies a mean-probability ensemble of SVM classifiers to three prediction tasks.
Results
The ensemble outperformed the prior method across law-area, ruling, and time-span prediction tasks on the same dataset.
Takeaways & Limitations
Text classification can provide assistive technology for law professionals seeking guidance and orientation from large corpora of previous court rulings.
Takeaways & Limitations
Temporal predictions are estimates because time-span boundaries are arbitrary and supervised methods do not capture the linearity of time.
Abstract
from arXiv · showhide
In this paper, we investigate the application of text classification methods to support law professionals. We present several experiments applying machine learning techniques to predict with high accuracy the ruling of the French Supreme Court and the law area to which a case belongs to. We also investigate the influence of the time period in which a ruling was made on the form of the case description and the extent to which we need to mask information in a full case ruling to automatically obtain training and test data that resembles case descriptions. We developed a mean probability ensemble system combining the output of multiple SVM classifiers. We report results of 98% average F1 score in predicting a case ruling, 96% F1 score for predicting the law area of a case, and 87.07% F1 score on estimating the date of a ruling.
1 Introduction
The paper examines text classification in the relatively under-explored legal domain, asking whether machine learning can predict decisions from draft case descriptions. It uses French court rulings to support predictions of law area, ruling, and ruling date.
- Motivation: Legal text classification remains relatively under-explored beyond forensic applications.The paper positions its work within broader NLP applications while emphasizing the limited use of classification in legal settings.
- Motivation: The central question is whether classifiers can predict legal decisions from a professional’s textual draft case description.Target decisions include the applicable law area, likely ruling, and relevant laws.
- Study scope: 126,000+ French court rulings spanning the 1800s to the present provide the corpus for predicting law area, ruling, and ruling date.The study uses lexical features and Support Vector Machine ensembles and compares results with prior work using the same data.
- Study scope: The experiments mask target information in case descriptions to create synthetic drafts resembling what lawyers would have before court.The masked descriptions form the training and test inputs for all reported predictions.
2 Related Work
Prior legal NLP work covers domain classification, argumentation, summarization, and ruling prediction, but court-ruling prediction remains a newer application. The paper contributes to this area with performance substantially above earlier work.
- Existing legal NLP: Legal text classification has addressed domain identification, but its use in the legal domain remains under-explored.Earlier work used TF-IDF, Information Gain, and SVMs for legal-domain classification.
- Existing legal NLP: Argumentation analysis classified legal sentences as argumentative or non-argumentative with 80% accuracy using European Court of Human Rights texts.The task identified argumentative propositions, functions, and structures.
- Existing legal NLP: Court-ruling summarization studies identified thematic structure and classified sentences into summarization-related classes.Reported results included 65% F1 on seven classes and 64% and 79% F1 in a 3,000-category legal-concept task.
- Ruling prediction: Earlier ruling-prediction studies reported 70% accuracy for US Supreme Court behavior and 19% F1 for patent-litigation outcomes.Patent time-to-litigation prediction reached 87% F1 for intervals under four years but 43% F1 for intervals under one year.
- Ruling prediction: Court-ruling prediction is presented as a new text-classification area in which this paper reports substantially higher performance than previous work.The comparison is made against prior work cited as [24].
3 Corpus and Data Preparation
The study builds a cleaned corpus of French Supreme Court rulings with natural metadata labels and defines three classification tasks. Target-label mentions are masked to simulate realistic draft case descriptions.
- Corpus: The collection contains 131,830 French Supreme Court rulings with metadata including law area, timestamp, ruling, and cited laws.After removing duplicate and incomplete entries, the corpus contains 126,865 unique rulings.
- Data preparation: Target-class mentions are removed from training and test data to simulate realistic prediction from draft case descriptions.The masking removes explicit references to the classes being predicted.
- Tasks: The three tasks predict law area, predict court ruling from case description, and estimate the time span of a case description and ruling.These tasks use metadata labels available in the ruling collection.
- Data constraints: Label selection is complicated by missing explicit information, imbalanced class distributions, and occasional inconsistencies in the dataset.The paper follows prior work’s label decisions in response to these data conditions.
- Task setup: Law-area prediction retains eight labels appearing more than 200 times among the initial labels.The distribution of cases across these law-area labels is reported in Table 1.
- Task setup: Ruling prediction uses both six-class and eight-class setups, with labels selected by frequency and treated either by first word or in full.The eight-class setup retains labels with more than 200 dataset entries without reducing them to their first word.
- Task setup: Temporal classification uses seven classes, grouping cases dated 1959 and earlier into one interval.The study examines whether case descriptions contain indicators of their writing period.
- Data preparation: Law-area labels are masked by removing all words contained in the corresponding label, while ruling prediction also removes nominal and verbal ruling forms.The additional ruling masking aims to eliminate direct target cues from case descriptions.
4 Methodology
The paper uses word unigram and bigram features with a mean-probability ensemble of classifiers for three tasks. Performance is evaluated with stratified 10-fold cross-validation using standard classification metrics.
- Model: The system combines classifier ensembles with word unigram and bigram features for all three tasks.The ensemble approach is motivated by strong performance in several other classification tasks.
- Model: The mean-probability ensemble averages class probability estimates and predicts the class with the highest average probability.This preserves support for a true class even when an individual classifier ranks it below first place.
- Baseline: The ensemble is compared with prior results from a LIBLINEAR SVM trained on bag-of-words and bag-of-bigrams features.The comparison uses the same three tasks described in the paper.
- Evaluation: All experiments use stratified 10-fold cross-validation to account for class imbalance and enable comparison with earlier results.The paper reports average precision, recall, F1 score, and accuracy for all classes.
5 Results
The experiments evaluate ensemble-based classification for law area, court ruling, and temporal prediction. Performance is strong overall, while errors concentrate in smaller or mixed classes and temporal estimates have methodological limits.
- 5.1 Law Area: 96.8% accuracy was achieved for law area prediction, compared with 90.3% for the linear SVM baseline.
- 5.1 Law Area: Chambre mixte and assemblee pleniere were the most difficult law-area classes, partly because they had only 222 and 544 instances.Chambre mixte also contains mixed cases from civil and commercial courts.
- 5.2 Case Ruling: The mean probability ensemble outperformed the comparison method for ruling prediction, while adding classes reduced average F1 by 2.9 percentage points.The decrease occurred when moving from six to eight classes, making classification more challenging.
- 5.2 Case Ruling: Non-lieu and annulation were the most difficult ruling classes in the six-class setup and also had the fewest examples.
- 5.3 Temporal Text Classification: 87% F1 was achieved for seven-class temporal classification, compared with 73.2% for the comparison method.
- 5.3 Temporal Text Classification: Temporal predictions are estimates because the boundaries of month, year, or decade categories are arbitrary, and supervised methods do not capture time linearity.Ranking or regression methods could produce more accurate predictions.
6 Conclusions and Future Work
The paper finds that SVM ensembles can predict French Supreme Court law areas, rulings, and case time spans from case descriptions, outperforming a previous system on the same dataset. Masking ruling-related information creates a description that simulates what a lawyer would know before entering court, while future work targets more accurate draft case descriptions.
- SVM ensembles obtain high scores when predicting French Supreme Court law areas, rulings, and case time spans from case descriptions.
- Masking ruling-related information produces case descriptions that convey little information about the ruling and simulate a lawyer’s knowledge before court.
- Future work will investigate whether a more accurate draft form can be induced from the court’s case descriptions.