Source-linked AI summary
A General Approach for Predicting the Behavior of the Supreme Court of the United States
Daniel Martin Katz, Michael J Bommarito, Josh Blackman
TL;DR
The paper addresses the need for judicial prediction models that are general, consistent, and usable out of sample. It builds a time-evolving random forest using pre-decision Supreme Court data, achieving strong prediction accuracy across nearly two centuries and both Justice- and case-level tasks.
Problem
Existing judicial prediction approaches are limited in generality, consistency, or out-of-sample applicability, despite the importance of predicting Court behavior.
Method
The paper uses Supreme Court Database features and engineered variables in a time-evolving random forest trained only on information available before each decision.
Results
70.2% case-outcome accuracy and 71.9% Justice-vote accuracy were achieved from 1816–2015, with nearly 5% improvement over an in-sample optimized null model over the past century.
Takeaways & Limitations
The model provides a generalized, consistent, out-of-sample baseline for quantitative legal prediction across changing Courts and historical periods.
Takeaways & Limitations
Prior comparison models could not be generalized across all Supreme Court cases because they were evaluated within a single natural Court and personnel period.
Abstract
from arXiv · showhide
Building on developments in machine learning and prior work in the science of judicial prediction, we construct a model designed to predict the behavior of the Supreme Court of the United States in a generalized, out-of-sample context. To do so, we develop a time evolving random forest classifier which leverages some unique feature engineering to predict more than 240,000 justice votes and 28,000 cases outcomes over nearly two centuries (1816-2015). Using only data available prior to decision, our model outperforms null (baseline) models at both the justice and case level under both parametric and non-parametric tests. Over nearly two centuries, we achieve 70.2% accuracy at the case outcome level and 71.9% at the justice vote level. More recently, over the past century, we outperform an in-sample optimized null model by nearly 5%. Our performance is consistent with, and improves on the general level of prediction demonstrated by prior work; however, our model is distinctive because it can be applied out-of-sample to the entire past and future of the Court, not a single term. Our results represent an important advance for the science of quantitative legal prediction and portend a range of other potential applications.
Introduction
Supreme Court decisions raise consequential prediction questions, but existing forecasts are often difficult to formalize, reproduce, or test prospectively. This paper develops an out-of-sample model for both Court-level outcomes and individual Justice votes.
- Court decisions address diverse legal questions and can affect society beyond the litigants.
- Prediction efforts are widespread, but their quality and underlying models are often unclear, historically untested, or difficult to reproduce.
- The paper evaluates whether the Court will affirm or reverse the status quo at both the case and individual-Justice levels.
- The model combines Supreme Court Database features, feature engineering, and random forests to predict decisions from 1816 through 2015 against multiple baselines.
- Using only pre-decision information, the model outperforms baseline models at both Justice and Court observation levels under parametric and non-parametric tests.
Research Principles and Prior Work
The paper motivates three principles for judicial prediction—generality, consistency, and out-of-sample applicability—because prior approaches do not satisfy them simultaneously. Its model is designed to remain usable across changing Courts and future decisions.
- Prior Work: Prior classification-tree models focused on a single unusually stable natural Court, limiting evaluation across other personnel periods.
- Generality: Generality requires predictions to continue across changes in Court composition, recusals, retirements, deaths, and abnormal case circumstances.
- Consistency: Consistency means maintaining performance across time, case issues, and Justices rather than optimizing only for isolated settings.
- Out-of-Sample Applicability: Out-of-sample applicability requires every input needed for an estimate to be knowable before the decision date.
- Synthesis: Existing approaches each satisfy only subsets of the three principles, whereas this approach is presented as satisfying all three.
Data and Feature Engineering
The model uses the Supreme Court Database and engineered features to represent decisions, Justice votes, case context, timing, and historical behavior. Outcomes are coded to handle both ordinary appeals and nonstandard cases.
- SCDB: SCDB supplies more than two centuries of coded Court data, including chronological, case-background, Justice-specific, and outcome variables.
- SCDB: SCDB is described as the longest-duration and highest-quality database for Supreme Court decisions, while retaining important documented limits.
- SCDB: The study uses SCDB Legacy data beginning in 1791 and reruns the analysis with the expanded historical release.
- Targets: Justice votes are coded as Reversed, Affirmed, or Other, and case outcomes are Reverse when Reverse votes outnumber Affirm votes.
- Features and Feature Engineering: Features include case characteristics, jurisdiction, lower-court information, issues, timing, and behavioral summaries of Justices and Courts.
- Features and Feature Engineering: Timing features can be updated as undecided cases progress, allowing predictions to change without using post-decision information.
Model Construction
The model trains sequentially on information available before each target term and applies a random forest that evolves as new data arrive. The reported implementation emphasizes a growing forest for speed, stability, and longitudinal applicability.
- Temporal Training: Training for term T* uses only docket-vote records from terms earlier than T*, preventing later information from entering the training phase.
- Data Representation: The dataset contains 249,793 docket-votes and 1,501 engineered feature columns as of 2015.
- Random Forests: Random forests combine diverse trees trained on bootstrap samples and random feature subsets to produce predictions from the ensemble.
- Model Selection: Random forests outperformed support vector machines and feedforward neural networks in the authors’ testing.
- Growing Forest: A growing forest adds trees to an existing model, while fresh forests are rebuilt mainly when the Court’s membership changes.
- Growing Forest: The presented implementation uses five trees per step because growing forests provide faster simulation and more stable predictions.
Model Testing and Results
The model is evaluated at justice and case levels across nearly two centuries, using historical baselines and statistical tests to assess accuracy, stability, and out-of-sample performance. It achieves 71.9% justice-vote accuracy and 70.2% case-decision accuracy, generally outperforming the null models while showing weaker relative performance for the Roberts Court.
- Justice-level results: 71.9% justice-vote accuracy was achieved over 1816–2015 in the model’s justice-level predictions.The justice model predicts three classes—Affirm, Reverse, and Other—while case outcomes use whether votes are Reverse or not.
- Case-level results: 70.2% of Supreme Court decisions were correctly predicted from 1816 through the October 2014 term.Case outcomes are determined by whether a majority of Justice votes favor reversing the prior status quo.
- Consistency over time: The model typically delivers stable case- and justice-level performance across nearly two hundred years, although accuracy varies by year and decade.Figure 1 presents the time series of accuracy by term for both prediction levels.
- Baseline models: 66.2% justice-level and 67.5% case-level accuracy were obtained by the in-sample-optimized M = 10 finite-memory null model from 1816–2015.This provides a historical baseline for comparison with the prediction model.
- Comparison against baselines: The model generally outperforms the baseline models across much of the last two centuries, especially against always-Reverse and infinite-memory strategies during sustained periods.The comparison includes always guess Reverse, infinite memory, and optimized finite memory; statistical tests indicate significant aggregate and per-case improvement.
- Limitations and changing predictability: The model’s performance dipped during the Roberts Court, which appears less predictable than its immediate predecessors within the study’s scope.The authors state that the available evidence does not establish whether this reflects a systematic change in the Court’s macro-dynamics.
Conclusion and Future Research
The paper presents a generalized, consistent, out-of-sample machine-learning model for Supreme Court prediction and positions it as a baseline for future judicial-prediction research. It also connects the approach to broader applications of machine learning and ensemble methods in legal prediction.
- 70.2% case-outcome accuracy and 71.9% justice-vote accuracy were achieved across nearly two centuries of Supreme Court decisions.The model also outperformed an in-sample optimized null model by nearly 5% over the past century.
- The model is designed as a general, consistent, out-of-sample approach that can remain applicable across many Justices and changing historical conditions.The authors distinguish this goal from optimizing performance for a particular case or time period.
- The approach contributes to broader legal-studies research applying machine learning and predictive analytics to law.The authors encourage further applied research in existing and new legal domains.
- The paper relies on an ensemble method that transforms weak learners into a strong learner, motivating future combinations of experts, crowds, and algorithms.The authors suggest such blends may produce strong performance across a wide class of prediction problems.