PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable Classification Models for Recidivism Prediction

Jiaming Zeng, Berk Ustun, Cynthia Rudin

arXiv:1503.07810v6stat.MLstat.AP

TL;DR

The paper asks how recidivism models can be accurate, transparent, and interpretable across decision contexts requiring different ROC operating points. It evaluates machine-learning methods across prediction problems and uses SLIM to construct constrained scoring systems. SLIM scoring systems are often as accurate as powerful black-box models while remaining transparent and interpretable.

  • Problem

    The paper addresses how to create recidivism models that are sufficiently accurate, transparent, and interpretable for decisions with different desired true and false positive rates.

  • Method

    The paper compares popular machine-learning methods across the ROC curve and uses Supersparse Linear Integer Models to learn sparse, integer-coefficient scoring systems from binary input variables.

  • Results

    Traditional methods such as Ridge Regression can perform as well as more modern methods, while SLIM often produces accurate scoring systems comparable to powerful black-box models.

  • Takeaways & Limitations

    SLIM can produce simple, accurate, and interpretable scoring systems at decision points across the ROC curve for high-stakes recidivism applications.

  • Takeaways & Limitations

    SLIM requires solving an integer programming problem and remains more computationally demanding to train than CART, C5.0, and penalized logistic regression.

Abstract

from arXiv · show

We investigate a long-debated question, which is how to create predictive models of recidivism that are sufficiently accurate, transparent, and interpretable to use for decision-making. This question is complicated as these models are used to support different decisions, from sentencing, to determining release on probation, to allocating preventative social services. Each use case might have an objective other than classification accuracy, such as a desired true positive rate (TPR) or false positive rate (FPR). Each (TPR, FPR) pair is a point on the receiver operator characteristic (ROC) curve. We use popular machine learning methods to create models along the full ROC curve on a wide range of recidivism prediction problems. We show that many methods (SVM, Ridge Regression) produce equally accurate models along the full ROC curve. However, methods that designed for interpretability (CART, C5.0) cannot be tuned to produce models that are accurate and/or interpretable. To handle this shortcoming, we use a new method known as SLIM (Supersparse Linear Integer Models) to produce accurate, transparent, and interpretable models along the full ROC curve. These models can be used for decision-making for many different use cases, since they are just as accurate as the most powerful black-box machine learning models, but completely transparent, and highly interpretable.

1. Introduction

Recidivism models must balance accuracy, transparency, and interpretability across decision contexts that require different operating points on the ROC curve. The paper evaluates existing methods and introduces SLIM scoring systems to address this need.

  • Research Question: The paper is motivated by conflicting claims about whether sophisticated machine learning methods provide accuracy benefits over established statistical approaches.The authors argue that the answer depends on how models are used for decision-making rather than being a simple yes-or-no question.
  • Decision Context: Different decisions may require different true positive and false positive rates, corresponding to distinct points along the ROC curve.The relevant operating point depends on the use case, including sentencing, parole, and policy interventions.
  • Approach: The paper compares popular machine learning methods across recidivism prediction problems and decision points along the ROC curve.The comparison includes methods such as support vector machines, random forests, and scoring systems.
  • SLIM: SLIM directly optimizes scoring systems for accuracy and sparsity while accommodating constraints such as false-positive-rate bounds and coefficient monotonicity.The method produces simple scoring systems at different decision points across the full ROC curve.
  • Interpretable Models: Existing criminal-justice scoring systems use sparse linear forms that users can calculate by adding, subtracting, and multiplying small numbers.These systems are designed for quick use without a computer, calculator, or nomogram.

2. Data and Prediction Problems

The study constructs six binary recidivism prediction problems from a large U.S. prisoner database, using accessible binary input rules and outcomes defined over three years after release. It also examines associations between inputs and outcomes while documenting important data limitations.

  • Prediction Problems: Each prediction problem uses N = 33,796 prisoners and P = 48 input variables to predict arrest for a crime type within three years of release.The six outcomes cover general, drug-related, and several violent-offense categories.
  • Database: The problems derive from a U.S. Department of Justice database tracking 38,624 prisoners released from 15 states in 1994.The database followed prisoners for three years after release and is described as the largest publicly available U.S. prisoner-recidivism database.
  • Data Limitations: The dataset omits or incompletely records important factors, including education, family history, reentry programs, and employment history.Drug-treatment information, for example, has a valid entry for fewer than 14% of prisoners, leading the authors to exclude extremely sparse factors.
  • Input Variables: The 48 inputs are encoded as binary rules, allowing linear models to represent nonlinear functions of the original variables while improving understandability.The variables use information commonly accessible to police officers and judges and exclude socioeconomic factors such as race.
  • Prediction Problems: The six binary outcomes indicate whether a prisoner is arrested for any crime, a drug offense, or specified violent offenses within three years after release.The violent-offense categories are treated as broader categories, and their outcomes are mutually exclusive.
  • Input–Outcome Relationships: Conditional probabilities reveal expected associations: later release age is linked to lower arrest probability, whereas more prior arrests are linked to higher arrest probability.These associations are used to provide insight into the problems and qualitatively validate predictive models.
  • Input–Outcome Relationships: Prior offense type is associated with crime-specific recidivism: prior drug arrest corresponds to a 32% drug rearrest probability, while prior domestic violence corresponds to 13% domestic-violence arrest probability.Table 3 organizes conditional probabilities by input variable and prediction problem.

3. Supersparse Linear Integer Models

SLIM creates sparse linear scoring systems with small integer coefficients by directly optimizing classification error and model simplicity. Its formulation supports operational constraints, class-specific accuracy preferences, and computational improvements for repeated binary feature patterns.

  • Model definition: SLIM scoring systems are sparse linear classifiers that make predictions by adding, subtracting, and multiplying a few small numbers.Their compact form is intended to support quick, transparent predictions without a computer or extensive statistical training.
  • Optimization objective: The optimization directly minimizes misclassification error while penalizing the number of non-zero coefficients under discrete coefficient constraints.Coefficients are selected from a finite set, such as integers from −10 to 10, and additional operational constraints can be included.
  • Coefficient selection: A tiny absolute-coefficient penalty selects coprime coefficients among classifiers with identical predictions and sparsity.For example, SLIM prefers sign(x1 + x2) over sign(2x1 + 2x2) because both predict identically but the former has smaller coefficients.
  • Operational controls: The C0 parameter controls the accuracy sacrificed to remove a feature, while W+ and W− control accuracy on positive and negative examples.When C0 lies between 1/N and 2/N, one observation’s accuracy can be sacrificed for a model with one fewer feature; typically W+ + W− = 2.
  • Customization and computation: SLIM trades scalability and rapid fitting for direct optimization and the ability to encode constraints on sparsity, class-specific accuracy, structure, and prediction.The models are trained by solving an integer-programming problem with a solver such as CPLEX, Gurobi, or CBC.

4. Experimental Results

Across six recidivism prediction problems, the experiments compare nine methods for accuracy across the ROC curve, interpretability, calibration, and constrained model size. SLIM generally combines accurate performance across decision points with sparse, transparent scoring systems, while tree and rule methods struggle on imbalanced problems and under interpretability constraints.

  • All methods performed well on arrest: every method except CART exceeded a test AUC of 0.72, while SGB, Ridge, and RF reached 0.73.
  • Imbalanced problems produced major method differences, and some methods were difficult or impossible to tune to specific ROC-curve points.The difficulty increased for highly imbalanced problems such as fatal violence.
  • SVM RBF, SGB, Lasso, and Ridge produced accurate models across different ROC-curve points for most problems, although cost tuning remained difficult for highly imbalanced tasks.SGB usually achieved the highest AUC, while Lasso, Ridge, and SVM RBF often produced comparable AUCs.
  • C5.0T, C5.0R, and CART failed to produce accurate models across ROC-curve points on imbalanced problems, often yielding trivial AUC-0.5 models as imbalance increased.The authors attribute this pattern to the greedy tree-fitting algorithms rather than tree models generally, and note that better optimization might avoid it.
  • SLIM stayed near the efficient ROC frontier while using at most 8 non-zero coefficients ranging from -10 to 10, including on highly imbalanced problems.Its formulation makes it responsive to changes in misclassification costs.
  • Under constrained model size, SLIM and non-sparse Lasso were the only methods found to consistently maintain accuracy across the full ROC curve while retaining interpretability potential.The other baseline methods did not maintain their unconstrained accuracy when model size was restricted.

5. Discussion

The paper combines predictive accuracy with transparency by comparing models across decision points on the ROC curve. It argues that SLIM can provide accurate, interpretable scoring systems, although training requires substantially more computation than several alternatives.

  • The study compares machine learning methods using accessible features across different decision points on the ROC curve.This evaluates both predictive performance and interpretability for recidivism decision-making.
  • Ridge Regression can perform as well as more modern methods such as Stochastic Gradient Boosting.
  • SLIM can dependably produce simple scoring systems that remain accurate and interpretable across the ROC curve.Its mechanism is visible and it can accommodate constraints reflecting decision-makers’ ethical values.
  • SLIM’s main drawback is increased computation from solving an integer programming problem.The authors report that training time was comparable to random forests or stochastic gradient boosting but longer than CART, C5.0, and penalized logistic regression.

A. Additional Results on Predictive Accuracy

The additional results report training and five-fold validation AUC across all prediction problems and methods, alongside transparent-model comparisons. Calibration plots require special interpretation when class imbalance makes them appear flat.

  • Training, five-fold validation, and test AUC results are reported across all methods and prediction problems.Table 6 contains training AUC, Table 7 contains mean five-fold validation AUC and ranges, and Table 5 contains test AUC.
  • Additional transparent-model comparisons cover recidivism prediction problems beyond arrest.
  • The displayed Lasso and Ridge models operate at similar TPR decision points and share the same FPR criteria for each problem.CART and C5.0 models are omitted because they were trivial or contained too many leaves to print.
  • Highly imbalanced problems can produce calibration plots that appear flat even for well-calibrated classifiers.For fatal violence, the passage gives P(y = +1) = 0.7% as an example.

B.1. drug

For the drug prediction problem, SLIM, Ridge, and Lasso models are presented with test and five-fold validation TPR/FPR results, together with their scoring features. The figure is a risk calibration plot for drug.

  • SLIM achieves a test TPR/FPR of 85.7%/51.1% and mean five-fold validation TPR/FPR of 82.3%/49.7%.
  • Lasso achieves a test TPR/FPR of 82.0%/45.9% and mean five-fold validation TPR/FPR of 81.2%/45.9%.
  • The drug models include prior-arrest features such as drugs, property, multiple crime types, other violence, sexual offenses, firearms, and public order.These features appear across the SLIM, Ridge, and Lasso scoring-system listings.
  • Ridge achieves a test TPR/FPR of 84.0%/48.2% and mean five-fold validation TPR/FPR of 83.1%/48.4%.

B.2. general violence

For general violence, SLIM, Ridge, and Lasso achieve strong performance while using transparent models. Their reported test operating points show similar trade-offs between true- and false-positive rates.

  • Model results: 76.7%/45.4% test TPR/FPR is reported for the SLIM general-violence model.Its mean 5-CV validation TPR/FPR is 76.8%/47.6%.
  • Model results: 79.7%/45.5% test TPR/FPR is reported for the best Lasso general-violence model.Its mean 5-CV validation TPR/FPR is 77.3%/45.7%.
  • Model results: 81.4%/48.1% test TPR/FPR is reported for the best Ridge general-violence model.Its mean 5-CV validation TPR/FPR is 80.0%/48.5%.

B.3. domestic violence

For domestic violence, SLIM, Ridge, and Lasso all achieve high reported test true-positive rates with differing false-positive rates. The listed models use prior violence, criminal history, prison, and release features.

  • Model results: 85.5%/46.0% test TPR/FPR is reported for the SLIM domestic-violence model.Its mean 5-CV validation TPR/FPR is 81.4%/48.0%.
  • Model structure: The domestic-violence models include prior arrests for misdemeanor, felony, domestic violence, sexual offenses, and other violence.Additional predictors include prison infractions, jail or prison time, release status, and prior probation or fines.
  • Model results: 87.0%/45.8% test TPR/FPR is reported for the best Lasso domestic-violence model.Its mean 5-CV validation TPR/FPR is 84.5%/45.8%.
  • Model results: 87.0%/47.7% test TPR/FPR is reported for the best Ridge domestic-violence model.Its mean 5-CV validation TPR/FPR is 85.2%/47.5%.

B.4. sexual violence

Reported performance differs substantially across sexual and fatal violence prediction problems. The models use crime history, incarceration experiences, release status, and other criminal-justice features.

  • Sexual violence: 44.3%/17.7% test TPR/FPR is reported for the SLIM sexual-violence model.Its mean 5-CV validation TPR/FPR is 43.7%/19.9%.
  • Sexual violence: 46.9%/18.1% test TPR/FPR is reported for the best Lasso sexual-violence model.Its mean 5-CV validation TPR/FPR is 43.7%/17.9%.
  • Sexual violence: 48.6%/19.3% test TPR/FPR is reported for the best Ridge sexual-violence model.Its mean 5-CV validation TPR/FPR is 44.9%/19.4%.
  • Fatal violence: 55.4%/35.5% test TPR/FPR is reported for the SLIM fatal-violence model.Its mean 5-CV validation TPR/FPR is 64.2%/42.4%.
  • Fatal violence: 68.9%/44.5% test TPR/FPR is reported for the best Lasso fatal-violence model.Its mean 5-CV validation TPR/FPR is 67.6%/42.4%.
  • Fatal violence: 62.2%/34.0% test TPR/FPR is reported for the best Ridge fatal-violence model.Its mean 5-CV validation TPR/FPR is 60.1%/33.0%.

C. Additional Results on the Trade-off between Accuracy and Interpretability

The paper tests whether transparent methods retain accuracy under a model-size constraint. Restricting models to at most eight coefficients, leaves, or rules compromises predictive accuracy for methods other than SLIM.

  • Experimental setup: Models are restricted to at most 8 non-zero integer coefficients, 8 leaves, or 8 rules to compare transparent model classes.The experiments contrast SLIM’s constrained space with slightly less constrained baseline spaces.
  • Evaluation: Table 8 reports test AUC for each method under the model-size restriction.Trivial models of size 1 are omitted.
  • Results: For all models other than SLIM, predictive accuracy was compromised by the size constraint.C5.0R and C5.0T could not produce suitably sparse models for some problems because their implementation lacked control over model sparsity.
  • Evaluation: Table 9 reports percentage changes in test AUC relative to SLIM under the restriction.The comparison covers all prediction problems and transparent methods.

D. Trade-off between Risk Calibration and Interpretability

With model size constrained to 8 or fewer, transparent models show trade-offs in risk calibration, and Ridge has no models meeting the stated constraint and decision criteria.

  • Risk calibration: Ridge produced no transparent models of size 8 or less under the decision criteria used for Figure 16.The comparison uses the same decision criteria as Appendix B, with Ridge’s infeasibility also discussed in Appendix C.
  • Risk calibration: Lasso’s risk-calibration performance was worse than in Figures 11–15 under the size-constrained comparison.
  • Risk calibration: No Lasso model was available at the desired decision point for fatal violence.

E.1. Change in Predictive Accuracy using Only Continuous Input Variables

Replacing binary inputs with continuous or combined encodings changes test AUC inconsistently across methods and problems, while association-rule mining provides interpretable IF-THEN patterns for arrest prediction.

  • Continuous variables: 4.6% for CART and 7.7% for SVM RBF were the largest test-AUC increases, while RF had the largest decrease at −19.6% when using continuous variables.The comparison changes the encoding from 48 binary variables to 25 continuous variables.
  • Continuous variables: For most methods, test AUC increased on at least one problem and decreased on at least another when binary variables were replaced by continuous variables.
  • Continuous variables: CART improved slightly on 5 of 6 problems with continuous variables, although it still performed poorly relative to other methods.
  • Association rules: Association rules are small IF-THEN predictive models generated through search or optimization techniques.For arrest, the rules use input-variable conditions and a positive outcome indicating arrest within 3 years of release.
  • Association rules: The mined arrest rules were selected for high lift and confidence with minimum support of 5%, intended to reduce spurious correlations.They were generated with the apriori method, and 24 rules are listed in Table 14.

F.3. Falling Rule Lists for Imbalanced Problems

Falling Rule Lists provide globally optimized ordered rules for imbalanced classification, where conventional tree and rule-based methods can struggle; the paper also examines race-related inputs and prediction.

  • Falling Rule Lists: Traditional tree and rule-based methods are difficult to use for non-trivial models on imbalanced problems such as sexual violence.The paper suggests greedy splitting and pruning procedures may contribute to this difficulty.
  • Falling Rule Lists: The paper trains Falling Rule Lists to test whether rule-based models can perform well on imbalanced classification problems.
  • Falling Rule Lists: Falling Rule Lists are ordered IF-THEN rules whose confidence decreases down the list, assigning the highest-risk group to the first rule.Their algorithm globally optimizes the list without greedy splitting and pruning.
  • Falling Rule Lists: The arrest Falling Rule List uses rules with at most two input variables and minimum support of 5%.Table 15 reports support as the percentage of remaining examples satisfying the conditions and probability as the percentage with the outcome.
  • Race-related inputs: Including race-related indicators did not substantially increase prediction accuracy for the recidivism problems.The added indicators were white, black, and hispanic, compared with models using the standard variables.
  • Race-related inputs: Race was predicted much better than random guessing from the existing features, but not perfectly, for white, black, and hispanic outcomes.The comparisons are shown with ROC plots in Figure 17.
Loading 1503.07810v6…