Source-linked AI summary
Peeking Inside the Black Box: Visualizing Statistical Learning with Plots of Individual Conditional Expectation
Alex Goldstein, Adam Kapelner, Justin Bleich, Emil Pitkin
TL;DR
Black-box models can be accurate yet difficult to interpret, and PDPs can conceal heterogeneous relationships when interactions are present. The paper introduces ICE plots and related visual tools that display observation-specific fitted relationships, revealing interactions and extrapolations. Simulations and real-data examples show that ICE can expose model structure that PDPs obscure, while a visual test assesses additivity in the data-generating process.
Problem
Black-box models offer limited interpretability, while PDP averages can be misleading when interactions make partial relationships heterogeneous.
Method
The paper introduces ICE plots and complementary centered and derivative ICE plots for visualizing observation-specific fitted relationships and their heterogeneity.
Results
ICE plots reveal fitted-model structure, including interaction-driven curve heterogeneity and extrapolations, in simulations and real-data examples where PDPs can obscure it.
Takeaways & Limitations
The ICE toolbox provides additional insight into how black-box learning algorithms use covariates to generate predictions and supports visual investigation of interactions.
Takeaways & Limitations
PDP interpretation depends on selected features not being too strongly dependent on remaining features, and extrapolation desirability depends on the application.
Abstract
from arXiv · showhide
This article presents Individual Conditional Expectation (ICE) plots, a tool for visualizing the model estimated by any supervised learning algorithm. Classical partial dependence plots (PDPs) help visualize the average partial relationship between the predicted response and one or more features. In the presence of substantial interaction effects, the partial response relationship can be heterogeneous. Thus, an average curve, such as the PDP, can obfuscate the complexity of the modeled relationship. Accordingly, ICE plots refine the partial dependence plot by graphing the functional relationship between the predicted response and the feature for individual observations. Specifically, ICE plots highlight the variation in the fitted values across the range of a covariate, suggesting where and to what extent heterogeneities might exist. In addition to providing a plotting suite for exploratory analysis, we include a visual test for additive structure in the data generating model. Through simulated examples and real data sets, we demonstrate how ICE plots can shed light on estimated models in ways PDPs cannot. Procedures outlined are available in the R package ICEbox.
1 Introduction
The article introduces ICE plots as a toolbox for visualizing how black-box supervised-learning models map predictors to fitted values. By disaggregating PDP averages into observation-specific curves, ICE plots expose interactions and extrapolations that average summaries may miss.
- ICE plots visualize the mapping from predictors to fitted values produced by arbitrary black-box supervised-learning algorithms.These algorithms may achieve low generalization error while remaining difficult to understand.
- ICE extends PDPs by displaying the estimated functional relationship separately for each observation rather than only the average partial relationship.
- Observation-specific ICE curves help identify interactions in the fitted model and extrapolations in predictor space.
- The paper uses simulations, real-data examples, and a visual additivity test to examine model features that PDPs may hide or misrepresent.
2 Background
The background motivates model-agnostic visualization for accurate but opaque black-box models and reviews PDPs as an average summary. It then shows, through a simulated interaction, how PDP averaging can obscure a meaningful modeled relationship.
- 2.1 Survey of Black Box Visualization: Black-box algorithms can improve predictive accuracy but provide limited interpretability, motivating tools that reveal how they use covariates.
- 2.1 Survey of Black Box Visualization: Existing visualization methods include algorithm-specific techniques and model-agnostic approaches for examining black-box predictions.
- 2.2 Friedman’s PDP: A PDP plots the change in the average predicted value as selected features vary over their marginal distribution.
- 2.2 Friedman’s PDP: 1,000 observations were generated and fit with stochastic gradient boosting; the resulting PDP suggested that X2 lacked a meaningful association with predicted Y.
- 2.2 Friedman’s PDP: PDPs can be misleading when selected features interact strongly with remaining features and are not particularly effective for revealing extrapolations in predictor space.
3 The ICE Toolbox
ICE plots display observation-specific fitted relationships, revealing heterogeneity and interactions that average PDP curves can obscure. Centered, derivative, and colored variants further isolate cumulative effects, local heterogeneity, and the role of a second predictor.
- The ICE Procedure: ICE plots graph N conditional expectation curves, each showing predicted response as a function of xS while holding observed xC fixed.The PDP is the average of these ICE curves, making ICE a disaggregated view of the fitted model.
- The ICE Procedure: In simulated data, ICE reveals that SGB fitted values are approximately linearly increasing or decreasing in X2 depending on an observation’s region of X.This relationship contrasts with the corresponding PDP’s average pattern.
- The ICE Procedure: For Boston Housing Data, ICE shows observations whose predicted median home prices increase with age even though the PDP is largely flat or slightly decreasing.The curves expose departures from the average behavior represented by the PDP.
- The Centered ICE Plot: Centered ICE removes level effects by pinning curves at a reference x∗, thereby isolating the cumulative effect of xS while holding xC fixed.Choosing the minimum observed xS makes all curves originate at 0; choosing the maximum displays cumulative effects relative to that baseline.
- The Centered ICE Plot: In the BHD c-ICE plot, age increases predicted median value for some cases and decreases it for others, suggesting interactions with other predictors.The right axis expresses the effect magnitude as a fraction of y’s observed range, reaching about 14% in this example.
- The Derivative ICE Plot: d-ICE plots partial derivatives: equivalent curves indicate no interactions, whereas heterogeneous derivative lines highlight interaction regions and their standard-deviation summary.For the RF BHD model, derivatives are near zero below age 60 but have both positive and negative values above 60.
- Visualizing a Second Feature: Coloring ICE curves by a second predictor allows its influence on the xS–fitted-response relationship to be assessed.In the BHD, age is positively associated with predicted home value when average rooms exceed the median and negatively associated otherwise.
4 Simulations
The simulations show that ICE plots diagnose additivity, expose interaction effects and their regions, and reveal extrapolations that PDPs can obscure.
- Additivity Assessment: ICE curves with a common shape and level shifts diagnose an additive fitted model, while differing shapes indicate interactions.For an additive model, differences between curves depend only on remaining features and not on the feature being varied.
- Additivity Assessment: In the GAM example, ICE curves for X1 share a parabolic shape with constant shifts, and d-ICE shows little variation.The fitted interaction term contributes relatively little, whereas additive structure cannot be observed from the PDP alone.
- Finding interactions and regions of interactions: Non-parallel ICE curves reveal that X3 modulates fitted values through interactions with X1 and X2, despite the PDP showing no association with X3.This demonstrates how averaging can obscure heterogeneous fitted relationships.
- Finding interactions and regions of interactions: The d-ICE plot localizes the fitted interactions to X3 ∈[−0.025, 0.025], where derivative lines are highly variable and their standard deviation peaks near X3 ≈0.This interval is called the region of interaction (ROI).
- Extrapolation Detection: ICE plots expose extrapolations in sparsely populated feature-space regions, including an RF model’s extrapolated polynomial relationship where both x1 and x2 are positive.Observed training points are marked on each curve, distinguishing interpolation from predictions outside the observed data region.
- Extrapolation Detection: Whether extrapolated behavior is desirable depends on the application and the character of the extrapolations, which can differ across algorithms.ICE plots help users inspect these behaviors when choosing a suitable model.
5 Real Data
Across three real-data examples, ICE plots expose heterogeneous fitted relationships that average PDPs can obscure. The examples show treatment-by-marital-status interactions, alcohol-dependent pH effects, and heterogeneous diabetes-risk responses to skin thickness.
- Depression Clinical Trial: In the depression trial, BART achieved an in-sample R2 ≈0.40 after modeling 37 covariates and treatment for 156 active-treatment subjects.The analysis compared cognitive therapy and paroxetine while excluding placebo subjects.
- Depression Clinical Trial: A flat PDP masked treatment effects ranging from -3 to +3 Hamilton points, with married subjects predicted to benefit more from cognitive therapy and unmarried subjects from paroxetine.Crossing ICE curves and the c-ICE plot indicated treatment-by-marital-status interactions.
- White Wine: For white wine, ICE plots showed increasing predicted scores with pH for high-alcohol wines but mostly decreasing, nonlinear associations for low-alcohol wines.The increasing region for low-alcohol wines contained no data points and was interpreted as likely extrapolation; the cumulative difference was roughly 40% of the response range.
- White Wine: Derivative plots supported alcohol-dependent interactions in the wine model, with mostly positive derivatives for high-alcohol wines and mostly negative derivatives for low-alcohol wines.Derivative standard deviation decreased as pH increased, suggesting less prevalent interactions at higher pH.
- Diabetes Classification in Pima Indians: In the diabetes model, ICE plots showed that comparable skin thicknesses could produce either flat or substantially increasing predicted centered log odds, with greatest heterogeneity slightly above 30.The random forest achieved an out-of-bag misclassification rate of 22%, and younger subjects’ high-thickness effects were identified as extrapolation.
6 A Visual Test for Additivity
The section develops a visual lineup test for whether interactions suggested by ICE plots could arise under an additive data-generating model. It applies the test to depression and white-wine data, finding strong evidence of non-additivity for treatment but only modest evidence for pH.
- Motivation: The test asks whether observed ICE heterogeneity reflects interactions in the data-generating model rather than only features of the fitted model.Researchers may instead treat the fitted model as the object of interest for future predictions.
- Procedure: The observed ICE plot is randomly inserted among K−1 null plots, and correctly identifying it yields a valid discovery at α = 1/K conditional on null-generation procedures.Non-parallel ICE curves are the visual feature used to suggest non-additivity.
- Procedure: Under the additive null, the procedure estimates component functions by backfitting, resamples residuals, refits the original learning algorithm, and generates null ICE plots.The resampled response preserves an additive conditional expectation while allowing the refitted model to be potentially non-additive.
- Examples: In the white-wine data, the 20-image lineup provides only modest evidence that pH interacts with the remaining predictors in the underlying model.The true plot could be confused with another extreme-looking plot in the lineup.
7 Discussion
The discussion presents ICE as a model-agnostic extension of PDPs that exposes individual conditional relationships and supports formal visual testing. It also identifies un explored applications and future extensions of the testing methodology.
- Discussion: ICE visualizes individual conditional relationships that PDP averages can mask when partial effects vary with other variables.This helps researchers inspect how fitted black-box models use covariates to generate predictions.
- Discussion: An ICE plot displays the distribution of individual conditional expectation functions for a variable xS.Parallel, coincident, or differently shaped curves correspond to distinct structures in the fitted model.
- Discussion: ICE curves can reveal additivity, interactions, and extrapolations in the fitted model, with centered and derivative variants sometimes making these properties easier to distinguish.The suite provides a view of the fitted map from predictors to predicted values.
- Discussion: The work leaves out-of-sample ICE, multi-algorithm comparisons, and ICE-curve clustering for subsequent research.These applications are described as possible uses rather than evaluated results of this work.
- Discussion: The visual testing methodology compares the observed ICE plot with plots generated from additive null models to distinguish interaction effects from noise at a known significance level.Future work is proposed for testing other null hypotheses.
Supplementary Materials
The supplementary materials describe software and replication resources while noting that the depression trial data cannot be released because of privacy concerns.
- Supplementary Materials: The Section 3 procedures are implemented in the R package ICEbox, and the supplementary script can reproduce the paper’s simulated results, tables, and figures.The depression data are unavailable for release due to privacy concerns.
A Algorithms
The ICE algorithm constructs one conditional prediction curve per observation by holding that observation’s complementary predictors fixed and varying the selected predictors across observed values.
- Algorithm: Given feature matrix X, fitted model ˆf, and predictor subset S, the algorithm returns estimated partial-dependence curves for constant values of xC.Here C is the complement of S.
- Algorithm: For each observation i, ICE fixes xC at that observation’s complementary-feature values and initializes its curve.This preserves the observation-specific conditioning context.
- Algorithm: The algorithm varies xS over the values observed for each row ℓ and evaluates ˆf([xS, xC]) to fill the ith curve.Each curve therefore records fitted responses as the selected predictors vary while xC remains fixed.
- Algorithm: After processing all observations, the algorithm returns the collection of N estimated conditional curves.The returned curves are the individual objects used for ICE visualization.
S ] 11: end function
The d-ICE algorithm computes numerical derivatives of estimated partial dependence functions for each observation and returns those derivatives for the ICE plot. Its implementation smooths the ICE plot before estimating derivatives.
- The d-ICE algorithm takes an N × p feature matrix, estimated partial dependence functions for subset S, and a numerical-derivative function D as inputs.
- In the implementation, D first smooths the ICE plot with the “supersmoother” and then estimates derivatives from the smoothed plot.
- For each observation i, the algorithm initializes an N × 1 derivative vector and retains that observation’s feature values in columns C.
- It iterates over observations ℓ, evaluates the numerical partial derivative at the estimated function for observation i with respect to x_S, and stores the results.
- The algorithm returns the derivatives d f̂_S for all N observations.