Source-linked AI summary
The What-If Tool: Interactive Probing of Machine Learning Models
James Wexler, Mahima Pushkarna, Tolga Bolukbasi, Martin Wattenberg, Fernanda Viegas, Jimbo Wilson
TL;DR
ML practitioners need ways to understand model performance across varied inputs, including hypothetical changes, feature effects, subgroup behavior, and fairness. This paper presents the What-If Tool, an open-source interactive system for probing models with minimal coding, and reports real-world use uncovering difficult-to-see issues and insights.
Problem
ML practitioners need to inspect performance across inputs, understand feature and subgroup effects, and investigate fairness and hypothetical changes.
Method
The What-If Tool is an open-source, model-agnostic visual interface that analyzes model inputs and outputs, supports hypothetical exploration, and provides fairness metrics and threshold optimization.
Results
Real-world usage showed WIT uncovering nontrivial, long-lived model bugs and providing practitioners with new insights into their models.
Takeaways & Limitations
Interactive exploration of hypotheticals, data subsets, and fairness metrics helps practitioners understand ML systems across real and hypothetical scenarios.
Takeaways & Limitations
WIT’s practical audience is constrained by the need for a trained model and dataset, while users requested more automation and customizable metrics and fairness constraints.
Abstract
from arXiv · showhide
A key challenge in developing and deploying Machine Learning (ML) systems is understanding their performance across a wide range of inputs. To address this challenge, we created the What-If Tool, an open-source application that allows practitioners to probe, visualize, and analyze ML systems, with minimal coding. The What-If Tool lets practitioners test performance in hypothetical situations, analyze the importance of different data features, and visualize model behavior across multiple models and subsets of input data. It also lets practitioners measure systems according to multiple ML fairness metrics. We describe the design of the tool, and report on real-life usage at different organizations.
1 INTRODUCTION
The What-If Tool addresses ML model-understanding needs by enabling interactive analysis of predictions, data subsets, features, and hypothetical scenarios. It supports local and global analysis across data and model types, including fairness investigations.
- Tool overview: The What-If Tool is an open-source, model-agnostic visual tool for understanding ML models, requiring only trained models and a sample dataset.It runs through TensorBoard and is also available in Jupyter and Colaboratory notebooks.
- Interactive exploration: Users can perform counterfactual reasoning, investigate decision boundaries, and explore how changes to data points affect predictions through iterative visual exploration.The interface supports switching between different views of input data and model performance.
- Fairness analysis: Intersectional analysis slices data by combinations of features, supporting investigations of model fairness across subgroups.The authors motivate this capability as important for examining fairness issues.
- Analysis scope: WIT supports both local analysis of individual decisions and global analysis of model behavior across an entire dataset, along with varied data and model types.
- Reported usage: Real-world users discovered surprising facts about ML systems, suggesting that exploring hypotheticals can help reveal model behavior.
2 RELATED WORK
WIT is a general black-box model-understanding and visualization tool that emphasizes hypothetical testing, intersectional analysis, and fairness. Its flexibility distinguishes it from specialized or preset alternatives.
- Model understanding frameworks: WIT probes only model inputs and outputs rather than relying on model internals, making it broadly applicable despite that constraint.
- Model understanding frameworks: WIT provides a configurable visualization framework that can reproduce displays like Manifold’s while supporting additional arrangements.
- Model understanding frameworks: Compared with ModelTracker, WIT places stronger emphasis on hypothetical outcomes, intersectional analysis, and fairness, while also supporting standalone use with third-party systems.
- Model understanding frameworks: Unlike Prospector’s reliance on orthogonal features for single-class models, WIT supports intersectional analysis of multiple potentially correlated or confounded features.
- Model understanding frameworks: Unlike GAMut and iForest, which target generalized additive models and random forests respectively, WIT is not restricted to those model classes.
- Model understanding frameworks: WIT calculates fairness metrics and can interactively optimize posttraining classification thresholds to improve those metrics.
- Flexible visualization platform: Facets Dive creates custom views of input data and model results, emphasizing intersections among multiple attributes within WIT.
- Flexible visualization platform: Facets Dive uses local, in-memory storage and calculation, protecting access-restricted data while enabling smooth exploration.The design incurs a stated cost, though the supplied passage does not specify it.
3 BACKGROUND AND OVERALL DESIGN
WIT was shaped through iterative user studies around five needs: low-code hypothesis testing, visual model understanding, black-box hypotheticals, intersectional analysis, and multi-model improvement evaluation.
- Design process: The tool’s intended audience became more technical than initially planned because using it requires a trained model and dataset.The authors restricted practical use to people with some existing ML experience.
- Design process: WIT was developed through a proof of concept and 15 months of internal and external studies that informed its design.The internal study involved four teams and eight company participants using TensorFlow models and TensorBoard.
- User needs: N1 requires graphical, minimal-code interaction with trained models for rapid exploration of multiple hypotheses.
- User needs: N2 treats visualizations as a medium for generating instance-, feature-, and subgroup-level explanations of model behavior.
- User needs: N3 requires testing hypothetical perturbations without access to model internals by treating models as black boxes.
- User needs: N4 calls for exploratory intersectional analysis of model performance across subsets defined by multiple attributes.
- User needs: N5 supports evaluating performance-improvement strategies across multiple models before changing training data or hyperparameters.
- Deployment: WIT is available in TensorBoard and as a standalone Jupyter or Colaboratory notebook extension, with support for TensorFlow and user-provided prediction functions.
4 TASK-BASED FUNCTIONALITY
The paper explains WIT’s task-based functionality using the UCI Census dataset and two income-classification models. The example examines model disagreement and the influence of capital gain.
- Running example: The task-based section uses the UCI Census dataset as a running example for describing and contextualizing WIT’s features.
- Running example: The example compares a multilayer neural network and a simple linear classifier on 500 UCI test-set data points.Both models classify individuals as high or low income.
- Running example: The analysis focuses on a data point where the two models disagree and examines how the capital-gain feature influences their predictions differently.
4.1 Exploring Your Data
WIT provides visual tools for exploring datasets, model results, and performance across feature-defined subsets. Its Features and Datapoint Editor views support customizable visualizations, slicing, and summary statistics for investigating model behavior.
- Customizable Analysis: WIT’s Datapoint Editor visualizes loaded data points and model inference values, with configurable encodings, zooming, panning, and feature-based slicing.Users can bin, position, color, and label points by feature values; small multiples can be created by binning one feature or an intersection of two features.
- Customizable Analysis: WIT supports performance-oriented visualizations including confusion matrices, feature histograms, two-dimensional charts, and prediction-error plots.Datapoints can be colored by correctness, classification, or demographic attributes depending on the visualization.
- Customizable Analysis: Users can compare two models by plotting their positive-class scores, with distance from the diagonal indicating model disagreement.In the Census example, the linear classifier and neural network scores are plotted on the Y- and X-axes, respectively.
- Features Analysis: Dataset Summary Statistics: The Features tab provides summary statistics and distribution charts for every loaded feature, helping users validate model-performance explanations on data subsets.Numeric features show minimum, maximum, mean, standard deviation, and a 10-bin equal-width histogram; categorical features show unique-value and frequency statistics.
- Features Analysis: Dataset Summary Statistics: Features can be sorted by non-uniformity, zeros, or missing values to expose imbalanced distributions; in the Census data, 90% of capital-gain values are zero while some reach 100,000.The distribution helps explain why the neural network essentially ignores zero capital-gain values.
4.2 Investigating What-If Hypotheses
WIT supports what-if analysis by letting users alter data points, identify similar counterfactuals, and inspect how predictions vary across feature values. These tools expose both local prediction changes and broader model relationships, including differences between models.
- Counterfactual Reasoning: Users can edit, add, or delete feature values and re-run inference to observe how those changes affect a datapoint’s prediction.Edited points update model scores, deltas, and change directions; users can also duplicate or delete points to compare multiple modifications.
- Counterfactual Reasoning: WIT identifies counterfactual examples by ranking datapoints with an aggregate feature-distance measure using standardized numeric differences, categorical-value distances, and a user-selected L1 or L2 norm.The default is L1 distance, while L2 is available for additional exploration.
- Counterfactual Reasoning: Changing capital-gain from 3,411 to 20,000 increased the score for label 1 (≥50K) from 0.336 to 0.991, essentially flipping the prediction.The example demonstrates a local what-if intervention on the Census datapoint.
- Partial Dependence Plots: Partial dependence plots show prediction scores as a selected feature is varied across a range, using line charts for numeric features and column charts for categorical features.Plots can show multiple models together, original feature values, and classification thresholds; global plots average inference results across the dataset.
- Partial Dependence Plots: In the Census example, the linear model’s capital-gain relationship is monotonic, whereas the neural network treats zero and high capital gains as indicative of high income rather than small gains.This relationship is consistent with the observed counterfactual behavior.
4.3 Evaluating Performance and Fairness
WIT evaluates aggregate and sliced model performance, supports intersectional comparisons, and lets users adjust thresholds to explore fairness objectives. In the Census example, demographic-parity optimization lowers female and raises male thresholds to address unequal positive-class representation.
- Performance slicing: WIT reports performance measures for complete datasets and feature-defined subgroups, including intersections of two features.Users can sort slices by count, alphabetically, or task-appropriate performance measures.
- Performance measures: For binary classification, ROC curves show true- and false-positive rates across thresholds, while confusion matrices summarize each model’s errors.Error counts are emphasized through color opacity.
- Threshold selection: WIT automatically selects positive-class thresholds using user-specified cost ratios and loaded-dataset inference results.With the default ratio of 1.0, it selects the threshold achieving highest accuracy.
- Census example: At default thresholds of 0.5, Census accuracy is higher for women, while predicted positive-class rates differ substantially between men and women.The ground-truth positive-class proportions also differ between the two sexes.
- Fairness optimization: Fairness optimization independently changes slice thresholds so performance metrics and visualizations update for the selected fairness definition.For demographic parity in the Census example, the male threshold rises and the female threshold falls.
4.4 Comparing Two Models
WIT supports comparing two models through shared performance measures and visualizations, helping practitioners inspect differing predictions and feature sensitivities. This comparison supports iterative model development after changes to datasets or models.
- Model comparison: All performance measures and visualizations, including ROC curves and partial dependence plots, support direct comparison of two models.The tool also plots inference-score differences and lets users select discrepant data points for investigation.
- Model comparison: Partial dependence plots can compare whether a new model is more or less sensitive to a feature than an older model.
4.5 Data Scaling
WIT’s capacity depends mainly on feature count and feature size. On a standard recent laptop, it handles substantially more tabular points than small images.
- Capacity: ∼100k tabular points with 10–100 numeric or string inputs can be loaded, compared with ∼2000 small 78x64-pixel images.Smaller image sizes allow more images to be loaded.
- Capacity: Figure 6’s performance view displays two Census models broken down by sex using confusion matrices and threshold settings.It contrasts default 0.5 thresholds with thresholds optimized for demographic parity.
5 CASE STUDIES
Three case studies show WIT used to inspect production regressions, compare model versions, and investigate police stop-and-frisk predictions. The studies uncovered data-pipeline bugs, influential features, and group disparities.
- 5.1 Model debugging: A researcher found a serving-time feature-ingestion bug after a flat partial dependence plot revealed that one feature affected no predictions.WIT helped uncover and fix training-serving skew that had not previously been detected.
- 5.2 Model comparison: A software engineer discovered that a feature-building bug swapped measurement values, causing the first health-metric model to solve a different problem.Comparing two models exposed prediction and target-value patterns that ordinary error metrics had not revealed.
- 5.3 Counterfactual analysis: Students used counterfactual analysis to discover that changing officer ID could flip many stop-and-frisk classifications.The nearest counterfactual often differed only in officer ID.
- 5.3 Counterfactual analysis: The students found officer identity more salient than anticipated, alongside age, gender, and race, for predicting whether a person was frisked.
- 5.3 Fairness analysis: Equality-of-opportunity analysis indicated that White people had the most advantage, while Asian and Black people faced significant and slight disadvantage, respectively.The students adjusted thresholds by slice to examine alternative fairness constraints.
6 LESSONS FROM AN ITERATIVE DESIGN PROCESS
Over 15 months, user feedback expanded WIT from a simple no-code hypothetical-editing workflow into three workflows for hypothetical testing, data sense-making, and performance and fairness evaluation.
- Over 15 months, the team added interaction capabilities and refined WIT to support common analysis tasks beyond editing datapoints and rerunning predictions.The initial design centered on a single no-code user journey for editing a datapoint and rerunning prediction.
- WIT’s three core workflows cover hypothetical scenario testing, general data sense-making, and performance and fairness evaluation.The latter two workflows were added alongside the original hypothetical-testing workflow.
- In-application documentation addresses users’ first encounters with components such as partial dependence plots and concepts such as counterfactuals.
- Users’ feedback led the designers to reduce certain interactivity in the Fairness + Performance view rather than maintain tightly linked threshold and visualization controls.Initially, changing a classification threshold immediately rearranged items in the visualizations.
7 CONCLUSION AND DIRECTIONS FOR FUTURE RESEARCH
WIT provides no-code graphical analysis of ML performance, hypothetical scenarios, data subsets, and fairness metrics. Real-world use found value for novices and experienced engineers, while future work targets model-internal information and lower expertise requirements.
- WIT lets practitioners analyze ML performance on real and hypothetical data, inspect input and model-result overviews, perform intersectional analysis, and assess or optimize fairness metrics without coding.
- Real-world usage found WIT valuable to both ML novices and experienced engineers, helping expose otherwise hard-to-see model issues.Practical use uncovered nontrivial, long-lived bugs and generated new model insights.
- A future direction is incorporating available information about ML model internals, including gradients for differentiable models and TCAV for feedforward neural networks.
- Another future direction is reducing the ML and data-science expertise needed to use WIT, including automating outlier and underperforming-subset discovery and allowing custom metrics and fairness constraints.These changes are described as potentially broadening stakeholder participation in model-understanding and fairness efforts.