PaperlayerView exact revision on arXiv

Source-linked AI summary

Manipulating and Measuring Model Interpretability

Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M. Hofman, Jennifer Wortman Vaughan, Hanna Wallach

arXiv:1802.07810v5cs.AIcs.CY

TL;DR

Interpretable models are intended to help people use predictions and detect model mistakes, but experimental evidence for these effects is limited. Across preregistered experiments, we varied feature count and model transparency and found that clear, low-feature models were easier to simulate but did not increase reliance and reduced mistake detection.

  • Problem

    Experimental evidence is limited on whether supposedly interpretable models actually improve people’s use of model predictions and ability to detect mistakes.

    Source: §1 · pp. 1–2

  • Method

    In preregistered experiments (N=3,800), participants evaluated functionally identical models varying in feature count and transparency across simulation, reliance, and mistake-detection outcomes.

    Source: §1 · p. 3, §1.2 · p. 4, §3 · p. 7

  • Results

    Clear models with few features were easier to simulate, but did not increase reliance and hampered participants’ ability to detect sizable model mistakes.

    Source: §1 · p. 3, §8 · p. 25

  • Takeaways & Limitations

    These findings emphasize testing interpretable models rather than relying on intuition, because additional transparency may hinder mistake detection.

    Source: §1 · p. 3, §8 · p. 25

    The experiments studied laypeople using linear regression for real-estate valuation, so findings may differ for other stakeholders, models, tasks, or domains.

    Source: §7 · p. 24

Abstract

from arXiv · show

With machine learning models being increasingly used to aid decision making even in high-stakes domains, there has been a growing interest in developing interpretable models. Although many supposedly interpretable models have been proposed, there have been relatively few experimental studies investigating whether these models achieve their intended effects, such as making people more closely follow a model's predictions when it is beneficial for them to do so or enabling them to detect when a model has made a mistake. We present a sequence of pre-registered experiments (N=3,800) in which we showed participants functionally identical models that varied only in two factors commonly thought to make machine learning models more or less interpretable: the number of features and the transparency of the model (i.e., whether the model internals are clear or black box). Predictably, participants who saw a clear model with few features could better simulate the model's predictions. However, we did not find that participants more closely followed its predictions. Furthermore, showing participants a clear model meant that they were less able to detect and correct for the model's sizable mistakes, seemingly due to information overload. These counterintuitive findings emphasize the importance of testing over intuition when developing interpretable models.

1 INTRODUCTION

The paper treats interpretability as a latent human property that must be studied by manipulating model factors and measuring behavioral outcomes. Across pre-registered experiments, clear models with few features improved simulation but did not improve reliance and sometimes impaired error detection, emphasizing empirical testing over intuition.

1 INTRODUCTION

  • Machine-learning systems increasingly participate in decisions across high-stakes and everyday domains, creating collaborations in which people may follow or override model predictions.Following models can improve accuracy, speed, and consistency, but can be harmful when predictions are incorrect.Source: §1 · pp. 1–2
  • Interpretability is framed as a latent human property influenced by manipulable factors and revealed through outcomes such as simulation, reliance, and mistake detection.The authors argue that interpretability should be defined by people’s behavior rather than intuition.Source: §1 · pp. 2–3
  • Clear models with few features improved participants’ simulation of model predictions, but did not increase beneficial reliance and reduced detection of sizable model mistakes.These findings came from highly powered, pre-registered experiments with representative stimuli and contradicted common intuition about interpretability.Source: §1 · p. 3

1.1 Domain

  • The experiments used real-estate valuation with lay participants and held model predictions constant while varying only whether the presentation showed two or eight features and clear or black-box internals.All participants saw the same apartments, feature values, and predicted prices, allowing behavioral differences to be attributed to model presentation.Source: §1.1 · pp. 3–4

1.2 Overview of experiments

  • The first experiment replicated improved simulation for a clear two-feature model but found no greater beneficial reliance and poorer detection of mistakes for clear than black-box models.Participants were especially less able to detect and correct mistakes on apartments with unusual configurations, prompting three additional experiments.Source: §1.2 · p. 4
  • Scaled-down prices replicated the first experiment, while an alternative weight-of-advice measure again found no condition difference in beneficial reliance but did not reproduce the clear-model disadvantage in mistake detection.The differing error-detection result motivated explanations involving anchoring, simulation, and information overload.Source: §1.2 · pp. 4–5
  • In the final experiment, an outlier-focus message eliminated the clear-model disadvantage in mistake detection, supporting information overload as an explanation for the earlier effect.Post-hoc analyses similarly found that clear eight-feature presentations worsened simulation, reliance, and prediction accuracy.Source: §1.2 · p. 5
  • Overall, the authors conclude that commonly presumed interpretability factors often have negligible or detrimental behavioral effects, so interpretable models should be tested rather than evaluated by intuition.The experiments comprised several thousand participants and examined multiple behavioral outcomes.Source: §1.2 · p. 5

2 RELATED WORK

Prior work has examined interpretability through controlled user studies, human–computer interaction research on mental models and sensemaking, and decision-making research on trust in computational aids. This study extends that literature by testing how presentation differences in functionally identical models affect people’s behavior.

Machine learning user studies

  • Controlled studies have linked model complexity and explanation types to people’s simulation abilities, understanding, trust, fairness perceptions, and identification of model irregularities.Lage et al. found that cognitive chunks and model size affect simulatability, while other studies examined specific interpretability tools and explanatory interfaces [11] [29] [89].Source: §2 · p. 6

Mental models and intelligibility

  • Human–computer interaction research treats users as active participants who form mental models, with transparency, interaction, and explanation completeness shaping trust and understanding.Kulesza et al. found that completeness matters more than soundness for shaping mental models, although very low soundness reduces trust.Source: §2 · p. 6

Sensemaking

  • Sensemaking research frames model understanding as organizing information to build situation awareness, while information overload may interfere with examining predictions and correcting model mistakes.The experiments do not directly measure cognitive processes but test a hypothesis about interference in information intake.Source: §2 · pp. 6–7

Decision-making

  • Decision-making research has studied trust in computational aids and simple linear models, but has rarely tested how people use such models; this work compares presentation differences in functionally identical models.The tested differences concern two factors thought to make machine learning models more interpretable.Source: §2 · p. 7

3 EXPERIMENT 1: PREDICTING APARTMENT SELLING PRICES

Experiment 1 found that a clear, two-feature model improved participants’ simulation of predictions but not their beneficial adherence, and reduced their ability to correct sizable mistakes.

3 EXPERIMENT 1: PREDICTING APARTMENT SELLING PRICES

  • The experiment measured simulation error, deviation from model predictions, and detection of sizable mistakes using typical and unusual apartment configurations.Mixed-effects models accounted for repeated responses from each participant.Source: §3 · pp. 7–8, §3.2 · p. 12

3.1 Experimental design

  • The study used a controlled 2 × 2 design varying two versus eight features and clear versus black-box presentation, plus a no-model baseline, while holding model predictions and near-identical accuracies constant.Participants were randomly assigned across five conditions; 1,250 U.S. participants completed training and testing on apartment prices.Source: Figure 1 · p. 8, §3.1 · p. 10

3.2 Findings

  • Clear two-feature models reduced simulation error (t(994) = −12.06, p < 0.001), but did not increase beneficial adherence (t(994) = 0.67, p = 0.5); clear models also worsened correction of apartment 12’s mistake (F(1, 994) = 8.81, p = 0.003).Participants deviated less from clear-model predictions than black-box predictions on the unusual apartment 12, contrary to the preregistered intuition.Source: §3.2 · pp. 12, 14, Figure 3 · p. 13, Figure 5 · p. 14
  • Using a model improved accuracy for typical apartments but hurt accuracy on apartment 11, while clear models made apartment-11 predictions less accurate than black-box models (F(1, 994) = 31.98, p < 0.001).Participants’ prediction errors did not differ significantly across the four primary conditions overall (F(3, 994) = 2.43, p = 0.06).Source: §3.2 · pp. 14–15, Figure 6 · p. 15

4 EXPERIMENT 2: REPRESENTATIVE U.S. PRICES

Experiment 2 replicated the first experiment using representative U.S. apartment prices, indicating that New York City’s unusually high prices did not drive participants’ behavior. Clear, two-feature models improved simulation but not beneficial reliance and impaired mistake detection on unusual data points.

4.1 Experimental design

  • Experiment 2 scaled prices and maintenance fees down by tenfold, adjusted all coefficients except maintenance fee, and recruited 750 new screened participants across five conditions.The remaining experimental design was unchanged from Experiment 1, with condition sizes ranging from 147 to 152 participants.Source: §4.1 · p. 16, §4 · p. 16

4.2 Findings

  • Participants assigned to the clear, two-feature model had lower simulation errors than other primary conditions, t(594) = −10.41, p< 0.001.This replicated the first experiment’s simulation result.Source: §4.2 · p. 16
  • For apartment 12, clear-model participants followed the model more closely than black-box participants, indicating reduced ability to detect and correct the model’s overly high prediction.The difference was significant, t(594) = −4.16, p< 0.001; for apartment 11, clear-versus-black-box differences were not significant, t(594) = −1.82, p= 0.069.Source: §4.2 · pp. 16–17
  • Experiment 2 closely replicated Experiment 1: clear, two-feature models improved simulation, did not increase beneficial reliance, and impaired detection of sizable mistakes on unusual data points.Simulation errors were lower than in other primary conditions (t(594) = −10.41, p< 0.001), while beneficial reliance did not differ from black-box, eight-feature models (t(594) = 0.49, p= 0.626). For apartment 12, clear-model participants followed an overly high prediction more closely than black-box participants, resulting in worse final predictions (t(594) = −4.16, p< 0.001).Source: §4.2 · pp. 16–17

5 EXPERIMENT 3: WEIGHT OF ADVICE

Experiment 3 used weight of advice to reassess whether participants followed clear, two-feature model predictions more closely than black-box, eight-feature predictions. It found no such difference, no greater reliance on a labeled human expert, and no clear-model disadvantage in detecting or correcting overly high predictions.

5.1 Experimental design

  • The experiment used the original New York City prices, four prior model conditions, and a new black-box, eight-feature condition labeled “Human Expert,” with initial and final predictions enabling weight-of-advice measurement.Participants made initial predictions for twelve apartments before seeing the model or expert prediction, then updated those predictions; 1,000 new participants were recruited, with one response later excluded.Source: §5.1 · pp. 18–20

5.2 Findings

  • Participants did not follow clear, two-feature model predictions significantly more closely than black-box, eight-feature predictions, measured by weight of advice or absolute prediction difference.This replicated the first two experiments’ null result using an alternative advice-taking measure.Source: §5 · p. 18, §5.2 · p. 20
  • Participants followed predictions labeled as coming from a “human expert” no more closely than predictions from black-box models.The authors suspect increasing experience with the model or expert during the experiment may explain the difference from Logg’s findings.Source: §5.2 · p. 20
  • Unlike in the first two experiments, participants shown clear models were not less able to detect and correct overly high predictions for apartments 11 or 12.This result motivated the final experiment.Source: §5.2 · p. 20

6 EXPERIMENT 4: OUTLIER FOCUS AND DETECTION OF MISTAKES

Experiment 4 found that an outlier-focus message improved participants’ detection and correction of model mistakes, eliminating the clear-versus-black-box difference observed without the message. These results support information overload as an explanation and suggest a simple mitigation for clear models’ unintended disadvantage.

6 EXPERIMENT 4: OUTLIER FOCUS AND DETECTION OF MISTAKES

  • Experiment 4 used a 2 × 2 design manipulating model transparency and an outlier-focus message, while removing potential anchoring by not asking participants to guess the model’s predictions.The study recruited 800 new participants and used only the two-feature linear regression model.Source: §6 · p. 21, §6.1 · pp. 22–23

6.2 Findings

  • Outlier-focus messages increased participants’ deviations from the model’s predictions for both unusual apartments, improving detection and correction of the model’s sizable mistakes.Apartment 6: t(791) = −4.72, p < 0.001; apartment 8: t(795) = −5.00, p < 0.001.Source: §6.2 · p. 23
  • Without an outlier-focus message, participants shown the clear model deviated less from its predictions than those shown the black-box model for both unusual apartments.Apartment 6: t(393) = −3.65, p < 0.001; apartment 8: t(395) = −3.51, p < 0.001.Source: §6.2 · p. 23
  • With an outlier-focus message, clear- and black-box-model participants showed no significant difference in deviations, suggesting the message counteracted information overload.Apartment 6: t(401) = −0.004, p = 0.996; apartment 8: t(394) = −1.64, p = 0.101.Source: §6.2 · pp. 23–24
  • Post-hoc analyses found higher prediction errors for the clear, eight-feature model in Experiments 1 and 2, but the maximum pairwise differences were small relative to average selling prices.Experiment 1: t(994) = 2.68, p = 0.007, with a maximum difference of about $16,000; Experiment 2: t(594) = 4.78, p < 0.001.Source: §6.2 · p. 24

7 LIMITATIONS

The study’s conclusions are limited by its narrow stakeholder, model, and domain scope, controlled model comparisons, potentially influential design features, and lack of process or longitudinal measures.

Scope limitations

  • The experiments studied laypeople using linear regression for real-estate valuation, so findings may differ for other stakeholders, tasks, models, or domains.Suggested extensions include experts, classification, decision trees, rule lists, deep neural networks, medical diagnosis, credit risk, sentencing, bail, and hiring.Source: §7 · p. 24

Model comparisons

  • Because the first three experiments equated predictions across two- and eight-feature models, they excluded settings where complex deep models outperform simpler ones and may not generalize to such domains.The authors avoided complex models to prevent confounding presentation, model fidelity, and feature count, while arguing that evidence should outweigh intuition about interpretability.Source: §7 · p. 25

Potential confounds

  • Unmeasured model and feature characteristics may have influenced behavior, including feature intuitiveness, participants’ extra information, and confusion about the eight-feature model’s negative total-rooms coefficient.Participants may have viewed the two-feature model differently if told that the omitted features would not improve accuracy, while the eight-feature model may have appeared questionable because of its negative coefficient.Source: §7 · p. 25

Measurement limitations

  • The absence of process measures and the experiments’ short, one-shot design limited conclusions about participants’ cognitive and sensemaking processes.For example, deviations from the model on unusual apartments could show mistake correction without revealing whether participants understood why the mistakes occurred; qualitative and longitudinal studies could address this gap.Source: §7 · p. 25

8 DISCUSSION AND CONCLUSION

The experiments challenge the assumption that clear models with few features improve reliance or error detection, while suggesting information-sensitive presentation strategies and emphasizing behavioral testing of interpretability.

Main findings

  • Clear models with few features did not significantly increase participants’ adherence to model predictions, even though simply following those predictions would have reduced their prediction errors.This challenges a commonly assumed behavioral benefit of interpretability.Source: §8 · p. 25
  • Clear models hampered participants’ detection of sizable model mistakes, seemingly because information overload impaired attention to the model’s details.The behavior disappeared under an outlier-focus message, supporting the information-overload explanation.Source: §8 · p. 25
  • In the highest-information condition—the clear, eight-feature model—participants were worst at simulating predictions, following the model, and accurately predicting apartment selling prices.This pattern appeared in the first two experiments relative to the other primary conditions.Source: §8 · p. 26

Implications

  • Potential improvements include alerting users to outliers, eliciting their predictions before revealing the model, and hiding model internals until requested.These suggestions are intended to reduce information overload and encourage careful inspection; eliciting predictions with feedback has been linked to better memory and comprehension.Source: §8 · p. 26

Conclusion

  • Because interpretability has multiple possible goals, whether a model is interpretable should be assessed through people’s behavior and testing rather than intuition [65].The paper argues this is increasingly important as people make more decisions in collaboration with machine learning models.Source: §8 · p. 26

Implications

  • Clear models with few features may still support simulation, ethical scrutiny, debugging, and possible adoption, so feature count and transparency should not be dismissed.The experiments found better simulation of clear models with few features, while other benefits were proposed rather than directly tested.Source: §8 · p. 26

APPENDICES … Appendix C APARTMENT SELECTION DETAILS

The appendices document the study domains, participant instructions, and construction of the apartment stimuli. They also describe settings in which users possess information unavailable to the models.

Domain

  • Decision-aid examples span mammography, hospital readmission, wildfire risk, bail decisions, venture success, and apartment selection.The listed models use between 3 and 29 features, depending on the domain.Source: Domain · p. 32

Appendix A SCENARIOS WHERE USERS HAVE ACCESS TO MORE INFORMATION THAN MODELS

  • Users in these scenarios may have access to information unavailable to the models, including full records, interviews, local knowledge, or continuous measurements.Examples include full mammograms and medical records, experience of past fires, physical walkthroughs, and neighborhood knowledge.Source: Domain · p. 32

APPENDICES

  • The appendices provide supporting materials on decision domains, experimental instructions, and apartment-stimulus construction.These materials cover scenarios involving additional user information, the clear-2 instructions, and apartment selection details.Source: Table 1 · p. 33, Appendix B · p. 34, Appendix C · p. 48

Appendix B INSTRUCTIONS FROM THE CLEAR-2 CONDITION IN EXPERIMENT 1

  • The Experiment 1 clear-2 instructions asked participants to predict New York City Upper West Side apartment prices during training and testing phases.Training used ten apartments with model predictions and actual prices; testing used twelve new apartments without revealed actual prices.Source: Appendix B · pp. 35, 42
  • The clear apartment-price model used # Bathrooms and Square footage, assigning $350,000 per bathroom and subtracting a $260,000 adjustment factor.The instructions explain the prediction as feature contributions summed with the adjustment factor.Source: Appendix B · pp. 37–38
  • During testing, participants separately predicted the model’s output, assessed its confidence, and estimated the apartment’s actual sale price.The three steps elicited confidence in the model prediction and in the participant’s own estimate.Source: Appendix B · pp. 43–45

Appendix C APARTMENT SELECTION DETAILS

  • Apartment stimuli were selected to match a target distribution of model errors while limiting repeated bedroom-and-bathroom configurations.The target errors ranged from -$500K to $300K, and selection continued until each set contained at most three apartments with the same configuration.Source: Appendix C · p. 48, Table 2 · p. 48, Table 4 · p. 49

Appendix D EXPERIMENT 3 HYPOTHESES AND FINDINGS

Experiment 3 preregistered four hypotheses concerning prediction deviation, reliance on advice, differences between human and machine advice, and detection of model mistakes. The hypotheses extended earlier experiments by revisiting deviation and mistake correction.

H7. Deviation

  • Participants’ predictions were hypothesized to deviate less from a clear model with few features than from a black-box model with many features.Source: Appendix D · p. 51

H8. Weight of advice

  • Weight of advice was hypothesized to be higher for participants who saw a clear model with few features than for those who saw a black-box model with many features.Source: Appendix D · p. 51

H9. Humans vs. machines

  • Participants’ deviation and weight-of-advice measures were hypothesized to differ depending on whether predictions came from a black-box model with many features or a human expert.Source: Appendix D · p. 51

H10. Detection of mistakes

  • Participants in different conditions were hypothesized to vary in their ability to correct inaccurate model predictions on unusual examples.Source: Appendix D · p. 51

D.1 Results · Appendix E FULL DISTRIBUTIONS OF PARTICIPANTS’ PREDICTIONS · Appendix F ANOVA TABLES

The results show no significant effects of model transparency or comparison condition on participants’ deviation, advice weighting, or mistake correction. Appendix E provides full prediction distributions across experiments and phases.

D.1 Results

  • Participants’ deviation from the model did not differ significantly between clear-2 and bb-8 (t(798) = −0.87, p= 0.384; Figure 10a).Source: §D.1 · p. 51
  • Participants’ weight of advice did not differ significantly between clear-2 and bb-8 (t(819) = 1.27, p= 0.205; Figure 10b).Weight of advice was calculated only when initial predictions differed from the model’s prediction.Source: §D.1 · pp. 51–52
  • Participants did not deviate less from machine predictions than expert predictions, with no significant differences in deviation or weight of advice.The reported tests were t(994) = 0.45, p= 0.655 for deviation and t(1005) = −0.38, p= 0.704 for weight of advice.Source: §D.1 · p. 52
  • Participants in clear conditions were not significantly less able to correct inaccurate predictions than participants in black-box conditions for apartments 11 and 12.The contrasts yielded t(798) = −0.96, p= 0.337 and t(798) = −0.19, p= 0.847, respectively.Source: §D.1 · p. 52

Appendix E FULL DISTRIBUTIONS OF PARTICIPANTS’ PREDICTIONS

  • Experiment 3’s distributions distinguish participants’ initial predictions from their final predictions after seeing the model’s prediction.Figures 19–20 show mean initial and final predictions, with arrows indicating shifts in the mean predictions.Source: Figure 19 · p. 61, Figure 20 · p. 62

F.1 Experiment 1: Predicting Prices

Experiment 1 reports two-way ANOVAs examining simulation error and deviations between the model’s and participants’ price predictions, including analyses for two individual apartments.

F.1 Experiment 1: Predicting Prices

  • The experiment analyzes simulation error using a two-way ANOVA.Source: Table 6 · p. 65
  • A separate two-way ANOVA examines deviations between the model’s and participants’ predicted prices.Source: Table 7 · p. 65
  • Additional two-way ANOVAs examine prediction deviations for apartment 11 and apartment 12.Source: Table 8 · p. 65, Table 9 · p. 65

F.2 Experiment 2: Scaled-down prices

Experiment 2 analyzed scaled-down-price judgments using two-way ANOVAs of simulation error and deviations between model and participant price predictions, including apartments 11 and 12.

F.2 Experiment 2: Scaled-down prices

  • The experiment tested simulation error with a two-way ANOVA.Source: Table 10 · p. 65
  • A two-way ANOVA assessed deviations between the model’s predicted price and participants’ predicted price.Source: Table 11 · p. 66
  • The analysis separately examined prediction deviations for apartment 11.Source: Table 12 · p. 66
  • The analysis separately examined prediction deviations for apartment 12.Source: Table 13 · p. 66

F.3 Experiment 3: Weight of Advice

Experiment 3 analyzed how participants’ predictions deviated from the model’s predictions and how much weight participants gave the advice across experimental conditions.

Primary conditions

All conditions

Apartment-specific analyses

Related papers

Loading 1802.07810v5…