Source-linked AI summary
"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations
Himabindu Lakkaraju, Osbert Bastani
TL;DR
Black-box explanations are intended to support trust and bias detection, but high fidelity may fail to reveal problematic feature use. The paper develops a framework and generation approach for misleading explanations, then studies their effect on domain experts. It finds that such explanations can manipulate trust, including through correlated features that reconstruct prohibited information.
Problem
The paper addresses limited evidence about whether high-fidelity explanations reveal black-box biases and how misleading explanations affect user trust.
Method
The paper formalizes misleading explanations, extends MUSE to favor desired and omit prohibited features, and evaluates generated explanations with domain experts.
Results
Misleading high-fidelity explanations can manipulate trust, with the user study reporting a 9.8-times increase and trust varying with prohibited-feature presence.
Takeaways & Limitations
Correlations can let explanations reconstruct prohibited features from other features, allowing an untrustworthy black box to appear trustworthy to end users.
Takeaways & Limitations
The framework assumes users primarily judge explanations by included and omitted features, with desired, ambivalent, and prohibited features defined in advance.
Abstract
from arXiv · showhide
As machine learning black boxes are increasingly being deployed in critical domains such as healthcare and criminal justice, there has been a growing emphasis on developing techniques for explaining these black boxes in a human interpretable manner. It has recently become apparent that a high-fidelity explanation of a black box ML model may not accurately reflect the biases in the black box. As a consequence, explanations have the potential to mislead human users into trusting a problematic black box. In this work, we rigorously explore the notion of misleading explanations and how they influence user trust in black-box models. More specifically, we propose a novel theoretical framework for understanding and generating misleading explanations, and carry out a user study with domain experts to demonstrate how these explanations can be used to mislead users. Our work is the first to empirically establish how user trust in black box models can be manipulated via misleading explanations.
1 Introduction
Post hoc explanations are intended to help decision makers understand black boxes and detect errors or biases, but fidelity alone may not reveal problematic model behavior. This paper studies how adversarial explanations can exploit that gap to manipulate user trust.
- Motivation: Complex black-box models motivate post hoc explanations intended to help domain experts detect systematic errors and biases.These models are often difficult to understand despite their use in healthcare and criminal justice.
- Limits of Fidelity: High fidelity guarantees correlation with black-box predictions, not that explanations capture causal relationships, unique feature use, or robustness.The paper identifies multiple explanations with qualitatively different forms and instability under small input perturbations as additional challenges.
- Research Gap: These limitations create opportunities for explanations to mislead decision makers into trusting problematic black boxes, despite limited empirical study of this risk.Prior work had argued that misleading explanations were possible, but the paper identifies little to no empirical research on how they mislead users.
- Contributions: The paper introduces a framework for misleading explanations, an approach that favors desired and omits problematic features, and a domain-expert trust study.The study concerns experts from law and criminal justice and examines confirmation bias in black-box explanations.
- Contributions: 9.8 times greater trust was observed when domain experts saw the misleading explanation rather than the classifier itself.The misleading explanation excludes race and gender while including desired features such as prior jail incarcerations and prior FTA.
2 Problem Formulation
The paper formalizes explanations as interpretable approximations of black-box behavior and defines misleadingness by comparing user trust induced by an explanation with the black box’s actual trustworthiness. It proposes testing both quantities through user studies.
- Explanations: Given inputs, class labels, and a black box, an explanation is an interpretable model intended to describe the black box so users can decide whether to trust it.The explanation approximates the black box over the input distribution.
- Explanations: Explanation quality is measured by relative error, with fidelity defined as 1 − L(E, B).The loss may be the 0-1 loss comparing explanation and black-box predictions.
- Trust and Misleadingness: The user oracle O records whether a user trusts black box B after seeing explanation E.The paper estimates O through a user study that presents explanations and asks participants whether they trust the black box.
- Trust and Misleadingness: An explanation is misleading when user trust given the explanation differs from the black box’s ground-truth trustworthiness.Ground truth is represented by O∗, which indicates whether the black box is trustworthy.
- Construction Procedure: The construction procedure first creates and evaluates an expected-untrustworthy black box, then constructs and evaluates an explanation expected to mislead users.The black box is made interpretable so its trustworthiness can first be assessed directly in a user study.
3 Theoretical Framework
The framework models acceptable explanations and black boxes through desired, ambivalent, and prohibited features, then characterizes when a high-fidelity explanation can still induce mistaken trust. Its central mechanism is reconstructing prohibited features from correlated acceptable ones.
- Feature-Based Trust Model: The feature space is partitioned into desired features users expect included, ambivalent features, and prohibited features users expect omitted.This partition supports the paper’s operational model of acceptable explanations and black boxes.
- Feature-Based Trust Model: An acceptable explanation includes desired features and omits prohibited features, while estimated user trust also requires fidelity at least ϵ+.The analogous acceptability condition is defined for interpretable black boxes.
- Feature-Based Trust Model: The user-study estimates ˆO and ˆO∗ are presented as good approximations of explanation-based trust and black-box trustworthiness.The paper states that its user studies demonstrate the quality of both estimates.
- Existence Results: Perfect fidelity can coexist with a potentially misleading explanation, establishing that prediction agreement alone does not guarantee trustworthiness alignment.Theorem 3.1 asserts existence of a black box and explanation with L(E, B) = 0 while the explanation remains potentially misleading.
- Existence Results: Potentially misleading explanations arise when prohibited features P can be reconstructed from desired and ambivalent features D ∪ A.An acceptable interpretable explanation can then internally reconstruct prohibited information while presenting acceptable features.
- Existence Results: Theorem 3.2 states that, when O∗(B) = 0 and L(E, B) + 2ϵR + ϵA ≤ ϵ+, an acceptable explanation E+ is potentially misleading.The condition combines the explanation’s loss with restriction and acceptable relative errors.
4 Generating Misleading Explanations
The approach extends MUSE’s interpretable, fidelity-oriented two-level decision sets with constraints that generate explanations containing desired features while omitting prohibited ones.
- Background on MUSE: MUSE represents explanations as two-level decision sets with nested if-then rules, combining neighborhood descriptors with inner model-behavior rules.Each rule uses conjunctions of feature predicates such as “age ≥50” and assigns a label.
- Background on MUSE: MUSE optimizes interpretability and fidelity, using measures such as predicate count, feature overlap, rule overlap, cover, and fidelity to the black box.Fidelity is measured as accuracy relative to the black box, while additional size and width measures constrain explanation complexity.
- MUSE Optimization: Frequent itemset mining constructs candidate outer and inner if-then conditions, which form the search space for the optimization problem.Candidate neighborhood descriptors come from ND, while candidate inner rules come from DL.
- Our Approach: The misleading-explanation extension excludes prohibited attributes from candidate rules and adds an objective term rewarding coverage of desired features.Maximizing desired-feature coverage increases the chance that every desired attribute appears somewhere in the explanation.
- Our Approach: The resulting optimization remains non-normal, non-negative, non-monotone, and submodular with matroid constraints, so approximate local search can solve it with theoretical guarantees.Exactly solving the corresponding optimization problem is NP-hard; approximate local search provides the best known guarantees for this class of problems.
5 Experimental Evaluation
The evaluation constructs an intentionally untrustworthy bail-decision black box and tests whether different high-fidelity explanations alter domain experts’ trust. Explanations omitting prohibited features substantially increased reported trust, demonstrating misleading effects.
- Study Design: The study used bail decisions, a high-stakes prediction task, with 47 law-school students familiar with bail decision making.Participants identified relevant and prohibited features before evaluating the classifier and its explanations.
- Feature Selection: Race and gender were selected as prohibited features, while prior jail incarcerations (PJI) and prior failure to appear (PFTA) were selected as desired features.The first two features in each category received significantly more votes than the others.
- Black Box Construction: The constructed black box used race and gender, omitted PJI and PFTA, and achieved 83.28% accuracy on the held-out test set.The model was designed to be untrustworthy to experts who examined its internal workings.
- Explanation Construction: The three explanations varied which prohibited and desired features they displayed, while their test-set fidelities remained similar: E1 97.3%, E2 98.9%, and E3 98.2%.This controlled fidelity before comparing trust effects associated with explanation content.
- Trust Results: 9.1% trusted the actual black box, compared with 70% for E1 and 88% for E3; E2 produced 10% trust.E1 and E3 omitted race and gender, while E3 also included desired features PJI and PFTA.
- Trust Results: Participants’ stated reasons were consistent with trust being primarily driven by whether race and gender appeared in the explanation.This pattern persisted despite a warning that the explanations were not causal.
6 Discussion & Conclusions
The paper studies how explanations can manipulate trust in black boxes through a theoretical framework, misleading-explanation generation, and a domain-expert user study. It finds that high-fidelity explanations can induce trust in black boxes users would otherwise reject.
- Contributions: The study empirically examines whether and how black-box explanations mislead users and affect their trust.It combines a theoretical framework, a generation approach, and a user study with law and criminal-justice experts.
- Misleading Explanations: Perfect fidelity may fail to reveal black-box problems because correlated features can substitute for prohibited features in an explanation.For example, zip code can reconstruct race or gender information.
- Conclusion: The experiments show that adversarial actors can use misleading explanations to fool users into trusting an untrustworthy black box.The demonstrated mechanism involves hiding prohibited attributes while preserving high fidelity through correlated features.
- Discussion: Interactive explanations let users query perspectives such as how predictions vary across defendants’ races or genders.MUSE can return explanations using race or gender in outer if-then clauses.
- Discussion: Correlational explanations may be misleading and lack robustness, whereas causal explanations may address these issues.The paper presents causal explanation work as one possible response to correlation-based limitations.
A.1 Interpretability Measures
The paper measures interpretability and fidelity-related properties of two-level decision sets using objectives for structural, semantic, coverage, overlap, and disagreement characteristics.
- Measure Setup: The measures are defined for a two-level decision set R with M rules, a black box B, and training set D.The resulting measures are summarized in Table 2.
- Structural Measures: Four structural measures target smaller explanations: size, maximum width, total predicate count, and number of unique neighborhood descriptors.Each measure penalizes a different aspect of rule-set complexity.
- Semantic Measures: The semantic objective minimizes feature overlap between outer neighborhood descriptors and inner if-then rules.The distinction reflects their different semantic meanings.
- Decision-Set Measures: Decision-set measures minimize rule overlap and maximize cover so that examples are usually handled by one applicable rule.Rule overlap counts extra applicable rules, while cover counts instances satisfying some rule.
- Objective Functions: The objective functions combine predicate count, feature overlap, rule overlap, cover, and disagreement, with each bounded measure subtracted from an upper bound where needed.Wmax denotes the maximum width of any rule in either candidate set, and the transformations ensure non-negative objectives.
B Proofs of Theorems
The proofs construct a perfectly correlated two-feature example where the black box relies on a prohibited feature while a faithful explanation uses a desired feature, establishing misleadingness. They also establish structural properties of MUSE’s optimization objective, including non-normality, non-monotonicity, and submodularity.
- Theorem 3.1: The constructed distribution makes x1 and x2 perfectly correlated, with the outcome determined by whether x2 is nonnegative.The proof uses a standard Gaussian x1, sets x2=x1, and defines the label as I[x2 ≥0].
- Theorem 3.1: The black box achieves zero loss but is deemed untrustworthy because it uses the prohibited feature x2.The proof assigns the black box’s objective value 0 under this feature usage.
- Theorem 3.1: An explanation using the desired feature rather than x2 is acceptable yet misleading because its objective differs from the black box’s true objective.The proof concludes that the explanation is misleading despite matching the black box perfectly in fidelity.
- Theorem 3.2: The relative-error proof establishes a triangle-style decomposition, bounding L(F,F′) by L(F,F′′) + L(F′′,F′).The derivation expands the loss into indicator terms and uses symmetry of relative error.
- Theorem 4.1: The optimization objective inherits non-normality, non-monotonicity, and submodularity from its component objectives and the desired-feature coverage function.The proof notes that coverdesired has diminishing returns and that linear combinations of submodular functions remain submodular.