PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual Explanations Can Be Manipulated

Dylan Slack, Sophie Hilgard, Himabindu Lakkaraju, Sameer Singh

arXiv:2106.02666v2cs.LG

TL;DR

Counterfactual explanations are used to provide recourse, but their stability and reliability remain insufficiently understood. The paper formalizes their sensitivity to small input changes and constructs adversarial models that exploit it. Across loan and violent-crime prediction data, specific subgroups received 2-20× lower cost recourse under perturbation, while models appeared fair without it.

  • Problem

    Existing work provides limited systematic understanding of counterfactual explanations’ vulnerabilities, including whether their results are stable and reliable.

  • Method

    The paper develops a framework and adversarial training objective that manipulate hill-climbing counterfactual explanations through small input perturbations while retaining classification loss and fairness terms.

  • Results

    Across loan and violent-crime prediction datasets, specific subgroups achieved 2-20× lower cost recourse under perturbation without accuracy loss.

  • Takeaways & Limitations

    The results raise concerns about the trustworthiness of commonly used counterfactual explanations and motivate robust explanation methods.

  • Takeaways & Limitations

    The study focuses on neural networks, leaving manipulation of other model families such as decision trees and rule lists for future evaluation.

Abstract

from arXiv · show

Counterfactual explanations are emerging as an attractive option for providing recourse to individuals adversely impacted by algorithmic decisions. As they are deployed in critical applications (e.g. law enforcement, financial lending), it becomes important to ensure that we clearly understand the vulnerabilities of these methods and find ways to address them. However, there is little understanding of the vulnerabilities and shortcomings of counterfactual explanations. In this work, we introduce the first framework that describes the vulnerabilities of counterfactual explanations and shows how they can be manipulated. More specifically, we show counterfactual explanations may converge to drastically different counterfactuals under a small perturbation indicating they are not robust. Leveraging this insight, we introduce a novel objective to train seemingly fair models where counterfactual explanations find much lower cost recourse under a slight perturbation. We describe how these models can unfairly provide low-cost recourse for specific subgroups in the data while appearing fair to auditors. We perform experiments on loan and violent crime prediction data sets where certain subgroups achieve up to 20x lower cost recourse under the perturbation. These results raise concerns regarding the dependability of current counterfactual explanation techniques, which we hope will inspire investigations in robust counterfactual explanations.

1 Introduction

Counterfactual explanations support recourse in consequential decisions, but their limitations and reliability remain insufficiently understood. This work presents a framework for showing how commonly used explanations can be sensitive to small input changes and manipulated.

  • Counterfactual explanations can provide affected individuals with reasons for model decisions and guidance on changing unfavorable outcomes.
  • Prior work identifies invalidation after model updates and non-actionable recommendations from ignoring causal relationships, but leaves stability and reliability insufficiently examined.
  • The framework targets hill-climbing explanations, including Wachter et al.’s algorithm, DiCE, and prototype-guided counterfactuals, which are highly sensitive to small input changes.
  • Adversarial models can appear to provide fair recourse across subgroups while yielding much lower-cost recourse after a slight perturbation.
  • 2-20× lower cost recourse was obtained for specific subgroups under perturbation without accuracy loss across loan and violent-crime prediction datasets.

2 Background

Counterfactual explanations identify nearby data points that a model predicts will receive a desired outcome, with the required changes interpreted as recourse. The background defines their optimization objectives, distance-based costs, fairness criterion, and common optimization methods.

  • Counterfactual explanations return a nearby point predicted as positive, and the difference from the original point represents the changes needed for the desired outcome.
  • The counterfactual objective balances achieving the desired model outcome against keeping the counterfactual close to the original instance, with λ controlling that trade-off.
  • Distance Functions: Distance functions represent individual effort; weighted Manhattan distance encourages sparse changes while correcting for differing feature ranges.
  • Hill-climbing the Counterfactual Objective: Hill-climbing counterfactual methods optimize the objective through gradient descent or black-box optimization, as in Wachter et al.’s algorithm and DiCE.
  • Recourse Fairness: Recourse fairness is defined as limiting the disparity in average recourse distance between protected and not-protected groups to a threshold τ.
  • Figure 2 illustrates that similar apparent recourse costs can conceal much lower-cost recourse for males after perturbing their instances, revealing discrimination between sexes.

3 Adversarial Models for Manipulating Counterfactual Explanations

The paper constructs adversarial models that appear to provide fair recourse but yield lower-cost recourse for non-protected instances after a small perturbation. The training objective jointly manipulates the model and perturbation while preserving accuracy and exploiting counterfactual optimization.

  • The framework constructs models whose counterfactual explanations are unstable and can be manipulated through small input perturbations.The construction targets hill-climbing counterfactual methods and their sensitivity to perturbations.
  • The adversarial model jointly learns a small perturbation vector and model parameters so unperturbed recourses appear fair while perturbed non-protected instances receive lower-cost recourse.The perturbation has the same dimension as the input and is applied before counterfactual search.
  • The training objective combines recourse fairness on original data, lower-cost recourse for non-protected data after perturbation, small perturbation size, model accuracy, and counterfactual proximity.These terms are optimized over both model parameters and the perturbation vector.
  • Optimization first updates the perturbation and model using terms independent of the counterfactual algorithm, then updates model parameters using fairness terms involving counterfactual explanations.The second stage fixes the perturbation while retaining classification loss and adding competing recourse objectives.
  • The method computes counterfactual sensitivity through a Jacobian without requiring access to the explanation algorithm’s internal optimization details.The approach uses black-box access to counterfactual explanations; Hessian inversion is manageable for the small tabular feature sets considered.
  • The construction relies on non-convex counterfactual objectives so the model can make optimization converge to higher-cost local minima before perturbation and lower-cost minima after perturbation.The objectives are evaluated on held-out test data, while the training objectives use the training set.

4 Experiment Setup

The experiments evaluate four hill-climbing counterfactual algorithms on two public benchmark datasets using neural-network manipulated models. The setup compares manipulated models with an unmodified baseline on accuracy, perturbation size, and recourse costs.

  • The evaluation uses four hill-climbing algorithms: Wachter, Sparse Wachter, DiCE, and Counterfactuals Guided by Prototypes.DiCE generates four counterfactuals and uses the closest one by ℓ1 distance.
  • The experiments use Communities and Crime for violent-crime prediction and German Credit for financial prediction.Both are public benchmarks used in counterfactual-explanation and fairness research.
  • The manipulated models are four-layer feed-forward neural networks trained with tanh activations, Adam, and cross-entropy loss.The first optimization stage runs for 10,000 steps and the second for 15 steps on both datasets.
  • Table 2 compares recourse costs for both subgroups before and after adding the perturbation δ before counterfactual search.The setup includes one manipulated model for each counterfactual explanation algorithm and an unmodified comparison model.

5 Experiments

The experiments measure how effectively manipulated models hide recourse disparities, whether their counterfactuals remain realistic, and which search or model constraints improve robustness. Manipulations can appear fair while yielding substantially cheaper recourse after perturbation, but constrained attributes and simpler models reduce their effectiveness.

  • 5.1 Effectiveness of the Manipulation: The evaluation measures recourse-fairness disparity, non-protected-group costs with and without δ, and their ratio as the manipulation-success metric.Successful manipulation produces much lower perturbed cost for the non-protected group.
  • 5.1 Effectiveness of the Manipulation: In most cases, unperturbed recourse-cost disparity is small, while δ substantially reduces non-protected-group counterfactual costs.The manipulated models would therefore appear fair to auditors despite offering lower-cost recourse after perturbation.
  • 5.2 Outlier Factor of Counterfactuals: Manipulated recourses are only slightly less realistic than unmodified-model counterfactuals, while δ-generated counterfactuals are more realistic than the originals.The outlier-factor comparison uses lower-is-better realism scores for Wachter et al. and DiCE on Communities and Crime.
  • 5.3 Potential Mitigation Strategies: The experiments vary search initialization, attribute count, and model size to identify defenses against manipulation and their trade-offs.These strategies target different parts of counterfactual discovery and model capacity.
  • 5.3 Potential Mitigation Strategies: Reducing the number of attributes prevents low-cost adversarial recourse with minimal accuracy impact, supporting minimal-attribute searches in safety-concerned settings.The attribute experiment removes features from 99 down to 59 and averages cost reduction over five runs.
  • 5.3 Potential Mitigation Strategies: Increasing model size raises cost reduction by 1.5×, whereas using 4× fewer parameters substantially reduces it but lowers accuracy from 80 to 72.The results support using the most compact model that meets application accuracy requirements.
  • 5.3 Potential Mitigation Strategies: The proposed robustness options are random search perturbations, minimal attribute sets, and less complex models.Each option addresses manipulation through a different search or model-design constraint.

6 Related Work

Prior work studies recourse generation and several shortcomings of counterfactual explanations, including unrealistic recommendations, invalid recourse under distribution shift, and limited transfer across equally good models.

  • Recourse Methods: Wachter et al. use gradient search for the closest counterfactual, while Ustun et al. formulate actionable recourse for linear classifiers through linear programming.Other methods incorporate constraints to address unrealistic recommendations.
  • Shortcomings of Explanations: Prior studies show counterfactual explanations may fail to transfer across equally good models, become invalid under distribution shift, or be misused without clearer guidance.This work is positioned within research on predictive multiplicity, distribution shift, and responsible recourse use.

7 Potential Impacts

The paper discusses risks of developing adversarial models and explains why Communities and Crime is included despite critiques of crime prediction. The stated purpose is to expose misuse risks and motivate safeguards and robust counterfactual generation.

  • Impacts of Developing Adversarial Models: Adversarial-model techniques could be applied to generate manipulative models for harmful purposes.The risk includes modifying the objectives in other ways to train harmful models.
  • Impacts of Developing Adversarial Models: Exposing manipulations can inform recourse-system designers about risks and support appropriate checks and more robust counterfactual-generation algorithms.The paper presents disclosure as a way to help prevent such misuse.
  • Critiques of Crime Prediction Tasks: The Communities and Crime dataset is used to demonstrate how counterfactual explanations might be misused, not to advocate crime-prediction models.The paper notes that crime-prediction models have been extensively criticized and can have considerable negative impacts.

8 Discussion & Conclusion

The paper identifies a critical vulnerability in counterfactual explanations: they can be manipulated across several commonly used methods, raising concerns about their reliability. It proposes mitigation strategies while highlighting unresolved questions about model families, robustness, and explanation optimality.

  • Counterfactual explanations can be manipulated across Wachter, Sparse Wachter, prototype-guided counterfactuals, and DiCE.
  • These manipulations raise questions about the trustworthiness of counterfactual explanations for recommending recourse.
  • Three proposed mitigations are noisy initialization, fewer features in counterfactual computation, and lower model complexity.
  • The adversarial training procedure assumes the counterfactual explanation is known, although this assumption may not hold when explanations are private.
  • Future work includes testing attacks beyond neural networks, constructing robust counterfactuals, developing optimality guarantees, and studying simpler models in consequential domains.

A Optimizing Over the Returned Counterfactuals

This appendix formalizes optimization over counterfactuals returned by explanation methods and specifies several objective and evaluation choices. It also contrasts recourse costs for unmodified models with the intended effects of the adversarial objective.

  • The appendix treats optimization over a returned counterfactual as a bi-level problem involving an inner argmin that depends on model parameters.
  • The derivative of the returned counterfactual with respect to model parameters is derived using stationarity and the chain rule.
  • The derivative result is approximate when counterfactual search terminates before reaching a stationary point.
  • The appendix covers Wachter, Sparse Wachter, prototype-guided, and DiCE objectives, with DiCE using hinge loss rather than squared loss.
  • Evaluation selects the closest counterfactual by ℓ1 distance because the analysis targets a single least-cost counterfactual.
  • Unmodified models often show substantial recourse disparity, while δ has minimal effect on their counterfactual search.

B.2 Re-implementation Details

The re-implementation uses Adam-based optimization for several counterfactual methods and discusses practical failure and scalability considerations. It also reports that the unmodified models differ from the manipulated models in recourse disparity.

  • Wachter, Sparse Wachter, and prototype-guided counterfactuals are optimized with Adam using learning rate 0.01.
  • Adam-based counterfactual search can become stuck at its initialization and fail to find a successful counterfactual.
  • Unmodified models often exhibit large recourse disparities, whereas manipulated models show minimal subgroup disparity and lower-cost recourse after adding δ to non-protected instances.
  • Training complexity increases with batch size, dataset features, and the number of features optimized in the counterfactual perturbation.

D.2 Adult dataset

The adversarial training objective is reported to scale to the Adult dataset, which contains 33k datapoints, while producing a successful manipulation.

  • 33k datapoints: a successful manipulative model is trained on the Adult dataset.
  • The authors use DiCE with Adult-dataset preprocessing and numerical features to demonstrate scalability.

E Additional Results

Additional experiments reproduce the paper’s main patterns across datasets, initialization strategies, and categorical features. Manipulated + δ counterfactuals appear most realistic, while x+N is consistently the strongest initialization mitigation.

  • E.1 Outlier Factor of Counterfactuals: Manipulated + δ counterfactuals had the lowest percentage of predicted local outliers, indicating the most realistic counterfactuals in additional datasets and methods.This pattern was observed for German Credit and additional Communities & Crime settings.
  • E.2 Different Initializations: x+N was consistently the most effective initialization-based mitigation across additional German Credit and Communities & Crime experiments.The appendix reports this pattern for Sparse Wachter and prototype-guided counterfactuals, as well as German Credit comparisons.
  • E.3 Categorical Features: Manipulation succeeded in 3 of 4 categorical-feature cases, with rounded δ for prototype-guided counterfactuals as the exception.The appendix reports these experiments on German Credit using DiCE and prototype-guided counterfactuals.
Loading 2106.02666v2…