PaperlayerView exact revision on arXiv

Source-linked AI summary

Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contexts

Sebastian Bordt, Michèle Finck, Eric Raidl, Ulrike von Luxburg

arXiv:2201.10295v2cs.LGcs.AI

TL;DR

Legal obligations to provide information about machine-learning systems are often treated as demands for explanations, but post-hoc methods are proposed despite uncertainty about their suitability. The paper combines legal, philosophical, and technical analysis to assess them in adversarial contexts, finding that their ambiguity permits manipulation and limits their usefulness for transparency.

  • Problem

    It is unclear whether post-hoc explanations can meet transparency objectives when the provider and recipient have opposing interests.

    Source: §1 · p. 2

  • Method

    The paper combines legal, philosophical, and technical analysis of local post-hoc explanations in adversarial decision settings.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    In adversarial contexts, local post-hoc explanations are either trivial or harmful and may falsely suggest that a decision was justified or objective.

    Source: §6 · p. 13

  • Takeaways & Limitations

    The paper argues that post-hoc explanations are of limited use for transparency in adversarial contexts and that alternative means may be more suitable for assessing discrimination.

    Source: §6 · p. 13

    The criticism concerns adversarial scenarios; the paper notes that explainable machine learning may support discoveries in cooperative settings.

    Source: §6 · p. 13

Abstract

from arXiv · show

Existing and planned legislation stipulates various obligations to provide information about machine learning algorithms and their functioning, often interpreted as obligations to "explain". Many researchers suggest using post-hoc explanation algorithms for this purpose. In this paper, we combine legal, philosophical and technical arguments to show that post-hoc explanation algorithms are unsuitable to achieve the law's objectives. Indeed, most situations where explanations are requested are adversarial, meaning that the explanation provider and receiver have opposing interests and incentives, so that the provider might manipulate the explanation for her own ends. We show that this fundamental conflict cannot be resolved because of the high degree of ambiguity of post-hoc explanations in realistic application scenarios. As a consequence, post-hoc explanation algorithms are unsuitable to achieve the transparency objectives inherent to the legal norms. Instead, there is a need to more explicitly discuss the objectives underlying "explainability" obligations as these can often be better achieved through other mechanisms. There is an urgent need for a more open and honest discussion regarding the potential and limitations of post-hoc explanations in adversarial contexts, in particular in light of the current negotiations of the European Union's draft Artificial Intelligence Act.

1 INTRODUCTION

Post-hoc explanations are often proposed as a way to meet legal information and transparency objectives, but this paper argues they are unsuitable in adversarial settings. The provider’s opposing incentives and the ambiguity of local explanations leave room for subtle manipulation.

1 INTRODUCTION

  • Local post-hoc methods explain individual decisions without constraining model complexity or requiring model disclosure, prompting proposals to use them for legal requirements.The paper contrasts these methods with inherently interpretable models.Source: §1 · p. 1
  • Adversarial contexts involve opposing interests, unlike cooperative settings where providers and recipients share the goal of finding a useful explanation.A loan denial contested by an applicant is an example of an adversarial setting.Source: §1 · pp. 1–2
  • The paper combines legal, philosophical, and technical analysis to assess post-hoc explanations as tools for transparency and accountability in adversarial contexts.It examines EU legal uncertainty, the goals of explainability, and philosophical and technical limitations.Source: §1 · p. 2
  • Local post-hoc explanations are highly ambiguous, giving providers many subtle, undetectable ways to select explanations that suit their interests.The paper argues that the truth assumptions needed for legal goals do not hold in adversarial contexts.Source: §1 · p. 2

2 EXPLANATIONS IN COOPERATIVE AND ADVERSARIAL CONTEXTS

The paper distinguishes cooperative explanation settings, where parties share goals, from adversarial ones, where their interests conflict. It focuses on individual-level post-hoc explanations in supervised tabular decision systems and argues their role differs sharply across these settings.

2 EXPLANATIONS IN COOPERATIVE AND ADVERSARIAL CONTEXTS

  • Cooperative contexts align the goals of system builders, explanation providers, and users, while adversarial contexts such as loan denials pit the applicant’s interests against the bank’s.The authors argue local post-hoc explanations may be pointless or harmful in adversarial settings.Source: §2 · p. 2

2.1 Parties involved in the adversarial explanation process

  • The explanation recipient may be the decision subject or an external examiner acting on her behalf, while the system creator typically provides the explanation.The creator may be a private company or public entity.Source: §2.1 · p. 3

2.2 Machine learning problem: Supervised learning, tabular data, point-wise post-hoc explanations

  • The technical analysis assumes supervised learning on tabular person-level features, with an explanation algorithm queried about a data point and its decision.The decision may be binary or a numeric risk score; the explainer accesses the decision function and sometimes training data.Source: §2.2 · p. 3

2.3 Explanation algorithms that fall into this framework

  • LIME and SHAP provide local feature attributions, whereas counterfactual methods search for a nearby input that produces a different decision.These methods explain individual decisions rather than provide a global view of the decision function.Source: §2.3 · p. 3

3 LEGAL FRAMEWORK: EXPLAINABILITY IN EU LAW

The paper finds that EU law does not clearly impose a general explainability obligation, and that the relationship between explainability and transparency remains unsettled. It examines the AIA and GDPR, then considers the objectives explanations are meant to serve and their dependence on truthful information.

3 LEGAL FRAMEWORK: EXPLAINABILITY IN EU LAW

  • The EU legal framework includes information duties, but it remains unclear how they should be implemented technically and whether explainability is a standalone obligation or a means of transparency.The analysis focuses on the draft AIA and GDPR.Source: §3 · pp. 3–4

3.1 The draft Artificial Intelligence Act (AIA)

  • The draft AIA does not create a general explainability obligation; for high-risk systems it requires transparency that enables users to interpret outputs, without specifying the technical requirements.The draft also provides general disclosure duties for some non-high-risk systems, such as informing users that they are interacting with AI.Source: §3.1 · p. 4

3.2 The General Data Protection Regulation (GDPR)

  • The GDPR has no general right to an explanation, though qualifying solely automated decisions carry rights to human intervention, to express a viewpoint, and to contest the decision.Recital 71 mentions obtaining an explanation after human review, but recitals are not as legally binding as the GDPR’s text.Source: §3.2 · p. 5

3.3 Explainability as a sub-component of transparency

  • It remains unsettled whether EU legal references to explainability mean a distinct obligation or a component of transparency, and the paper calls for clarification to improve legal certainty.The paper notes that the AIA and GDPR can be read as requiring broader transparency rather than explainability as commonly understood in computer science.Source: §3.3 · pp. 5–6

3.4 Rationale and objectives of explainability norms in an adversarial setting

  • Trust and the ability to contest decisions or seek recourse are proposed objectives of explanations, but these uses presuppose that explanations are correct.The paper says trust may fail when explanations cannot be sensibly tested or the exchange is one-shot, and false explanations may not support recourse.Source: §3.4 · pp. 6–7

4 THE PROBLEMS WITH POST-HOC EXPLANATIONS IN ADVERSARIAL CONTEXTS

In adversarial contexts, post-hoc explanations are underdetermined, giving providers room to select explanations that suit their interests. Their limits arise from incomplete algorithmic representations and choices about explanation methods, parameters, classifiers, and datasets.

4 THE PROBLEMS WITH POST-HOC EXPLANATIONS IN ADVERSARIAL CONTEXTS

  • Because post-hoc explanations are underdetermined, adversarial providers can exploit their ambiguity to produce explanations that are difficult to contest.The paper identifies explanation methods and parameters, decision-boundary shape, and reference-dataset choice as sources of discretion.Source: §4 · p. 7

4.1 The algorithm’s view of the world is coarse-grained and incomplete - this limits potential explanations

  • An explanation algorithm can use only features, labels, and learned input-output associations, so real-world causes absent from that representation may be unidentifiable or inexpressible.For example, a model trained on age and blood pressure may suggest age as a cause when smoking, absent from the data, is the actual cause.Source: §4.1 · p. 8

4.2 Even within the algorithm’s own world, a unique preferred reason does not exist

  • No unique true reason generally exists even within the model’s world; feature attribution and counterfactual methods rely on different plausible assumptions and mechanisms.LIME examines the local gradient, while SHAP compares the point with a reference population.Source: §4.2 · p. 8

4.3 Different explanation algorithms lead to different explanations

  • Different methods can give markedly different explanations for the same decision, and one counterfactual method can yield more than 100 alternatives per German Credit individual.The authors note there is no principled way to choose among these counterfactuals in adversarial settings.Source: §4.3 · p. 9

(b) LIME

  • Sanity checks can exclude unreasonable explanation methods, but cannot establish which of two remaining explanations is more correct.Each method makes assumptions about which properties of the decision function it highlights.Source: (b) LIME · p. 9

4.4 The explanation provider can choose between a large number of possible explanation algorithms and parametrizations

  • Even within one method, parameter choices can change explanations, while an expanding set of plausible methods and configurations leaves providers substantial choice.Examples include LIME’s bandwidth and perturbation count, and counterfactual methods’ distance metrics and penalty terms; principled defaults may exist in some cases.Source: §4.4 · pp. 9–10

4.5 Explanations depend on the exact shape of the high-dimensional decision boundary

  • Classifiers with comparable predictive behavior can yield different explanations: linear regression and random forest predictions agree 94% of the time, yet their explanations differ.The decision boundary can be changed outside the data distribution without changing classification behavior, and fewer than 50% of counterfactuals for a gradient boosted tree also work for logistic regression.Source: Figure 3 · p. 10, §4.5 · pp. 10–11

4.6 It is unclear how to choose the reference dataset that many explanations depend on

  • Reference-dataset composition can shift which feature an explanation identifies, and broader human choices let adversarial providers select plausible explanations aligned with their incentives.The paper argues this makes acceptability difficult for external observers, including judges and regulatory bodies.Source: §4.6 · p. 11, §4.7 · p. 11

5 ONCE AN EXAMINER IS ALLOWED TO ASSESS THE PROVIDED POST-HOC EXPLANATIONS, SHE’D BETTER INVESTIGATE THE DECISION FUNCTION DIRECTLY

The authors argue that once an examiner can query the decision system, direct testing is preferable to relying on adversarially supplied post-hoc explanations. Even full access to the system, code, and training data leaves explanations without a uniquely correct standard, so examiners should test the system themselves.

5.1 Minimalist scenario where decision function and explanation algorithm can be queried

  • The minimalist scenario lets examiners query decisions and explanations to assess internal consistency, but excludes explicit access to the decision function, explanation algorithm, source code, and training data.The examiner can test properties such as faithfulness, robustness, and necessity or sufficiency of feature attributions.Source: §5.1 · p. 12
  • With query access, examiners can test decision behavior directly, such as fairness on an independent reference dataset, and are well-advised to ignore adversarial explanations.The authors note that what can be assessed through querying still requires further research.Source: §5.1 · p. 12

5.2 Fully transparent scenario where algorithms’ source code and training data are disclosed

  • Even with access to the decision function, source code, and training data, examiners should test the system independently and ignore the creator’s explanations.Parameter and dataset choices can influence explanations, and there is generally no uniquely correct explanation beyond internal-consistency criteria.Source: §5.2 · pp. 12–13

6 DISCUSSION

The discussion concludes that local post-hoc explanations are of limited use in adversarial settings, where ambiguity and manipulation can undermine transparency. The authors call for clearer legal objectives and further research into direct examination and the practical limits of transparency.

6 DISCUSSION

  • In adversarial contexts, local post-hoc explanations are either trivial or harmful and may make unjustified decisions appear objective.The authors contrast this with cooperative settings, where explanations can support debugging, scientific discovery, and practical applications.Source: §6 · p. 13
  • The GDPR creates no general explainability obligation, while the draft AI Act would require some explainability for high-risk applications; the authors urge more specific legal provisions.If the final AI Act requires strong explainability, they argue black-box systems cannot meet it and inherently interpretable models may be needed, potentially impeding innovation.Source: §6 · p. 13
  • External examiners may assess discrimination more directly by testing systems, rather than relying on explanations that creators may not design to reveal hidden bias.The authors suggest external examination of black boxes as a potentially more suitable means of enabling accountable AI systems.Source: §6 · p. 13
  • Even full pipeline documentation leaves room for manipulation, and examining or retraining systems built by large teams may be very difficult or impossible with available resources.The authors say more research is needed to determine which legal objectives extended documentation can satisfy.Source: §6 · p. 13
  • Testing and certifying machine-learning systems in adversarial settings remains underexplored, and even complete transparency may not achieve every desired transparency and control goal.The authors call for clarifying which objectives can be achieved by which means; their criticism of post-hoc explanations is limited to adversarial scenarios.Source: §6 · p. 13

7 FUNDING DISCLOSURE

The authors report partial support from several research and funding institutions and declare no additional funding sources or financial interests.

7 FUNDING DISCLOSURE

  • The work was partially supported by the German Research Foundation, Baden-Württemberg Foundation, BMBF Tübingen AI Center, IMPRS-IS, and Carl Zeiss Foundation.The authors declare no additional funding sources or financial interests.Source: §7 · p. 14

A.2 Datasets

The experiments use five datasets spanning income, credit risk, disease progression, and cancer diagnosis, with the specified models’ test accuracies reported for each.

A.2 Datasets

  • 87% test accuracy was achieved on Adult-Income using a gradient boosted tree to predict income above $50,000.The dataset contains 12 features drawn from the 1994 US Census.Source: §A.2 · p. 16
  • 76% test accuracy was achieved on German Credit with a gradient boosted tree, versus 74% with logistic regression.The task is binary credit-risk prediction using 20 features.Source: §A.2 · p. 16
  • 83% test accuracy was achieved on Folktables’ ACSIncome task using a gradient boosted tree to predict income above $50,000.The data come from the 2016 California Census, with eight personal characteristics.Source: §A.2 · p. 16
  • 74% test accuracy was achieved on the Diabetes dataset with a random forest, versus 71% with linear regression.The task predicts disease progression one year after baseline, dichotomized at the median.Source: §A.2 · p. 16
  • 96% test accuracy was achieved on Wisconsin Breast Cancer with linear regression, versus 36% when trained on random labels.The binary task predicts malignant or benign diagnosis from 30 features describing cell nuclei.Source: §A.2 · p. 16

A.3 Explanation Algorithms

The experiments use SHAP, LIME, and DiCE through their accompanying Python packages, with specified implementations or parameter choices for each.

A.3 Explanation Algorithms

  • SHAP [30] uses exact computation for gradient-boosted trees and Kernel SHAP for other classifiers; the package also implements Janzing et al. [20].Package-documented parametrizations are used whenever available.Source: §A.3 · p. 16
  • LIME [41] is used through its accompanying Python package, with documented parametrizations whenever available.Source: §A.3 · p. 16
  • DiCE [34] generates counterfactual explanations using its accompanying package’s model-agnostic randomized sampling method.Source: §A.3 · p. 16

A.4 Figures

Feature attributions are normalized before the figures are created.

A.4 Figures

  • Feature attributions are normalized to have l1-norm 1 for figure creation.Source: §A.4 · p. 17

A.5 Additional Figures

The additional figures follow those in the main paper and show initial test-set observations, rather than hand-selected examples.

A.5 Additional Figures

  • The additional figures show the first test-set observations and are not hand-selected; the main-paper figures were selected from these.Figures for all test observations are stated to be available with the code upon publication.Source: §A.5 · p. 17

Additional Figures Related to Figure 1 in the Main Paper

The additional figures illustrate variation in post-hoc explanations across algorithms and decision-boundary shapes. They show comparisons for the first six test observations.

Additional Figures Related to Figure 1 in the Main Paper

  • Figure A.1 compares explanations from four algorithms for each of the first six test observations.Each row depicts a different individual.Source: Figure A.1 · p. 18

Additional Figures Related to Figure 2 in the Main Paper

  • Figure A.2 shows that two algorithms can produce similar, different, or partly similar and different explanations for a datapoint.Each row depicts a different individual among the first six test observations.Source: Figure A.2 · p. 19

Additional Figures Related to Figure 3 (a), (b) in the Main Paper

  • Figure A.3 illustrates that explanations depend on the exact shape of the decision boundary.Each row depicts two algorithms’ explanations for a different individual among the first six test observations.Source: Figure A.3 · p. 20

Additional Figures Related to Figure 3 (c), (d) in the Main Paper

  • Figure A.4 likewise illustrates that explanations depend on the exact shape of the decision boundary.Each row depicts two algorithms’ explanations for a different individual among the first six test observations.Source: Figure A.4 · p. 21

Related papers

Loading 2201.10295v2…