Source-linked AI summary
Adversarial attacks and defenses in explainable artificial intelligence: A survey
Hubert Baniecki, Przemyslaw Biecek
TL;DR
XAI explanations can be manipulated through attacks on data or models, undermining their security and trustworthiness in practical applications. This survey systematizes research on attacks against explanations, defenses, and fairness metrics across more than 50 papers. It identifies existing insecurities, discusses robust interpretation methods, and outlines future AdvXAI research directions.
Problem
Adversarial machine learning has exposed vulnerabilities in explanation methods, creating a need for comprehensive coverage of attacks on explanations and fairness metrics.
Method
The survey systematizes attacks and defenses on model explanations, relates them to fairness-metric attacks, and reviews more than 50 papers gathered from major venues and citation networks.
Results
The survey identifies existing XAI insecurities, including insufficiently addressed backdoor, poisoning, language-specific, and concept-based attacks, alongside sparse defenses for global explanations.
Takeaways & Limitations
Explanation methods need resilience to adversarial action, with future AdvXAI work extending robustness research to transformer-based and foundation models.
Takeaways & Limitations
The surveyed explanation approaches are non-exhaustive, and studies remain sparse for some attack and defense settings, including concept-based attacks and global explanations.
Abstract
from arXiv · showhide
Explainable artificial intelligence (XAI) methods are portrayed as a remedy for debugging and trusting statistical and deep learning models, as well as interpreting their predictions. However, recent advances in adversarial machine learning (AdvML) highlight the limitations and vulnerabilities of state-of-the-art explanation methods, putting their security and trustworthiness into question. The possibility of manipulating, fooling or fairwashing evidence of the model's reasoning has detrimental consequences when applied in high-stakes decision-making and knowledge discovery. This survey provides a comprehensive overview of research concerning adversarial attacks on explanations of machine learning models, as well as fairness metrics. We introduce a unified notation and taxonomy of methods facilitating a common ground for researchers and practitioners from the intersecting research fields of AdvML and XAI. We discuss how to defend against attacks and design robust interpretation methods. We contribute a list of existing insecurities in XAI and outline the emerging research directions in adversarial XAI (AdvXAI). Future work should address improving explanation methods and evaluation protocols to take into account the reported safety issues.
1. Introduction
XAI methods support interpretation in applications such as auditing, debugging, autonomous driving, and scientific discovery, but adversarial manipulation threatens the trustworthiness and security of their explanations. This survey systematizes attacks, defenses, fairness-related work, and future directions in adversarial explainable AI.
- XAI methods provide mechanisms for interpreting machine-learning predictions and support applications including autonomous driving, drug discovery, auditing, debugging, and scientific discovery.
- Adversarial attacks can manipulate explanations by changing data or models, causing misinterpretation of model behaviour by explanation recipients.
- Common failure modes include adversarial examples, data poisoning, model manipulation, and backdoors.
- Explanation aggregation and model regularization are proposed defenses, although the relative effectiveness of patching explanations versus improving models remains open.
- The survey addresses a gap in comprehensive coverage by systematizing attacks and defenses on explanations, relating them to fairness metrics, and reviewing over 50 papers.
- Its review identifies research gaps and future directions using papers from major venues and citation-network searches while excluding work focused only on non-adversarial explanation evaluation.
2. Background
The background introduces adversarial machine learning, explanation methods, and the notation used to relate model predictions to explanations. It covers attack classes, defenses, and local and global explanation families, while noting that the surveyed explanation methods are non-exhaustive.
- Notation: The paper uses supervised models fθ mapping inputs X to outputs Y and an explanation function g(f,x) to represent explanations of predictions.
- Adversarial machine learning: Adversarial examples minimally modify inputs to fool models, while defenses aim to reduce attack risk through methods such as data augmentation, regularization, distillation, and detection.
- Adversarial machine learning: Backdoor attacks poison training data so models behave incorrectly on inputs containing an adversary-known trigger while remaining indistinguishable from the desired model.
- Explainable artificial intelligence: XAI methods are used to understand model predictions in regulatory audit, debugging, and scientific discovery, often through human-comprehensible reasoning.
- Explainable artificial intelligence: Local post-hoc feature attributions indicate how much input features contributed to an individual prediction, with perturbation-, gradient-, and propagation-based variants.
- Explainable artificial intelligence: Shapley-value algorithms provide a state-of-the-art approach for obtaining explanation vectors, with variations across modalities, models, quality metrics, and computational efficiency.
- Explainable artificial intelligence: Global explanations summarize prediction patterns across a data distribution through feature importance, aggregated attributions, feature effects, and concept-based explanations.
- Scope: The survey covers a non-exhaustive subset of explanation approaches most targeted by adversarial attacks, alongside related robustness-evaluation and adversarial-application research.
3. Adversarial attacks on model explanations
Adversarial attacks can manipulate explanations while preserving predictions, using changes to inputs, models, data, or combinations of these. The surveyed attacks expose vulnerabilities across local and global explanation methods, including opportunities to conceal unfair model behavior.
- Adversarial examples: Gradient-based adversarial examples perturb inputs to drastically change explanations while preserving the model’s predicted class.The attack iteratively updates inputs in the explanation-gradient direction under small-perturbation and same-class constraints.
- Attack taxonomy: The survey organizes attacks by whether they change data, models, explanations, or combinations, across image, tabular, and language settings and local or global methods.Its notation and taxonomy distinguish neural-network-specific from model-agnostic attack algorithms.
- Model and backdoor attacks: Model manipulation changes weights, fine-tuning, or regularization to alter explanations without affecting predictive performance.Backdoors can trigger manipulated explanations for specific input patterns, while other inputs may retain original explanations.
- Adversarial models and fairness: Black-box attacks exploit perturbation-based LIME and SHAP estimation to hide biased in-distribution predictions behind fair-looking explanations.A surrogate model controls out-of-distribution behavior while the original black-box remains biased on in-distribution inputs.
- Data poisoning and global attacks: Data poisoning and biased sampling manipulate local or global feature explanations while leaving sensitive features or much of the data distribution unchanged.Reported attacks target feature-effect explanations and global SHAP, including attempts to conceal the contribution of sex or other sensitive features.
- Insights and limitations: Attack coverage remains uneven: most studies assume neural-network knowledge, image work repeatedly uses ImageNet, and concept-based, counterfactual, and language-model explanations receive relatively sparse attention.Interactions between local and global attacks may also affect detectability in practice.
4. Defense against the attacks on explanations
Defense research targets attacks on explanations through regularization, aggregation, constrained sampling, and robust explanation construction. The survey identifies important attack classes and explanation settings that remain insufficiently addressed.
- Defense approaches: Regularization can force integrated gradients explanations to remain unchanged under perturbation attacks.Chen et al. defend against adversarial examples by regularizing a neural network.
- Defense approaches: Aggregating explanations from different algorithms is less susceptible to attacks targeting a single explanation method.The ensemble is more robust because the attacker targets only one explanation method.
- Defense approaches: Robustness defenses include model regularization, focused sampling, user-specified input constraints, and certifiably robust neural-network explanations.LIME and SHAP defenses restrict or adapt perturbations, while certification can reassure that no adversarial explanation exists for specified inputs or model weights.
- Scope of defenses: The defense literature includes image, tabular, and language data, neural networks and black-box models, and local and global explanations.Table 3 records the datasets, models, explanation algorithms, modalities, and explanation scopes considered in experiments.
- Open gaps: Unaddressed insecurities include backdoor and data-poisoning attacks, language-specific attacks, concept-based explanations, and most attacks on global explanations.The survey notes sparse defense studies for global explanations and excludes robustness work unrelated to adversarial attack scenarios.
5. Adversarial attacks on fairness metrics
Adversarial attacks on fairness metrics modify models, data, inputs, or graphs to alter measured fairness while sometimes preserving predictive behavior. The surveyed work concentrates on group fairness and includes fairwashing, poisoning, robustness-bias, and information-retrieval scenarios.
- Attack mechanisms: Fairness attacks can manipulate measured bias through model changes, data poisoning, adversarial edges, or adversarial examples.Table 4 categorizes attacks by changed data or model and covers image, tabular, and graph settings.
- Model attacks: Fairwashing approximates an unfair black-box model with a faithful adversarial model that appears fair.Experiments analyze the fidelity–unfairness trade-off and fairwashing’s effect on feature-effect explanations.
- Attack mechanisms: Data-poisoning attacks can transfer across model algorithms in black-box settings where the attacker has no model access.Experiments report transferability from neural-network optimization to other model algorithms.
- Attack mechanisms: Anchoring attacks manipulate a model’s decision boundary by generating same-group points with opposite labels.Experiments report improved fairness manipulation over prior work, including in black-box settings.
- Scope and gaps: Dataset extraction attacks seek sensitive information from metric values, while related membership-inference attacks use explanations for privacy attacks.These scenarios retrieve information rather than directly changing fairness metrics.
- Scope and gaps: Fairness-metric research focuses on group fairness and omits subgroup and individual fairness.The distinction is between treating groups equally and predicting similarly for similar individuals.
6. AdvXAI: Opportunities, challenges and future research directions
AdvXAI research must address attacks, defenses, robustness, evaluation, human factors, and broader societal and legal implications across models and modalities. The survey identifies concrete directions including transformer and multimodal studies, benchmarks, incident tracking, certification, and stakeholder-aware design.
- Attacks: AdvXAI must continue tracking emerging attacks, including threats against newer explanation methods, tree-based models, interpretable-by-design systems, and defenses themselves.The survey notes that attacks circumventing XAI defenses remain unexplored.
- Defenses: Robust defenses should improve explanation stability, address attacks on fairness metrics, and recognize that manipulated explanations undermine evidence for model predictions.Proposed directions include robust gradient- and perturbation-based explanation algorithms and theoretical analysis under data shifts.
- Scope expansion: Adversarial robustness requires dedicated study of explanations for transformer architectures and models operating beyond image and tabular data.Highlighted modalities include language, graphs, time series, multimodal systems, and reinforcement learning agents.
- Software, datasets and benchmarks: Open-source attack and defense implementations, real-world datasets, security benchmarks, and competitions could support reproducible and comparable AdvXAI research.The survey suggests that a security benchmark could resemble Safety Gym.
- Good practices: Evaluation should include adversarial and distribution-shift settings, while incident catalogues and certification schemes could document harms and support safer XAI deployment.The survey connects these practices to existing AI incident databases and certification approaches.
- Human stakeholders and societal impact: Human information overload, domain-dependent attack success, regulation, and XAI-washing make stakeholder effects and societal consequences central AdvXAI concerns.The survey specifically raises questions involving the AI Act, fairness rules, and the right to explanation.
Appendix A. List of abbreviations and proper names (with references)
The appendix defines abbreviations and names for explanation methods, model architectures, fairness metrics, software, and standard machine-learning algorithms used throughout the survey.
- Named methods and tools: Additional named entries include Anchors, CAM, FFV, and fairlearn, covering model-agnostic explanations, activation mapping, feature visualization, and fairness tooling.The appendix associates each name with its cited reference.
- Explanation methods: Explanation-method abbreviations include GBP, GI, Grad-CAM, GS, IG, IME, LIME, LRP, MUSE, PDP, Relevance-CAM, RTS, SG, SHAP, and TCAV.These entries expand the names of gradient, saliency, relevance, feature-dependence, and concept-based explanation methods.
- Models and algorithms: Model and algorithm abbreviations include CNN, DT, GBDT, GCN, KNN, LR, MLP, RF, SVM, ViT, and XGBoost.The appendix provides expansions ranging from neural and graph models to trees, nearest neighbours, regression, and boosting.
- Explanation terminology: The appendix distinguishes related explanation variants such as Grad-CAM, Relevance-CAM, and SHAP alongside their expanded names and references.These entries support consistent terminology across the survey.
- Fairness and software: Fairness-related entries include EOdds, EOpp, PE, and SP, denoting equalized odds, equal opportunity, predictive equality, and statistical parity.The list also names fairlearn software.
- Model terminology: ViT and XGBoost are expanded as vision transformer and extreme gradient boosting, respectively, while DT and GBDT denote decision-tree variants.These abbreviations appear among the appendix’s model and algorithm entries.