Source-linked AI summary
SoK: Privacy Attacks on Machine Learning via Explainable AI
Abdullah Caglar Oksuz, Anisa Halimi, Erman Ayday
TL;DR
Explanation interfaces create privacy and model-confidentiality risks, but existing black-box/white-box labels obscure what adversaries can actually obtain. This SoK systematizes 25 studies using separate model-knowledge and explanation-acquisition dimensions, finding that risk and defense effectiveness depend on the exposed signal, acquisition path, protected asset, and attacker knowledge.
Problem
Existing explanation-privacy studies use comparable labels for materially different security boundaries, obscuring how explanation signals reach adversaries.
Method
The SoK systematizes 25 attacks across extraction, membership inference, and inversion using a framework that separates model knowledge from five explanation-acquisition paths.
Results
Explanations reduce extraction cost, expose membership signals, and support reconstruction of private inputs, while no explanation family is uniformly unsafe.
Takeaways & Limitations
Explanation privacy should be evaluated end-to-end, with defenses matched to the acquisition path and protected asset.
Takeaways & Limitations
DP training can reduce training-membership leakage but does not automatically protect new query privacy or proprietary model confidentiality, and may damage utility and explanation quality.
Abstract
from arXiv · showhide
Machine learning explanations reveal model behavior beyond predictions, creating attack surfaces for model confidentiality and data privacy. We systematize 25 studies that exploit explanations for model extraction, membership inference, and model inversion, treating attribute inference as partial inversion. Existing work is often labeled only black- or white-box, obscuring substantial differences in what explanation signal reaches an adversary. We therefore separate model knowledge from explanation acquisition and identify five paths: target-released, attacker-derived, secondary disclosure, privileged access, and released global artifacts. Across these paths, explanations reduce extraction cost, expose membership signals through explanation statistics, recourse distance, and explanation-guided robustness, and support spatial or algebraic reconstruction of private inputs. We compare system and threat models, explanation signals, auxiliary knowledge, target models, modalities, query budgets, evaluation metrics, reported performance, and defenses. Our analysis shows that no explanation family is uniformly unsafe and no defense is uniformly effective. Risk depends on which signal is exposed, how it is acquired, which asset is targeted, and what the attacker already knows. We argue that explanation privacy should therefore be evaluated as an end-to-end disclosure problem, with defenses matched to the acquisition path and protected asset.
1 Introduction
This SoK systematizes explanation-assisted attacks on model confidentiality and data privacy while addressing incomparable threat-model labels. It separates attack families, explanation acquisition, and system assumptions to support more disciplined comparison.
- Problem: The central gap is comparability: identical black-box or white-box labels can conceal materially different model knowledge, interfaces, explanation signals, and trust boundaries.The paper distinguishes label-only services from gradient or explanation-rich interfaces and multiple disclosure sources.
- Approach: The framework compares attack goals, protected assets, model knowledge, outputs, explanation acquisition, auxiliary resources, query budgets, modalities, metrics, and defenses.This avoids ranking heterogeneous evaluations as though their metrics and threat models were equivalent.
- Scope: The SoK analyzes 25 explanation-assisted studies spanning model extraction, membership inference, and model inversion, including attribute inference as partial inversion.It retains extraction because explanations directly support it and extracted surrogates can enable downstream privacy attacks.
- Scope: The corpus excludes explanation manipulation without privacy or model-confidentiality objectives, generic adversarial examples, prediction-only attacks except as baselines, and training-gradient leakage outside explanation artifacts.Linkage and re-identification from counterfactuals remain adjacent objectives rather than part of the three-family comparison.
- Corpus: The review uses an analytical systematization of 25 core studies, with a narrower attack-path unit than broader surveys or explanation-type taxonomies.Its scope prioritizes operational assumptions and quantitative evidence rather than a PRISMA-style review or synthetic ranking.
3 Background and Terminology
The paper treats explanations as artifacts whose content and computation resources can cross a trust boundary. It defines marginal explanation leakage by comparing the same attacker with and without the explanation signal.
- Explanation Outputs: The explainer may use reference data, training examples, gradients, or a counterfactual generator, making those resources part of the disclosure analysis.The explanation is not treated as a single uniform output type.
- Explanation Outputs: Explanation mechanisms expose different artifacts, including importance vectors, local approximations, counterfactuals, influence information, and deep-network attribution signals.The security implications depend on the mathematical object exposed and the resources used to compute it.
- Marginal Leakage: Marginal leakage compares attack metric M with predictions and explanations against the same adversary using predictions, auxiliary knowledge, and query budget without explanations.For error metrics, the sign is reversed, and the remaining threat model must stay fixed.
- Marginal Leakage: Many studies alter the optimizer, query strategy, or architecture alongside the explanation channel, making reported XAI gains difficult to isolate.The SoK contrasts such evaluations with cleaner prediction-only versus prediction-plus-explanation comparisons.
4 System and Threat Model
The threat model separates model knowledge from interface disclosure and represents attacks through goals, outputs, explanation acquisition, auxiliary knowledge, and query behavior. It identifies five recurring explanation-acquisition paths.
- Model Knowledge and Interface Disclosure: Model knowledge and interface disclosure are distinct: hidden parameters can coexist with a gradient oracle that provides near-white-box learning value.Therefore, “black-box” alone does not specify the adversary’s effective capability.
- Model Knowledge and Interface Disclosure: The attack setting is represented by G, K_M, O, A_E, K_D, and Q for goal, model knowledge, prediction output, explanation acquisition, auxiliary knowledge, and query behavior.These dimensions make attacks comparable only when those relevant to the goal are aligned.
- Explanation-Acquisition Paths: Five acquisition paths are target-released, attacker-derived, secondary disclosure, privileged access, and released global artifact.The taxonomy separates how an explanation is obtained from conventional model knowledge and prediction access.
- Explanation-Acquisition Paths: Attacker-derived explanations can preserve attacks after an explanation endpoint is disabled, while secondary disclosure treats another user’s explanation report as sensitive derived data.These paths shift the control point away from a simple target explanation endpoint.
- Identity and Trust Boundaries: Threat models should identify the caller, data subject, and adversary because these roles may differ, especially when an attacker calibrates on its own queries but reconstructs another person’s record.Stating only “black-box” omits the trust boundary and who is being explained.
5 Model Extraction
Explanation-assisted extraction spans direct parameter recovery, boundary discovery, adaptive querying, and auxiliary training targets. Its effectiveness depends on the information preserved by the explanation, modality, threat model, and query cost rather than on a single XAI category.
- 5.1 Gradient Explanations: 10 gradient queries reach 95% MNIST accuracy versus 1,000 label queries, showing that raw derivatives can sharply reduce extraction cost.For a linear model, one input-gradient query identifies the weight vector; for a two-layer ReLU network, recovery requires O(hlogh) gradient queries under stated assumptions.
- 5.1 Gradient Explanations: MEGEX reaches 81.13% surrogate accuracy at 2.5M CIFAR-10 queries versus 20M for DFME, and reaches 91.61% at 20M.MEGEX consumes Vanilla Gradient explanations directly and is equivalent to the corresponding white-box data-free distillation update under its formulation.
- 5.1 Gradient Explanations: Explanation processing matters: SmoothGrad and Integrated Gradients help on Fashion-MNIST and SVHN but are not equally effective on CIFAR-10.Leakage depends on how faithfully the released explanation preserves the optimization signal required by the attacker.
- 5.2 Counterfactual Explanations: 93% Adult fidelity at 1,000 queries shows that counterfactuals can expose decision-boundary information even with only feature-marginal knowledge.Multiple and diverse counterfactuals further improve extraction, while structured-model attacks can use locally optimal counterfactuals for exact functional recovery.
- 5.3 Feature-Importance-Guided Extraction: Feature importance can guide active traversal, while explanation losses provide auxiliary training targets for reproducing predictions or decision logic.AUTOLYCUS reports high similarity with 100 queries for Breast Cancer logistic regression, but image explanations and complex trees have important limitations.
- Evaluation: Extraction success should be compared using fidelity, accuracy, and query cost because these metrics capture different objectives and can diverge.The review recommends anytime fidelity curves and fixed-fidelity query comparisons, while accounting for internal model evaluations in model-agnostic explanations.
6 Membership Inference
Explanations support membership inference through explanation statistics, recourse geometry, and explanation-guided perturbations. Reported leakage is heterogeneous across models, data geometries, recourse methods, and attacker operating points.
- 6.1 Explanation Statistics: Feature-explanation statistics distinguish members from non-members, with learned attacks on the full explanation vector slightly stronger than variance-only thresholds.Integrated Gradients and LRP are also reported as leaking membership information.
- 6.1 Explanation Statistics: Global surrogate explanations can remain more vulnerable to membership inference than the original black-box model after release.This channel is governed by how the global explainer approximates or overfits the target rather than by repeated local explanation queries.
- 6.2 Counterfactual Distance: Counterfactual-distance leakage is heterogeneous: HELOC CCHVAE recourse reaches AUC 0.679 and 5.13% TPR at 1% FPR, while several Adult and Diabetes settings remain near random.The existence of a counterfactual API therefore does not imply uniformly strong membership inference.
- 6.2 Counterfactual Distance: 77% accuracy and recall with counterfactuals versus 58% without them shows that counterfactual information can amplify EEG membership leakage.Because the study uses a different evaluation protocol and is a preprint, the result is evidence of amplification rather than a direct ranking against other attacks.
- 6.3 Explanation-Guided Robustness: Explanation-guided perturbation achieves roughly 0.915–0.931 AUC and 2.2–3.7% TPR at 0.1% FPR across seven methods in representative CIFAR-100 experiments.An explanation-free segmentation and perturbation strategy also improves substantially, so the raw gain cannot be attributed entirely to XAI.
- 6.4 Low-FPR Evaluation: Low-FPR metrics, coverage, and explanation-specific ablations are necessary because average scores can hide vulnerable subpopulations or attribution to the attack algorithm.Evaluations should report TPR at fixed low FPR, alongside AUC and balanced accuracy, and distinguish population-wide risk from high-risk subset identification.
7 Model Inversion
Model inversion uses explanations to reconstruct private query inputs, training information, or selected hidden attributes. The attack surface depends on explanation structure, acquisition path, model family, and whether the target record was used for training.
- 7.1 Spatial Explanations: Spatially structured saliency and CAM explanations improve image reconstruction because they preserve input location information.U-Net architectures using prediction-plus-explanation inputs outperform flattened-explanation processing across reconstruction and downstream attack metrics; richer class-specific CAM sets increase risk.
- 7 Model Inversion: Explanation attacks can reconstruct query records from breached, intercepted, or socially shared reports, making secondary disclosure a distinct threat model.This query-record risk does not require model memorization and can persist through surrogate explanations when the target does not directly expose them.
- 7.2 Shapley-Value Inversion: Shapley explanations can reconstruct a user’s private tabular features even when the target record was not in the training set.Luo et al. report feature inference across cloud services and multiple model families, with neural networks and highly important features generally more vulnerable.
- 7.2 Shapley-Value Inversion: Exact Shapley values permit error-free input reconstruction for the studied linear-regression model, whereas approximate settings become model- and implementation-dependent.Experiments on Adult, Bank Marketing, and Credit Card Client illustrate why SHAP privacy cannot be assessed independently of the target model and implementation.
- 7.3 Attribute Inference as Partial Inversion: Explanations alone outperform prediction-only baselines for sensitive-attribute inference in the studied settings, including when the sensitive attribute is censored from model inputs.Proxy features can make explanations distinguishable by hidden attributes, while adding predictions to explanations does not consistently improve inference.
- 7.3 Attribute Inference as Partial Inversion: Attribute inference reconstructs selected hidden components, so its evaluation should use prior-normalized accuracy rather than full-record image or tabular reconstruction metrics.The paper recommends task-specific privacy measures alongside fidelity metrics because average reconstruction quality can obscure recovery of especially sensitive features.
8 Cross-Attack Analysis
Across attack families, explanation risk is conditional on the information preserved, how adversaries acquire it, and the target’s modality and complexity. Cross-paper comparisons remain limited because metrics, query units, threat models, and explanation contributions are not standardized.
- 8.1 How Explanations Change the Attack Surface: Acquisition path determines the security boundary: explanations may be target-released, attacker-derived, secondarily disclosed, privileged, or globally released.Disabling an explanation endpoint does not stop attacker-derived signals, and sanitizing a released artifact does not necessarily prevent attacks using independently derived explanations.
- 8 Cross-Attack Analysis: Risk is not uniformly ordered across explanation families: leakage increases when a signal preserves an attack-relevant statistic, but richer explanations do not always leak more.Raw gradients, diverse counterfactuals, and multiple CAMs can strengthen attacks, while processed gradients or perturbation-based explanations can reduce leakage in some settings.
- 8.2 Factors Affecting Leakage: Model complexity, auxiliary knowledge, and modality shape attack effectiveness, including query savings, coverage needs, algebraic inversion, and architecture-dependent gains.Complex targets may reduce query savings or require broader coverage, while exact Shapley values, counterfactuals, and graph explainers behave differently across settings.
- 8.3 Interpreting Reported Results: Reported results are not a cross-paper leaderboard because extraction, membership, and inversion studies use different metrics and inconsistent query units.Evaluation should separately report external requests, response dimensionality and precision, explanation alternatives, and internal target-evaluation cost where measurable.
- 8.3 Interpreting Reported Results: Explanation-assisted attack performance does not by itself establish marginal leakage from explanations because optimizers, adaptive queries, attack models, and auxiliary data may also improve results.An explanation-free ablation shows that both explanations and a multi-query perturbation procedure can contribute to final membership-inference performance.
- 8 Cross-Attack Analysis: Future evaluations should specify the protected asset, attack output, model knowledge, explanation artifact and acquisition path, auxiliary data, query costs, prediction-only baseline, and remaining defenses.These fields make threat-model differences and disclosure channels explicit rather than collapsing them into black-box or white-box labels.
9 Defenses
Defenses operate at different layers and protect different assets, so no single mechanism covers all explanation-acquisition paths. The evidence favors matching controls to the attack family, acquisition path, and intended explanation utility.
- 9.1 Training-Time Defenses: DP-SGD provides the strongest general protection for individual training records, but it does not automatically protect query records or model confidentiality.Explanations derived from a differentially private model inherit its formal training-data guarantee, while query-record protection may require separate mechanisms.
- 9.2 Explanation-Layer Defenses: Explanation perturbation must remove attack-relevant statistics while retaining the utility property required by the recipient.Evaluated approaches include private local explanations, noisy feature explanations, clipping, masking, and calibrated noise.
- 9.2 Explanation-Layer Defenses: 0.49 percentage-point and 0.65-point changes in SVHN and CIFAR-10 attack accuracy show that simple explanation perturbation may leave extraction-relevant structure intact.Defense evaluation should pair attack success with explanation utility rather than relying on visual degradation alone.
- 9.2 Explanation-Layer Defenses: Private counterfactuals reduce knowledge-distillation extraction toward no-counterfactual baselines, while more realistic and diverse counterfactuals improve extraction.Limiting recourse information can therefore conflict with actionability.
- 9.3 Prediction-Output Defenses: MemGuard leaves explanation-assisted membership inference effective, with AUC 0.872–0.885 across LIME, SHAP, Grad-CAM, and SmoothGrad.The result shows that protecting confidence scores does not protect parallel explanation channels.
- 9.4 Service and System Controls: Query controls, report protections, and path-specific defenses are all required because low-query or secondary-disclosure attacks can evade a single service-boundary mechanism.Controls include budgets and anomaly detection for adaptive queries, sensitive-data handling for stored reports, and access restrictions across all five paths.
10 Research Gaps and Open Problems
Open problems center on measuring marginal and compositional leakage under matched conditions, extending coverage beyond mature image and tabular attacks, and evaluating adaptive acquisition. The field also needs interface-aware benchmarks that compare explanation channels without treating their information content as equivalent.
- 10 Research Gaps and Open Problems: Matched experiments should hold the target, auxiliary knowledge, attack algorithm, and query budget fixed while varying only the explanation channel.Evaluation should measure both individual outputs and their composition, including o(x), each e_i(x), and combined channels.
- 10 Research Gaps and Open Problems: Query-record privacy requires adjacency or sensitivity notions for explained inputs and private reference data, not only for training datasets.Several studies show that explanations can leak records absent from training.
- 10 Research Gaps and Open Problems: Audio, sequential, text, and LLM explanation interfaces remain sparsely covered, while repeated noisy outputs and surrogate-derived explanations challenge fixed-attacker evaluations.These gaps call for modality-specific and adaptive evaluation rather than extrapolation from image and tabular results.
- 10 Research Gaps and Open Problems: Interface-aware benchmarks should compare label-only, score, feature-importance, and counterfactual interfaces under matched auxiliary knowledge.Such benchmarks can measure marginal leakage and operational cost without assuming equivalent information across responses.
11 Related Surveys and SoKs
Prior surveys organize privacy attacks by explanation type and review privacy-preserving XAI, while broader SoKs cover XAI security and machine-learning privacy. This paper’s narrower contribution is an operational comparison based on explanation acquisition and protected assets.
- 10 Related Surveys and SoKs: Existing surveys cover explanation-specific privacy attacks, privacy-preserving XAI, broader XAI security, and general machine-learning privacy.The paper distinguishes its contribution by tracing explanation signals to attack mechanisms and protected assets through acquisition paths.
12 Conclusion
Across 25 core studies, explanation privacy depends on the information path rather than on explanation method alone. The paper concludes that attack assumptions, marginal leakage, and defense coverage should be evaluated end to end.
- 12 Conclusion: Explanations reduce extraction cost, expose membership signals, and support reconstruction of private inputs and attributes, but no explanation family is uniformly unsafe.Security claims should identify the protected asset, output channels, acquisition path, auxiliary knowledge, and query budget.
A Open Science
The paper systematizes published literature rather than introducing a new attack implementation, dataset, trained model, or benchmark. Its evidence comes from cited papers and classifications in Tables 1–5.
- A Open Science: The paper provides a literature systematization, not a new attack implementation, dataset, trained model, or benchmark.Its evidence is drawn from cited papers and study classifications reported in Tables 1–5.
- A Open Science: The authors report only information already present in the literature while emphasizing threat-model limits, defensive coverage, and deployment safeguards.This is intended to help identify disclosures requiring protection and avoid overclaiming defenses whose guarantees do not match the protected asset.