PaperlayerView exact revision on arXiv

Source-linked AI summary

ExplainGuard: A Zero Trust Framework for Post-Hoc Explanation Integrity Guarantees in Blackbox XAI Models

Maraz Mia, Shovan Roy, Mir Mehedi A. Pritom, Maanak Gupta

arXiv:2608.21803v1cs.CRcs.AI

TL;DR

Post-hoc XAI auditing can be compromised by adversarial manipulation when auditors are implicitly trusted. ExplainGuard applies zero-trust verification to explanation generation and reports detection of output shuffling, scaffolding OOD, and biased sampling attacks.

  • Problem

    Post-hoc explanations lack security guarantees, while adversarial manipulation can compromise their trustworthiness and deceive non-technical stakeholders.

  • Method

    ExplainGuard embeds zero-trust, dynamic-policy verification in the XAI pipeline, combining behavioral monitoring with semantic axiomatic checks before explanation release.

  • Results

    0.105244 detected output shuffling, 0.111597 flagged scaffolding OOD, and stricter margins trapped biased sampling with pKS = 0.1124 and pWald = 0.1614.

  • Takeaways & Limitations

    The framework transforms XAI auditing from an assume-trusted process into a verification-based operation for maintaining explanation integrity.

  • Takeaways & Limitations

    The stated detection scenario assumes a seed dataset and black-box input-output access for applying the verification checks.

Abstract

from arXiv · show

As machine learning (ML) models are increasingly deployed in high-stakes environments, explainable AI (XAI) methods like SHAP and LIME have become essential for regulatory compliance and trust. However, the current auditing paradigm relies on an implicit "chain of trust" where third-party auditors are assumed to be trusted. Recent research demonstrates that this assumption is flawed and adversarial auditors can manipulate XAI explanations through manipulation attacks such as output shuffling or scaffolding out-of-distribution (OOD) to conceal model biases while maintaining high prediction accuracy aiming for fairwashed explanation. In this paper, we introduce a novel defense framework, ExplainGuard, that leverages a Zero-Trust architecture (ZTA) design to be incorporated within the XAI explanation supply chain and ensures the integrity of the generated explanation. This framework would help us to replace the ambiguous default assumption of "auditor is trustworthy," with a continuous "verify-then-trust" approach. Our design architecture establishes a Policy Decision Point (PDP) that enforces three distinct pillars of verification before any explanation is released to the user: (1) asset integrity via behavioral fingerprint to detect model substitution, (2) semantic validity using axiomatic consistency checks to reject mathematically impossible explanations, and (3) feature faithfulness verification utilizing a ranking stability approach with minimal computational overhead. Finally, we evaluate how ExplainGuard can effectively neutralize state- of-the-art explanation manipulation attacks while transforming the auditing process into a verifiable operation.

I. INTRODUCTION AND MOTIVATION · II. PROBLEM FORMALIZATION AND BACKGROUND · A. Mathematical Formulation

Black-box models in critical sectors create an urgent need for explainability, but adversarial manipulation and blind trust in the XAI supply chain can conceal model bias. ExplainGuard addresses this problem with zero-trust verification of explanation integrity and formalizes adversarial wrapper models that preserve predictive accuracy while altering explanations.

  • I. INTRODUCTION AND MOTIVATION: Black-box ML and deep-learning deployment in cybersecurity, finance, and healthcare has created an urgent need for explainability and transparency.XAI emerged to interpret AI-driven decision outcomes.
  • I. INTRODUCTION AND MOTIVATION: XAI methods include model-agnostic and model-specific mechanisms, with examples such as SHAP, LIME, Integrated Gradients, and Grad-CAM.Model-agnostic methods interpret any black-box system, whereas model-specific methods target particular architectures.
  • I. INTRODUCTION AND MOTIVATION: Adversarial attacks can deceive post-hoc explanations, obscure underlying model biases, and exploit stakeholders’ focus on global model behavior.Because post-hoc explanations approximate internal workings, explanation attacks exacerbate that approximation and can mislead non-technical stakeholders.
  • I. INTRODUCTION AND MOTIVATION: Blind trust in the explainability supply chain is vulnerable because malicious third-party auditors can manipulate generated explanations despite their intended verification role.The paper motivates replacing the assume-trusted auditing model with independent, continuously verified auditing.
  • I. INTRODUCTION AND MOTIVATION: ExplainGuard embeds zero-trust architecture within the XAI pipeline to ensure explanation integrity through dynamic policy enforcement rather than implicit trust.Its stated design replaces the traditional assume-trusted auditing model with a verification-based model.
  • I. INTRODUCTION AND MOTIVATION: ExplainGuard introduces behavioral fingerprinting to detect manipulated explanations and a method to identify model substitution or adversarial wrapper-model attacks.These are listed as core framework contributions.
  • A. Mathematical Formulation: An adversarial wrapper model f′ preserves approximately the prediction behavior of an unfair black-box model f while producing a different explanation for at least one input.Formally, ∃x ∈ X such that g(x, f) ≠ g(x, f′) while ∀x ∈ X, f(x) ≈ f′(x).

B. Attacks on XAI System … 1) Stakeholder Roles and Trust Perimeters:

ExplainGuard addresses manipulation of post-hoc XAI explanations by modeling the auditor as untrusted and requiring continuous verification of explanation integrity and provenance. Its zero-trust design separates explanation generation from verification and defines explicit stakeholder trust boundaries.

  • B. Attacks on XAI System: XAI attacks comprise prediction-preserving attacks that manipulate explanations, dual attacks that alter predictions and explanations, and explanation-preserving attacks that leave explanations unchanged.
  • B. Attacks on XAI System: Malicious auditors can deceive target XAI methods by applying wrapper or scaffold models to the original black-box model, including output-shuffling attacks.
  • A. Design Overview: ExplainGuard secures the XAI supply chain with a zero-trust architecture that treats generated explanations as untrusted artifacts requiring continuous integrity and provenance verification.
  • A. Design Overview: The architecture follows NIST zero-trust principles, using dynamic policy-based access, enforced authentication and authorization, and separate control and data planes.
  • B. XAI Pipeline: Roles, Assumptions, and Adversarial Goals: ExplainGuard targets a sophisticated, potentially malicious model auditor and defines the audit environment through stakeholder interactions and technical constraints.
  • 1) Stakeholder Roles and Trust Perimeters:: The model developer and end user are mutually trusted; the developer provides the original black-box model f to the end user, typically through MLaaS.
  • 1) Stakeholder Roles and Trust Perimeters:: The external model auditor is untrusted, has black-box access to f, and uses a model-agnostic explainer under limited query time.
  • 1) Stakeholder Roles and Trust Perimeters:: A trusted third-party verifier is responsible for validating the integrity of the audit.

2) Technical Assumptions: … 1) The Untrusted Zone (The Auditor):

ExplainGuard assumes deterministic, model-agnostic auditing of tabular binary classifiers in a Python environment, while treating the auditor as adversarial and capable of model substitution, explanation forgery, fairwashing, or malicious payload submission. Its protocol registers models and fingerprints, authorizes request-specific access, and encrypts structured explanation packages for verification.

  • 2) Technical Assumptions:: Audits use sample input vector x for binary classification, with deterministic prediction probabilities and class labels produced by f.The framework focuses on tabular models implemented in Python.
  • 2) Technical Assumptions:: The audit is motivated by transparency needs of non-technical stakeholders or regulatory obligations requiring unbiased external evaluation of black-box models.The stakeholder is denoted E, the developer Dev, and the auditor A.
  • 3) Adversarial Objectives:: The auditor’s primary objective is Explanation Fairwashing: generating deceptive explanations ΦA that mislead both the evaluator E and verifier V.Two primary attack vectors are considered.
  • 3) Adversarial Objectives:: Model substitution uses an adversarial wrapper f′ that mimics f’s predictions while suppressing sensitive features or producing biased importance scores.This attack targets the apparent model identity rather than only the submitted explanation.
  • 3) Adversarial Objectives:: Explanation forgery submits feature importance values unrelated to f, including randomized, manually manipulated, or corrupted-explainer outputs that conceal the model’s logic.The forged values are presented as if they were derived from the audited model.
  • C. Operational Protocol and Key Management: During registration, the model owner provides seed inputs to generate secret explanation and OOD fingerprint sets Ic = {Sexpl ∪ Sood} and establishes a cryptographic key pair.Fingerprint generation uses Algorithm 1 in Appendix C.
  • C. Operational Protocol and Key Management: Authorization issues the auditor a unique request-specific API key linked to f and Request ID Rid, while the auditor receives its own cryptographic key pair.The auditor is registered by the primary party before submission.
  • 1) The Untrusted Zone (The Auditor):: As an untrusted external entity, A uses model-agnostic post-hoc methods on black-box f′ and submits SA = {Eobj, ΦA, x, B}, potentially containing fairwashed explanations or malicious RCE payloads.The package is described as a structured submission containing the explainer object, attributions, input, and B.

2) Policy Enforcement Point (PEP): … 1) Pillar I: Black-Box Asset Integrity:

ExplainGuard places a zero-trust gateway and trust engine around the explanation supply chain, releasing explanations only after hierarchical verification. Its first pillar checks black-box asset integrity through behavioral, evaluation-mode, OOD, and sampling-distribution tests.

  • 2) Policy Enforcement Point (PEP):: The PEP acts as a secure gateway between the untrusted auditor A and environment E, preventing explanations from bypassing enforcement.It intercepts the explanation request and routes it to an isolated explainer container.
  • 2) Policy Enforcement Point (PEP):: PEP micro-segmentation isolates the explainer service and restricts its outbound communication to the PEP.This containment limits the explainer’s interactions with other components.
  • 3) Policy Decision Point (PDP):: The PDP serves as the architecture’s trust engine, computing a binary trust score T ∈{0, 1} from sanitized PEP artifacts.It accesses the ground-truth store containing model f and secret fingerprint set I_c.
  • E. The Verification Protocol: The PDP applies a Verify-Then-Trust policy, releasing an explanation only when it satisfies all three verification pillars.The supplied protocol specifies the three-pillar requirement but does not enumerate the remaining pillars in this excerpt.
  • 1) Pillar I: Black-Box Asset Integrity:: Pillar I verifies model identity and structural honesty using input-output observations against model substitution, output shuffling, and explainer-conditioned scaffolding attacks.Behavioral fingerprinting queries secret inputs I_c and compares f(I_c) with pre-recorded ground truth Y_GT.
  • 1) Pillar I: Black-Box Asset Integrity:: Batch-invariance checking detects output shuffling by testing whether predictions remain identical under individual and batch evaluation.Honest models satisfy this invariance, whereas output-shuffling models violate it.
  • 1) Pillar I: Black-Box Asset Integrity:: The explainer-conditioned OOD audit compares explainer-style perturbations S_expl(x) with non-explainer OOD perturbations S_ood(x) to detect scaffolding.An honest model has Δ(x) ≈0, while a scaffolded model shows systematic discrepancy from different internal predictors.
  • 1) Pillar I: Black-Box Asset Integrity:: Biased-sampling detection combines a two-sample KS test and Wald test on model outputs from seed data X_seed and background samples B.Using two-sided p-values p_ks and p_wald, rejection at user-defined significance level δ_sample flags statistically significant output shifts as potentially biased.

2) Pillar II: Semantic Validity:

Pillar II verifies that explanations are mathematically consistent with model outputs without requiring internal gradients. It applies efficiency and Null-Player axioms to detect omitted prediction shifts and fabricated importance for inactive features.

  • Semantic validity: Semantic validation checks explanations against model outputs without requiring access to internal gradients.The policy targets mathematical consistency in additive attribution methods such as KERNELSHAP and LIME.
  • Efficiency (Local Accuracy): The efficiency axiom requires feature attributions plus the baseline to equal the model prediction.For SHAP, ϕ0 + Σϕi = f(x), while LIME uses ϕ0 + Σϕixi ≈ f(x).
  • Null-Player Axiom: The Null-Player Axiom requires negligible attribution when a feature has negligible marginal contribution or does not change model output.Formally, σi < δ′ implies |ϕi| < δ2; for inactive features, the attribution must be zero.
  • Semantic validity: These checks ensure the auditor accounts for the entire prediction shift and assigns no importance to inactive features.They prevent both omission and fabrication of causal effects.

3) Pillar III: Feature Faithfulness Verification:

ExplainGuard verifies feature faithfulness by testing whether an auditor’s top-ranked attributions cause greater prediction degradation than random feature rankings. A self-adaptive significance threshold rejects explanations whose rankings are statistically indistinguishable from noise, detecting fairwashing.

  • Feature Faithfulness Verification: The protocol ranks features by descending absolute attribution magnitude and evaluates whether the auditor’s top-k features have greater behavioral impact than a random baseline.It uses comparative deletion-based evaluation to test whether identified influential features reflect the model’s causal logic.
  • Feature Faithfulness Verification: For k = 1, . . . K, the protocol replaces the top-k features with values drawn from validated background distribution B and measures prediction degradation.Directional degradation accounts for signed SHAP/LIME attributions by measuring the magnitude of output change relative to the original prediction.
  • Feature Faithfulness Verification: Faithfulness is assessed by comparing the auditor’s deletion-curve AUC with a random-baseline AUC computed from M random feature permutations.The comparison yields a faithfulness gap representing the discriminative margin between the auditor’s ranking and random rankings.
  • Feature Faithfulness Verification: An explanation is faithful when the faithfulness gap exceeds pooled uncertainty by a significant margin under a self-adaptive threshold.The threshold can use z = 1.96 for 95% confidence and does not require explicit specification of δM.
  • Feature Faithfulness Verification: Failure indicates that the auditor’s top-ranked features are statistically indistinguishable from noise, thereby detecting fairwashing through irrelevant feature assignments.The mechanism is intended to expose attempts to distract from biased features by assigning high importance to irrelevant ones.

IV. EXPLAINGUARD COMPLIANCE AND LOGICAL PROOF OF VERIFICATION · A. Compliance with NIST Zero Trust Standards

ExplainGuard is presented as a functional implementation of NIST SP 800-207 that maps Zero Trust tenets onto architectural components protecting the explainability supply chain. Its design uses ephemeral isolation and dynamic, evidence-based policy evaluation before explanations reach users.

  • A. Compliance with NIST Zero Trust Standards: ExplainGuard operationalizes NIST SP 800-207 rather than remaining only a theoretical security framework.The framework maps foundational Zero Trust tenets to architectural components.
  • A. Compliance with NIST Zero Trust Standards: The architecture maps foundational Zero Trust principles directly onto the explainability supply chain.This mapping is intended to maintain resilience against adversarial manipulation.
  • A. Compliance with NIST Zero Trust Standards: Tenet 3, Per-Session Access, is implemented through ephemeral, micro-segmented containers at the Policy Enforcement Point.These containers ensure trust does not persist across explanation submissions.
  • A. Compliance with NIST Zero Trust Standards: The explainer object remains strictly isolated from the end-user until verification is complete.This isolation is part of the Policy Enforcement Point’s per-session access design.
  • A. Compliance with NIST Zero Trust Standards: Tenet 4, Dynamic Policy, is implemented through a multi-factor Policy Decision Point.The PDP evaluates trust score T using real-time evidence across three verification layers.
  • A. Compliance with NIST Zero Trust Standards: The Policy Decision Point first performs a Black-Box Asset Integrity audit of the black-box model’s observable state and identity.This audit constitutes Pillar I of the verification process.

B. Trust Decision and Feedback … B. Key Insights •

ExplainGuard uses a fail-closed trust decision that releases explanations only when all three verification pillars succeed. In COMPAS experiments, the framework detected output shuffling, scaffolding OOD behavior, biased sampling, and randomized attributions through targeted integrity, validity, and faithfulness checks.

  • B. Trust Decision and Feedback: The final trust decision is the logical conjunction of all three pillar outcomes, rejecting any explanation that fails model identity, mathematical consistency, or feature faithfulness.This implements a strict fail-closed policy.
  • B. Trust Decision and Feedback: If T = 1, the PDP issues a success token and the PEP releases the decrypted verified explanation; if T = 0, the PEP drops the submission and terminates the session.The trust score incorporates structural and distributional integrity, semantic validity, and feature-rank correlation checks.
  • V. EXPERIMENTAL CASE STUDY AND EVALUATION: The evaluation used an 8-feature COMPAS recidivism task with an XGBoost classifier achieving an F1 score of 0.65 under a standardized black-box wrapper.The dataset was partitioned into training, testing, and audit subsets, and the model was serialized as a standardized JSON artifact.
  • A. Comparative Audit Results Across Threat Models: The comparative audit evaluated a clean baseline alongside scaffolding OOD, output shuffling, biased sampling, and random-control scenarios.The attacks targeted protected or sensitive features including age, sex, and race_African-American, using distinct background-data seeds and manipulation strategies.
  • B. Key Insights •: 0.105244 exceeding the threshold detected output-shuffling attacks through the Vid and Vos checks.These checks address elevated metric values associated with batch-level attribution-rank permutations.
  • B. Key Insights •: 0.111597 against the < 0.080 threshold flagged out-of-distribution branch-switching behavior in scaffolding probes through Vsc.The result identifies the scaffolding OOD manipulation scenario.
  • B. Key Insights •: Stricter hardening to a ≥0.20 margin rejected biased sampling despite pKS = 0.1124 and pWald = 0.1614, bypassing lenient p ≥ 0.025 thresholds.The defense traps warped background distributions produced by optimization-based sampling.
  • B. Key Insights •: Randomized SHAP and LIME attributions were caught using axiom thresholds of 1.137555 and 1.454038, and adaptive fidelity thresholds of 0.006 and 0.0106, respectively.The fidelity thresholds dynamically scale with internal statistical importance values (z) and standard errors.

VI. DISCUSSION, LIMITATIONS AND FUTURE DIRECTIONS · VII. CONCLUSION

ExplainGuard is presented as a zero-trust framework for systematically verifying externally produced XAI explanations without assuming that model developers or auditors are trustworthy. The paper also identifies scope, threat-model, dataset-representativeness, and explainer-generalization limitations while outlining the framework’s role in addressing explanation-integrity vulnerabilities.

  • VI. DISCUSSION, LIMITATIONS AND FUTURE DIRECTIONS: ExplainGuard is described as the first work to systematically verify external-auditor XAI results using a zero-trust architecture.The design targets cloud or server-based environments where neither the model developer nor auditor is assumed fully trustworthy.
  • VI. DISCUSSION, LIMITATIONS AND FUTURE DIRECTIONS: Wrapper-based scaffolded models remain detectable through black-box input–output inconsistencies using the Batch-Invariance Check and OOD Stability audit.The tests use a seed dataset Xseed and verification rules in Eq. 2 and Eq. 3.
  • VI. DISCUSSION, LIMITATIONS AND FUTURE DIRECTIONS: For LIME Vax, threshold δ1=0.4 is adopted because surrogate-model approximation noise is relevant when the underlying COMPAS model has an F1 score of 0.65.The passage also recommends |F| as a standard rule of thumb for selecting top-k features.
  • VI. DISCUSSION, LIMITATIONS AND FUTURE DIRECTIONS: The evaluation is limited to tabular data and primarily SHAP and LIME, leaving broader cross-explainer, modality, and non-additive or gradient-based generalization open.The covered attacks are scaffolding and output shuffling under a black-box setting, not all explanation-forgery strategies.
  • VI. DISCUSSION, LIMITATIONS AND FUTURE DIRECTIONS: The black-box threat model excludes backdoors, direct weight manipulation, data poisoning, membership inference, white-box attacks, and inherently interpretable models.These threats are characterized as complementary to the explanation-integrity problem studied here.
  • VI. DISCUSSION, LIMITATIONS AND FUTURE DIRECTIONS: Detection robustness depends on the representativeness of the seed dataset Xseed, because inadequate coverage may allow adversarial evasion.The passage also notes that performance overhead is a consideration, although the supplied text is truncated.
  • VII. CONCLUSION: The conclusion identifies post-hoc explanation security guarantees as a critical gap and reports that wrapper-based fairwashing, biased sampling, and scaffolding can compromise trustworthiness without modifying the underlying model.ExplainGuard is introduced as a zero-trust architecture–based framework with per-session access control and dynamic policy verification.

APPENDIX A. Need for an Explainer Object · B. Defensive Serialization and Object Sandboxing · C. Algorithm for OOD Sample Generation

The appendices define the evidence, serialization safeguards, and fingerprint-generation procedure underlying ExplainGuard’s zero-trust verification. They require the original explainer object, prohibit executable serialization, and generate explainer and OOD fingerprints through deterministic–stochastic perturbations.

  • APPENDIX A. Need for an Explainer Object: ExplainGuard requires auditors to submit the original serialized explainer object Eobj used to generate feature attributions ΦA.The object provides technical evidence of the model manifold probed during explanation and enables verification against the legitimate black-box model f.
  • APPENDIX A. Need for an Explainer Object: Without Eobj, the verifier cannot authenticate whether reported attributions ΦA were derived from the legitimate black-box model f.
  • B. Defensive Serialization and Object Sandboxing: ExplainGuard prohibits executable serialization formats to mitigate Remote Code Execution and object injection attacks during explainer handling.The appendix identifies Python pickle as incompatible with Zero Trust because malicious byte streams can execute arbitrary code upon deserialization.
  • C. Algorithm for OOD Sample Generation: The fingerprinting algorithm takes a seed dataset Xseed, model f, sparsity parameter k, and samples per mode M as inputs.It outputs explainer fingerprints Sexpl and OOD fingerprints Sood, with f serving as a labeling oracle while fingerprint generation remains independent of f.
  • C. Algorithm for OOD Sample Generation: The procedure uses deterministic–stochastic OOD probing to explore model f by creating two perturbation sets from a seed set X.The algorithm’s stated output is explainer fingerprints Sexpl and OOD fingerprints Sood.
  • C. Algorithm for OOD Sample Generation: Explainer fingerprints Sexpl alternate between SHAP-style masking and LIME-style jittering.SHAP-style masking selects (d − k) indices, whereas LIME-style jittering selects k indices for perturbation.
  • C. Algorithm for OOD Sample Generation: OOD fingerprints Sood use non-explainer perturbations including group feature swapping with reference points and Gaussian feature noise.Group swapping draws a reference point xref from X \ {x}, while Gaussian noise is applied to k selected indices.

D. Feature Rank Analysis:

Feature-rank analysis shows that LIME and SHAP are consistent under clean conditions, while adversarial attacks destabilize LIME’s attributions and rankings. Random controls produce randomized rankings, whereas SHAP is comparatively more robust and captures a notable rank shift.

  • Clean baseline: Under the clean baseline, both LIME and SHAP exhibit consistent feature ranks.This establishes the expected ranking behavior without adversarial manipulation.
  • Adversarial scenarios: Adversarial scenarios cause severe fluctuations in LIME’s attribution values and feature rankings.The passage identifies these fluctuations as evidence that the attacks were successfully executed.
  • Random control: The random control case produces purely randomized feature attributions and rankings.This control reflects random attribution behavior rather than stable feature importance.
  • SHAP robustness: SHAP demonstrates greater overall robustness than LIME while appropriately capturing a notable rank shift.The supplied passage reports this comparative robustness and rank-shift behavior without specifying the affected features.
Loading 2608.21803v1…