Source-linked AI summary
XALIGN: A Human-centric Visual Analytics Approach to Understanding AI Explanations
Bahavathy Kathirgamanathan, Gennady Andrienko, Natalia Andrienko, Tatiana von Landesberger
TL;DR
Technical XAI evaluations do not capture whether explanations support human reasoning and domain-knowledge checks, especially when methods disagree. XALIGN compares rule-based and feature-attribution explanations of the same model; case studies and an expert evaluation found it useful for identifying influential features, with interpretation more immediate for users with some XAI background.
Problem
Technical explanation properties alone do not establish whether explanations are useful to human users, who also need to check model behavior against domain knowledge.
Source: §1 · pp. 2–3Method
XALIGN provides an interactive visual analytics workflow for comparing rule-based and feature-attribution explanations of the same model.
Source: §1 · p. 3Results
Across the case studies, agreement across methods offered a practical robustness signal, and experts found XALIGN useful for identifying important features.
Source: §4.3 · p. 20, §6 · p. 26Takeaways & Limitations
Agreement can strengthen evidence that an explanation is not an artifact of one technique, but does not establish that a learned relationship is correct or causal.
Source: §4.3 · p. 20The expert evaluation was preliminary, involved few participants, lacked a comparison with XALIGN, and included only one participant familiar with the domain data.
Source: §5.3 · p. 25
Abstract
from arXiv · showhide
Explainable AI (XAI) methods are usually evaluated for technical properties such as faithfulness and robustness. In high-stakes application domains, experts also need to check model predictions against established knowledge. Different explanation methods often disagree, and this disagreement is hard to see when each method is inspected separately. We propose XALIGN, a visual analytics workflow and interface that shows rule-based explanations and several feature attribution methods (TreeSHAP, KernelSHAP, and LIME) side by side for the same Random Forest model. Agreements and contradictions between the methods become visible signals that users can explore and compare with domain knowledge. We apply XALIGN in two case studies: the German Credit dataset and a meteorological dataset, in which events derived from ICON model output and Cloudnet observations are classified. In both, the views reveal discrepancies between the rule-based explanation and the attribution methods that a single method would not show. In an expert evaluation with three specialists, participants found XALIGN useful for identifying influential features and for avoiding over-trust in predictions. Linking rules to attributions was harder for the participant without an XAI background.
1. Introduction
Model-centric XAI evaluations emphasize technical properties, but high-stakes users also need explanations that support human reasoning and domain-knowledge checks. XALIGN brings rule-based and feature-attribution explanations together for interactive comparison.
1. Introduction
- Technical properties such as faithfulness and robustness do not fully capture whether explanations are meaningful or useful to people making high-stakes decisions.An explanation may be faithful to a flawed model or stable yet uninformative.Source: §1 · p. 2
- Attribution methods can be unstable or misaligned with domain expectations, while rule-based explanations can be intuitive but difficult to scale.These methods can offer complementary perspectives for identifying agreements and discrepancies.Source: §1 · p. 2
- Interactive visual analytics lets users compare explanations, investigate conflicts, and examine model behavior against domain knowledge.The framework is intended to help users question or incorporate model outputs rather than simply increase confidence in AI.Source: §1 · p. 3
- XALIGN compares rule-based and feature-attribution explanations of the same model so users can assess agreements and conflicts against domain knowledge.The comparison can inform refinement of data, rules, and explainer choice.Source: §1 · p. 3
- The paper extends earlier work with two case studies, expanded interface interactions, and an expert evaluation involving three specialists.The cases use German Credit and meteorological data.Source: §1 · p. 3
2. Related Work
Related work identifies limits in model-centric explanations, a gap in tools for comparing explanation types, and the need to evaluate interpretability with people and tasks in view. XALIGN combines rules, attributions, and visual analytics to support comparison and feedback-driven refinement.
2.1. Limitations of Model-Centric XAI
- SHAP and LIME provide feature importance, but their explanations may be difficult to interpret, unstable, or manipulable; rule-based explanations are intuitive but can be hard to scale and read.SHAP estimates importance using Shapley values [5], while LIME perturbs inputs and fits a local surrogate; instability concerns are documented [7], and adversarial manipulation is discussed.Source: §2.1 · p. 4
- Explanation trust involves scalability, human-alignment, stability, and actionability, and this work focuses on human-alignment, cross-explainer stability, and feedback-driven refinement.Human-alignment concerns whether people agree with the reasons models give for predictions; stability means similar inputs receive similar explanations [16].Source: §2.1 · pp. 4–5
2.2. Visual Analytics for human-aligned explanations
- Visual analytics tools commonly inspect model structure or behavior, while relatively few compare explanation approaches, and existing comparison systems focus on attribution methods.Prior systems include rule-visualization tools and attribution comparison systems.Source: §2.2 · p. 5
- XALIGN integrates rule-based representations, feature attributions, and visual analytics to support feedback-driven refinement of data and models.The approach builds on work linking domain knowledge to human alignment.Source: §2.2 · pp. 5–6
2.3. The Disagreement Problem and Comparative Visual Analytics
- The disagreement problem reflects common differences between explanation methods and a lack of streamlined ways for practitioners to understand them [4].Comparative tools support model exploration, but the cited systems do not jointly analyze rule-based and attribution-based explanations from the same model.Source: §2.3 · p. 6
2.4. Evaluating Interpretability
- Faithfulness and stability do not guarantee human alignment or actionability, so interpretability evaluation must account for human processing and task context [16].Evaluation approaches distinguish what is measured and for whom [1]; this expert evaluation is application-grounded because domain experts interpret explanations on a real dataset.Source: §2.4 · pp. 6–7
3. Human-Aligned Explanation Framework
XALIGN combines rule-based and attribution explanations in an interactive workflow so users can examine their agreements and conflicts against domain knowledge. The framework helps users assess explanations, but agreement does not establish that a model is correct, fair, or causally valid.
3. Human-Aligned Explanation Framework
- Rule-based explanations expose explicit conditions but can be unwieldy at scale, while attributions scale to complex models yet are hard to connect to domain knowledge and vary across methods or regions.Source: §3 · p. 8
3.1. Hybrid Visual Analytics Workow
- XALIGN trains one model, computes rules and attributions in parallel, integrates them, and lets users investigate conflicts with domain knowledge before refining data, rules, or explainer choice.The refinement loop returns to model training, but the case studies illustrate Steps 1–5; applying Step 6 is left for future work.Source: §3.1 · pp. 8, 14, Figure 1 · p. 9
- Feature rows and user-defined value bins form a heatmap of mean signed attribution, with overlaid square size marking rule counts and grey bars tallying distinct rules per feature.A rule count does not indicate how many instances pass through the corresponding splits, so a larger square does not imply a larger predictive effect.Source: §3.1 · pp. 12–13
- Agreement is judged visually, not quantified; mismatches in rule support, attribution sign, or magnitude mark regions for investigation rather than errors automatically removed.Source: §3.1 · p. 13
3.2. Supporting Human-Aligned and Trustworthy Explanations
- Cross-method agreement can increase confidence in a feature-value pattern, whereas disagreement exposes uncertainty; neither establishes correctness, fairness, or causal validity.Source: §3.2 · p. 14
4. Case Studies
The case studies use XALIGN to compare rule and attribution evidence across German Credit and meteorological classification, revealing both agreement and method-dependent uncertainty. The discussion treats convergence as stronger cross-method evidence, while disagreements and differences in rule counts versus attribution magnitude call for cautious interpretation.
4.1. German Credit
- In German Credit, coherent patterns appear for checking-account status and selected savings categories, while other regions vary in rule support or attribution strength.“No checking account” shows rule and attribution support; “< 0 DM” has attribution support without strong rule preference, and “< 100 DM” savings has cross-method agreement with substantial rule support.Source: §4.1 · pp. 15, 16, 18
- Duration and credit amount show the strongest numerical patterns, with higher-value bins generally positive and supported by larger rule boxes.Age is less consistently supported across explainers, while remaining numerical variables have comparatively small and scattered attributions.Source: §4.1 · p. 17
4.2. Meteorology Dataset
- Meteorological attributions share a general pattern across TreeSHAP, KernelSHAP, and LIME, but rule–attribution correspondence is less clear than in German Credit.Cloud cover, temperature, and ice-water-content distribution differentiate the classes; rule counts can be high for a feature even when attribution is pale because counts reflect use in splits, not prediction change.Source: §4.2 · pp. 19–20
4.3. Discussion
- Cross-method agreement on German Credit patterns provides stronger evidence that an explanation is not an artifact of one technique, while contradictions flag regions for closer investigation.The authors caution that convergence does not establish correctness or causality; disagreements may reflect method assumptions, nonlinear boundaries, interactions, limited data support, or local instability.Source: §4.3 · pp. 20–21
- Rule counts indicate how often features partition classes, whereas attributions indicate how features move predictions toward or against a class.The three-class meteorology task requires class-specific attribution directions, unlike the binary German Credit plots’ unambiguous positive direction.Source: §4.3 · p. 21
5. Expert Evaluation
A qualitative formative evaluation with three specialists assessed whether XALIGN supports interpretation of model predictions and meteorological features. Participants were positive about feature identification and avoiding over-trust, while reported understanding of why the model made a prediction varied.
5. Expert Evaluation
- The evaluation was a qualitative formative study with three XAI and/or meteorology specialists, using individual semi-structured think-aloud interviews.The authors characterize it as initial feedback rather than evidence of long-term use.Source: §5 · p. 23
5.1. Study Design and Participants
- Participants represented visualization expertise in attribution and rule-based models, attribution visualizations, and meteorological ICON data.Recruitment reflected the tool’s intended XAI-practitioner and domain-scientist user groups.Source: §5.1 · p. 23
5.2. Evaluation Results
- All three participants agreed they could identify prediction-driving features and that XALIGN would help them avoid over-trusting a model.All strongly agreed they would use the visualization on other datasets, but only one agreed they understood why the model made a prediction; two were unsure.Source: §5.2 · p. 24
Usability and Utility
Participants found XALIGN useful for identifying influential features and avoiding over-trust in predictions, but connecting attribution results to extracted rules was harder for users less familiar with XAI.
Usability and Utility
- All three participants found XALIGN useful for identifying features that may drive predictions and for avoiding over-trust in model predictions.They also reported that it was easy to see which features carried the most weight for a class.Source: Usability and Utility · p. 24
- The attribution panel and its relationship to feature-value ranges were relatively straightforward to interpret, while linking attribution results to extracted rules was more challenging.Participants with XAI backgrounds related the representations more readily than the meteorology expert with less XAI knowledge.Source: Usability and Utility · pp. 24–25
- Participants found it harder to understand why a particular prediction was made, with feature interactions remaining a challenge.They suggested natural-language summaries, and two participants expected further familiarization to help them extract more insights.Source: Usability and Utility · p. 25
Design Feedback and Post-Study Renements
Participant feedback prompted a post-study cross-class comparison tab, but the evaluation was preliminary and the added tab was not evaluated.
Design Feedback and Post-Study Renements
- After participants requested side-by-side comparison of predicted classes, the interface gained a tab for comparing attribution methods within a class or one method across all three classes.The change was intended to address the request and reduce the cognitive overhead of switching views, but it has not been evaluated.Source: Design Feedback and Post-Study… · p. 25
5.3. Limitations
- The evaluation was preliminary, with few participants, no comparison to XALIGN, and only one participant familiar with the domain data.The authors call for a more focused study with meteorologists to assess its domain-specific utility.Source: §5.3 · p. 25
6. Conclusions
XALIGN combines rules, feature attributions, and interactive visual analytics to compare explanations. Across two case studies, methods converged for strong feature influences but diverged for weak or scattered effects, while the expert evaluation suggested utility alongside interpretive and study limitations.
6. Conclusions
- Across the German Credit and meteorology case studies, explanation methods converged for strong feature influences but diverged for weak or scattered effects, increasing misinterpretation risk.The paper presents XALIGN as a hybrid of rule-based explanations, feature attributions, and interactive visual analytics.Source: §6 · p. 26
- The interface shows agreements and discrepancies simultaneously, enabling users to drill into features and bins and cross-reference them with domain knowledge.Contradictions that could go unnoticed in isolation become visible points for investigation.Source: §6 · p. 26
- The expert evaluation found XALIGN useful for identifying important features, but suggested it is most immediately suited to users with basic familiarity with attribution and rule-based methods.Natural-language summaries or guided walkthroughs may help users without that background; participants also identified feature interactions as a gap.Source: §6 · p. 26
- Future work includes natural-language summaries, automatically flagging contradictions for expert review, support for feature interactions, and studying the refinement loop.The authors also describe integrating interactive visual analytics into explanation validation as important.Source: §6 · p. 26