PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable AI for clinical and remote health applications: a survey on tabular and time series data

Flavio Di Martino, Franca Delmastro

arXiv:2209.06528v1cs.LGcs.AI

TL;DR

Healthcare XAI for tabular and time-series data lacks consistent explanation assessment. This survey reviews evaluation practices and finds that clinical validation, consistency, and quality assessment are central requirements.

  • Problem

    Healthcare XAI studies often lack systematic assessment of whether generated explanations are clinically valid, consistent, and high quality.

  • Method

    The survey reviews healthcare XAI studies and classifies them by their contributions to evaluating explainability in target health applications.

  • Results

    The reviewed literature includes clinical validation, consistency assessment, and quality evaluation, although many studies assess generated explanations inadequately or not at all.

  • Takeaways & Limitations

    Clinical validation and human-centered evaluation are important for the acceptance, deployment, and usability of healthcare AI explanations.

  • Takeaways & Limitations

    The XAI literature lacks user studies examining clinicians’ and patients’ needs for human-consumable explanations.

Abstract

from arXiv · show

Nowadays Artificial Intelligence (AI) has become a fundamental component of healthcare applications, both clinical and remote, but the best performing AI systems are often too complex to be self-explaining. Explainable AI (XAI) techniques are defined to unveil the reasoning behind the system's predictions and decisions, and they become even more critical when dealing with sensitive and personal health data. It is worth noting that XAI has not gathered the same attention across different research areas and data types, especially in healthcare. In particular, many clinical and remote health applications are based on tabular and time series data, respectively, and XAI is not commonly analysed on these data types, while computer vision and Natural Language Processing (NLP) are the reference applications. To provide an overview of XAI methods that are most suitable for tabular and time series data in the healthcare domain, this paper provides a review of the literature in the last 5 years, illustrating the type of generated explanations and the efforts provided to evaluate their relevance and quality. Specifically, we identify clinical validation, consistency assessment, objective and standardised quality evaluation, and human-centered quality assessment as key features to ensure effective explanations for the end users. Finally, we highlight the main research challenges in the field as well as the limitations of existing XAI methods.

1 Introduction

AI supports modern clinical and remote health services, but limited transparency hinders adoption, especially when systems interact directly with non-expert users. This survey reviews XAI for tabular and time-series healthcare data and emphasizes clinical validation, consistency, and explanation-quality assessment.

  • Limited AI transparency and interpretability remain major barriers to clinical adoption, particularly in remote monitoring and personalised support involving non-expert users.
  • Tabular data require suitable XAI methods because feature interactions and mixed continuous-categorical predictors distinguish them from images and text.
  • The survey examines XAI techniques for models trained on clinical EHR data and real-world data from IoT and personal mobile devices.
  • Effective healthcare explainability requires assessing explanations through clinical validation, consistency assessment, and quality evaluation rather than merely selecting applicable methods.
  • The paper surveys prominent methods and healthcare applications, then discusses methodological limitations and open research challenges from methodological and user-based perspectives.

2 Background

XAI methods are classified by scope, development stage, target model, explanation form, and input data type. For tabular and time-series healthcare data, the background emphasizes model-agnostic techniques, feature-based explanations, local surrogates, counterfactuals, sensitivity analysis, visualizations, rule extraction, and attention mechanisms.

  • XAI taxonomy: XAI taxonomies distinguish local versus global scope, development-stage applicability, model-agnostic versus model-specific targets, and explanation forms such as feature attribution.Taxonomy also depends on input modality, including images, text, graphs, tabular data, and time series.
  • Tabular-data methods: Most XAI techniques for tabular data are model-agnostic, including feature ablation, permutation importance, MDA, MDI, SHAP, and LIME.These methods estimate feature relevance through prediction-error changes, tree impurity decreases, marginal contributions, or local surrogate models.
  • Local interpretability: LIME supports local explanations through sparse surrogate models, but random perturbations, nonlinearities, and local approximation choices can reduce faithfulness and stability.Averaging local scores can also have high variance, while LIME-SP selects representative, non-redundant explanations.
  • Rule and counterfactual explanations: Anchors identify high-precision feature ranges that preserve a local prediction and express them as IF-THEN rules, whereas counterfactual methods seek minimal changes producing a desired output.Counterfactual explanations are generally formulated as optimization problems.
  • Inspection and visualization: Sensitivity analysis measures prediction changes under input or internal-component perturbations, while PDP and ICE visualize feature–outcome relationships for global or instance-level interpretation.PDP assumes feature independence, which may not hold.
  • Global and ante-hoc explanations: InTrees extracts compact decision rules from tree ensembles, while attentive transformers learn sparse input masks and adaptive weights for ante-hoc explainability.InTrees selects and prunes rules using frequency, error rate, and length.

3 Survey methodology

The survey searched major digital libraries and Scopus for healthcare XAI literature published from 2017 to 2021. It selected methods applicable to tabular and time-series data, including post-hoc and ante-hoc approaches while excluding imaging, NLP, and intrinsically interpretable glass-box models.

  • Search strategy: The search covered IEEE Xplore, Springer, ACM, Elsevier, and Scopus, using combinations of explainability, AI, and healthcare-related keywords.Scopus was additionally used to double-check screening and identify relevant works from sources such as Nature Research journals.
  • Search strategy: The review included papers published between 2017 and 2021, a period associated with rapidly increasing XAI applications in healthcare.Table 2 reports the percentage of selected papers for each year in this date range.
  • Eligibility criteria: The screening excluded biomedical-image applications and NLP tasks, focusing on XAI methods suitable for tabular and time-series or sequence data.The excluded imaging examples include CT, MRI, and ultrasound data represented as multidimensional tensors for deep neural networks.
  • Eligibility criteria: Eligible approaches included post-hoc methods applied to existing black-box models and ante-hoc methods that embed interpretability into model structure.The methodology distinguishes interpretability added after model development from interpretability available at the end of learning.
  • Eligibility criteria: The survey excluded glass-box models whose interpretability is intrinsic to sparse linear, discretisation-based, or instance-based model classes.Examples include linear or logistic regression, generalized additive models, rule-based learners, decision trees, and k-nearest neighbors; Table 1 summarizes the criteria.

4 Results

The survey organizes reviewed healthcare XAI studies by their contributions to explanation evaluation, emphasizing clinical validation, consistency, and quantitative or qualitative quality assessment. It also notes that preliminary tabular and time-series studies often lacked explanation evaluation, limiting confidence and trust in medical applications.

  • Survey organization: Preliminary tabular and time-series studies summarized in the appendices generally proposed interpretable models without evaluating their explanations, leaving further investigation necessary to build medical confidence and trust.The Results section classifies revised works according to their main contribution to evaluating explainability in target health applications.
  • Explanation evaluation: The review uses complementary assessment categories: clinical validation, consistency across models or XAI methods, and quantitative or clinician-rated explanation quality.When multiple procedures are used, the survey reports the most relevant findings to strengthen confidence in the explanations.
  • Clinical validation: Clinical validation checks whether explanations align with established medical knowledge and whether highlighted patient trajectories or physiological features are clinically meaningful.For longitudinal EHR, validation examines trajectory evolution toward a condition; for physiological signals, it assesses whether highlighted components support meaningful decisions.
  • Clinical validation: Clinical comparison of explanations can additionally support knowledge discovery by identifying emergent risk-factor relationships that may warrant integration into clinical practice.The survey presents these clinical-validation studies in Table 6.
  • Tabular data applications: In tabular EHR applications, SHAP-based analyses identify clinically established predictors while also revealing potentially novel associations for further investigation.Examples include recent cognitive-test features for dementia risk and body temperature or respiratory rate for childhood-obesity prediction.

4.2 Explanation consistency assessment

Explanation consistency assessment examines whether different models or explanation methods identify similar patterns, providing indirect evidence about model stability and robustness. However, formal comparison remains limited by the lack of established metrics, although quantitative measures such as AAM have begun to address this gap.

  • 4.2 Explanation consistency assessment: Agreement among explanations from different methods can provide preliminary evidence of model stability and robustness, but sensitivity analysis is needed for more accurate conclusions.Stability concerns unintentional perturbations such as noise, while robustness concerns intentional changes such as adversarial attacks.
  • 4.2 Explanation consistency assessment: Consistency assessment compares feature-attribution rankings or decision-rule overlap for tabular data, whereas saliency and attention are benchmark explanations for time series.The assessment generally applies when methods provide the same explanation format.
  • 4.2 Explanation consistency assessment: Applications reported agreement across models, including shared top-three diabetes features and common ICU mortality factors such as age and blood urea nitrogen level.The diabetes comparison used SHAP explanations for RF and GBDT, while the ICU comparison examined SHAP values, logistic-regression coefficients, and tree feature importance.
  • 4.2 Explanation consistency assessment: Clinical validation and consistency assessment remain indirect because globally recognised metrics and practices for formally evaluating and comparing explanation quality are not well established.Recently proposed quantitative metrics offer an initial objective evaluation, but their applicability is often limited to particular settings.
  • 4.2 Explanation consistency assessment: Approximately 25% was the best global AAM value, despite the model outperforming other attention-based networks in detecting relevant signal time points.AAM measures the average percentage of attention correctly detected across local predictions.

5 Discussion

The survey links data structure to model and XAI-method selection: tree ensembles dominate static data, while RNNs and CNNs dominate raw time series. SHAP offers stronger theoretical guarantees than heuristic LIME, whereas time-series explanations commonly rely on attention or saliency heatmaps.

  • Static and longitudinal data: Static data favor tree ensembles such as RF and XGBoost for modeling complex nonlinearities, although these models remain black boxes requiring XAI.Shallow MLPs, ad-hoc networks such as TabNet, and data-mining approaches are used only in limited cases.
  • Explainability methods: SHAP has theoretical foundations in optimal Shapley values and satisfies local accuracy, missingness, and consistency, supporting trustworthy explanations.LIME instead uses a heuristic sparse linear surrogate, which may be unfaithful when the underlying model is highly nonlinear locally.
  • Time series data: Raw time series commonly use RNNs and CNNs because they perform well without heavy signal preprocessing, despite complexity and limited interpretability.Attention is the reference ante-hoc method for RNNs, while gradient-based methods are usually applied to CNNs.
  • Time series data: For CNNs, Grad-CAM and LRP are influential post-hoc methods, while prototype learning, occlusion, and attention-equipped CNNs provide alternative explanations.Attention and saliency methods generally support local interpretability through heatmaps overlaid on input signals.

5.1 XAI Challenges in Healthcare

Healthcare XAI requires clinical validation, stable and robust models, objective and human-centered evaluation, and interactions aligned with users’ reasoning needs. Major limitations remain in explaining deep models’ internal processes, causal relationships, and time-series reasoning.

  • Clinical validation: Clinical validation is essential for deploying healthcare AI and can mitigate bias, detect model failures, and reveal overlooked biomarkers or risk factors.Clinical comparison may also save certification resources and support knowledge discovery with potential clinical utility.
  • Stability and robustness: Healthcare models should produce stable predictions and explanations for equal or nearby inputs, while outlier detection and confidence scores can expose unreliable decisions.Consistency comparisons across explanation methods offer inexpensive stability insights, but training and external validation cannot guarantee robustness against perturbations.
  • Explanation evaluation: Explanation quality needs objective quantitative metrics and standardized frameworks, complemented by human-centered studies addressing clinicians’ and patients’ needs.User studies remain scarce, and non-experts may struggle to translate algorithmic outputs into human-consumable explanations.
  • Human-AI interaction: Human-AI interaction must connect explanations to users’ goals through clinician-informed design, feedback, raw-data links, cohort evidence, context, and what-if or counterfactual tools.A study of more than 1300 physicians found that 88% positively evaluated AI systems with model-agnostic explanations for trust and understanding.
  • Deep-model explainability: Saliency and attention methods often identify relevant inputs without explaining how models use them, leaving deep neural networks’ global and latent reasoning opaque.Existing approaches such as Grad-CAM and LRP provide heuristic local information, while methods for probing strange explanations remain lacking.
  • Causability: Post-hoc explanations may show correlations without supporting causal medical decisions, making alignment between explainability and causability crucial for effective physician interactions.The cited literature frames causal understanding as necessary for successful healthcare interfaces.

6 Conclusions

Tabular and time series data support accurate, responsive, less-invasive, and low-cost healthcare applications, but XAI analysis and rigorous explanation evaluation remain insufficient. Despite potential benefits for patient-centered care, end-to-end XAI systems ready for large-scale deployment remain out of reach because interpretability, predictive performance, and trustworthiness involve unresolved trade-offs.

  • Conclusions: XAI is required for complex models using tabular and time series health data, yet explanation evaluation of clinical relevance, consistency, and quality is often missing.These data types are less studied from an XAI perspective than biomedical images and medical text records.
  • Conclusions: XAI can support patient-centered care by helping clinicians comprehend and report decisions, potentially improving patient acceptance and comfort with AI-enabled medical practices.Patients are described as active partners entitled to choose and control their treatments and recommendations.
  • Conclusions: End-to-end XAI systems suitable for large-scale deployment with minimal human supervision remain distant, requiring methodological improvements and user-centered studies.The conclusion combines ethical, social, medical, and technical challenges identified by the survey.
  • Conclusions: Inherently interpretable models may trade predictive performance for transparency, whereas post-hoc explanations preserve accuracy but approximate the model’s reasoning and cannot be completely trusted.Healthcare AI predictions will continue to carry risks and failures before robust interpretability is achieved.

Declarations

The work was partially funded by the European Commission through the H2020 SoBigData-PlusPlus project, and the authors report no relevant competing interests.

  • Declarations: The study received partial European Commission funding through H2020-INFRAIA-2019-1 SoBigData-PlusPlus, grant no. 871042, and the authors disclosed no relevant financial or non-financial competing interests.Funding and conflict-of-interest declarations are reported together.

Appendix A Acronyms

Appendix A provides an alphabetically ordered glossary of acronyms used throughout the paper, covering clinical terms, data types, models, and explainable AI methods.

  • Appendix A Acronyms: Table A1 lists the paper’s acronyms and their definitions in alphabetical order.The glossary includes terms spanning healthcare concepts, machine-learning architectures, data modalities, and XAI techniques.

Appendix B Preliminary works

Appendix B catalogs preliminary studies applying explainable AI to tabular and time series data, alongside cited dataset sources.

  • Appendix B Preliminary works: Table B2 catalogs preliminary studies applying XAI to tabular data.
  • Appendix B Preliminary works: The cited preliminary works draw on datasets from NHANES, COVID-19, heart-failure clinical records, and fetal-health classification.
  • Appendix B Preliminary works: Table B3 catalogs preliminary studies applying XAI to time series data.
Loading 2209.06528v1…