PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review on Explainability in Multimodal Deep Neural Nets

Gargi Joshi, Rahee Walambe, Ketan Kotecha

arXiv:2105.07878v2cs.AIcs.CV

TL;DR

Opaque multimodal neural networks are difficult to interpret even as they support richer representations and predictions. This paper surveys explainability in multimodal deep learning, especially vision and language, by organizing methods, evaluation approaches, applications, and future directions. It concludes that explanations still require better grounding, faithfulness, user adaptation, and human-centered evaluation.

  • Problem

    Multimodal deep neural networks are opaque, while heterogeneous data make it difficult to understand which modalities or features drive predictions and to evaluate explanations reliably.

  • Method

    The paper reviews explainability literature for multimodal deep neural networks, focusing on vision and language tasks and classifying explanation methods, evaluation approaches, datasets, applications, and stakeholder requirements.

  • Results

    The review identifies attention-based, counterfactual, interactive, graph-based, and attribute-based approaches, while describing explanation evaluation as functionally grounded, human-grounded, or application-grounded.

  • Takeaways & Limitations

    Multimodal explanations need better correlation, completeness, faithfulness, grounding, reasoning, context adaptation, and user-centric design.

  • Takeaways & Limitations

    No foundational explanation theory is established, and multimodal explainability remains domain-specific with persistent challenges in reasoning, contextual adaptation, data, architectures, and evaluation metrics.

Abstract

from arXiv · show

Artificial Intelligence techniques powered by deep neural nets have achieved much success in several application domains, most significantly and notably in the Computer Vision applications and Natural Language Processing tasks. Surpassing human-level performance propelled the research in the applications where different modalities amongst language, vision, sensory, text play an important role in accurate predictions and identification. Several multimodal fusion methods employing deep learning models are proposed in the literature. Despite their outstanding performance, the complex, opaque and black-box nature of the deep neural nets limits their social acceptance and usability. This has given rise to the quest for model interpretability and explainability, more so in the complex tasks involving multimodal AI methods. This paper extensively reviews the present literature to present a comprehensive survey and commentary on the explainability in multimodal deep neural nets, especially for the vision and language tasks. Several topics on multimodal AI and its applications for generic domains have been covered in this paper, including the significance, datasets, fundamental building blocks of the methods and techniques, challenges, applications, and future trends in this domain

I. INTRODUCTION

Multimodal deep learning combines heterogeneous modalities to improve representations and predictions, but its opaque decision processes make explainability especially important. This survey reviews explainability for multimodal systems, focusing on vision and language while covering methods, applications, evaluation, datasets, and future directions.

  • Deep neural networks perform effectively across vision, sensor, text, language, and multimodality tasks, but their hidden processing is difficult to interpret.
  • The survey focuses on explainable multimodal AI for vision and language, covering image, video, text, and audio modalities.
  • Existing interpretation methods, including Grad-CAM attention maps, may not establish that models use the correct evidence or provide explanatory power.
  • The review organizes multimodal explainability techniques around disentangled representations, multimodal explanations, and counterfactuals, alongside evaluation and stakeholder requirements.
  • Multimodal systems combine complementary information and inter-, intra-, and cross-modal relationships to produce richer representations and improved predictions.
  • Persistent multimodal heterogeneity complicates analytical methods and motivates benchmarks addressing evaluation flaws, dataset bias, robustness, and spurious correlations.

A. TYPICAL CHALLENGES IN MULTIMODAL SETTING

Multimodal systems face challenges in representing, aligning, and combining heterogeneous data. Early, late, and hybrid fusion differ in when modalities are combined, with hybrid fusion learning a shared representation but carrying overfitting risks.

  • Representation: Heterogeneous modalities differ in form, scale, and dimensionality, making representation crucial for extracting complementary or redundant information.
  • Early fusion: Early fusion combines modality features before subsequent processing, typically through concatenation, pooling, or gated units.
  • Late fusion: Late fusion combines separate modality sources at decision-making, using rules such as Bayes, max-fusion, average-fusion, or majority vote.
  • Hybrid fusion: Hybrid fusion combines modalities at a shared representation layer, propagating training loss back to modality-specific feature extractors.
  • Hybrid fusion: Hybrid fusion can improve performance over early and late fusion, but gradual fusion may overfit or fail to learn different modality correlations.

C. RECENT DATA FUSION TECHNIQUES

This section reviews traditional and recent multimodal data-fusion techniques, including joint representations, pooling, low-rank, memory-based, and other fusion approaches. It also frames explainability as necessary because multimodal AI models are difficult to interpret and justify.

  • Fusion approaches: Multimodal fusion methods combine modality-specific information into shared representations for integrated tasks.The reviewed approaches include fusion of unimodal embedding spaces with semantic visual and contextual language features.
  • Fusion approaches: Low-rank multimodal fusion reduces bilinear pooling complexity through Tucker decomposition.Multimodal Tucker fusion decomposes the fusion representation into a low-rank matrix structure.
  • Fusion approaches: Recent techniques include channel exchange, architectural search, low-rank multimodal fusion, memory-based fusion, tensor fusion, bilinear pooling, Tucker fusion, and multilinear fusion.These methods represent the range of traditional and recent approaches reviewed in the section.
  • Section scope: The section concludes by reviewing traditional and recent methods for multimodal data fusion.
  • Explainability motivation: Explainability is needed because multimodal AI models are black boxes whose internal functioning and outcomes are difficult to justify.The challenge is intensified by differences in representation scales across modalities.

III. EXPLAINABLE AI (XAI)

XAI addresses the opacity of deep neural networks, whose hidden processing can make decisions difficult to understand and explain. The field supports transparency, debugging, design improvement, and responsible AI development.

  • Motivation: Deep neural networks achieve strong results but remain opaque because their hidden-layer processing obscures internal states and decision-making.This opacity creates an accuracy–interpretability tradeoff in which more performing models are described as less interpretable.
  • Motivation: XAI seeks explanations that clarify which modalities or features drive predictions and why a model makes a particular decision.Interpretability concerns understanding the cause of a decision and consistently predicting model results.
  • Significance: Human-interpretable explanations are important for trust, transparency, reliability, fairness, and justification in consequential applications.The paper identifies legal, healthcare, finance, transport, military, and autonomous-vehicle settings as examples.
  • Stages of explainability: Explainability can be introduced before modeling, during modeling, or after modeling, spanning data practices, model design, and post hoc analysis.The paper notes that models may be right for the wrong reason while lacking clear justification.

A. LEVELS OF EXPLANATION MODELLING

Explainability is organized across three model-development stages: before modeling, during modeling, and after modeling.

  • Pre modeling: Pre-modeling explainability addresses data understanding, documentation, summarization, feature engineering, unbiased collection, and experimental design.
  • During modelling: During modeling uses inherently explainable models that are explainable by design and typically employ intrinsic methods.

B. SCOPE OF EXPLANATION

Explanations may be local or global, depending on whether they address an individual instance or the model as a whole. The taxonomy also distinguishes model-specific and model-agnostic approaches.

  • Scope of explanation: Local explanations derive predictions for a particular data point, whereas global explanations characterize the entire model.The paper states that local explanations are often preferred over global explanations.
  • Working principle: Model-specific explanations are designed for a particular model, including gradient-based methods proposed for convolutional neural networks.
  • Working principle: Model-agnostic explanations are independent of the model and can be applied across domains using modular post hoc approaches.Examples mentioned include partial dependence plots and individual conditional expectation plots.

C. FEATURE ATTRIBUTION BASED METHOD

Feature attribution methods explain predictions by highlighting input features that contribute to the model’s output. They include gradient-based and perturbation-based techniques, but may lack semantic reasoning and require model access.

  • Feature attribution methods highlight image regions that significantly contribute to decisions but lack semantic reasoning and interaction modeling.
  • Visualization techniques identify influential input features and are classified into backpropagation-based and perturbation-based methods.
  • Backpropagation-based methods use gradients passed through the network to identify relevant features, including weighted activations in CAM.
  • Gradient-based methods include saliency maps, Grad-CAM, Integrated Gradients, DeepLift, LRP, and SHAP, each attributing importance through gradients or contribution scores.These methods require access to model parameters and an understanding of the model architecture.
  • Perturbation-based methods compare outputs for original and altered inputs to identify sensitive features, using techniques such as occlusion sensitivity, RISE, and LIME.

D. DISTILLATION METHODS

Distillation methods explain deep models by constructing simpler approximations or inherently explainable components. The supplied taxonomy also distinguishes intrinsic attention-based and joint-training approaches.

  • D. DISTILLATION METHODS: Local approximation methods derive prediction insights from a single data point, while model translation builds a surrogate model over the original model.
  • D. DISTILLATION METHODS: Model translation can use inherently explainable proxy models such as decision trees to interpret predictions from the original model.
  • D. DISTILLATION METHODS: Intrinsic methods are inherently explainable and may use attention mechanisms that focus on important visual and textual regions.
  • D. DISTILLATION METHODS: Joint training approaches jointly model predictions and explanations to build explainable models.
  • D. DISTILLATION METHODS: Occlusion sensitivity compares outputs for original and altered inputs to identify features sensitive to prediction.

IV. EXPLAINABILITY IN MULTIMODAL DATA

Multimodal explainability seeks systems that combine visual and textual evidence to explain predictions in ways suited to human understanding and different stakeholders. It addresses the limitations of explanations confined to a single modality or viewpoint.

  • Multimodal explanations are considered important for intelligent systems with understanding and reasoning capabilities comparable to human multimodal perception.
  • Unimodal vision systems provide image-based feature visualizations, while language systems provide text-based post hoc justifications that may be insufficient across situations.
  • Explainability aims to answer how and why a deep neural network makes a prediction by inspecting factors driving its decisions.
  • Explanatory systems may be based on scientific modeling rather than explanation generation and may not satisfy different stakeholders’ needs.
  • Multimodal explanation interfaces are described as needing role-based justifications, multimodal interaction, and feedback mechanisms that incorporate system changes.
  • Justification-based systems are understandable to humans but lack causal interpretation, whereas introspection-based systems examine internal network behavior but are not well understood by humans.

V. MULTIMODAL EXPLAINATION METHODS

Multimodal explanation methods for vision-language tasks are organized around attention, counterfactuality, interactive, and attribute-based approaches. The literature combines visual regions, textual explanations, grounding, and human-centered evaluation, while attention-based explanations face faithfulness challenges.

  • V. MULTIMODAL EXPLAINATION METHODS: The paper classifies multimodal explanation techniques into attention-based, counterfactuality-based, interactive, and attribute-based methods.
  • A. ATTENTION BASED APPROACHES: Attention mechanisms align and fuse modalities by assigning greater importance to selected factors in tasks such as captioning, VQA, and visual entailment.
  • A. ATTENTION BASED APPROACHES: Guided backpropagation and occlusion visualizations identify image and text regions used by VQA models, while human-in-the-loop approaches improve human understandability.
  • A. ATTENTION BASED APPROACHES: PJ-X uses attention mechanisms, textual explanations, and corresponding visual regions to explain VQA answers and activity-recognition outputs.
  • A. ATTENTION BASED APPROACHES: Other multimodal methods address spatial grounding, user mental models, knowledge bases, multiple tasks, and self-critical alignment with human-relevant image regions.
  • A. ATTENTION BASED APPROACHES: VCR provides answers and justifications for visual common-sense reasoning, supporting prototype debugging and auditing of model reasoning.
  • A. ATTENTION BASED APPROACHES: Attention maps do not establish that models attend to the correct region, and explanation evaluation is restricted by absent ground truth and questionable faithfulness.
  • B. COUNTERFACTUALITY BASED METHODS: Counterfactual explanations reflect human contrastive reasoning by examining why one decision occurred instead of another and identifying changes that could reverse it.

C. INTERACTIVE APPROACHES

Interactive multimodal approaches combine explanations, user feedback, and task-specific evaluation to improve understanding and diagnose model behavior. The reviewed methods include active attention, graph-based reasoning, attribute-based explanations, and multiple evaluation strategies.

  • Interactive approaches: Interactive active attention models incorporate user feedback and alter attention when a forecast is incorrect.These systems connect model explanation with interaction rather than presenting a fixed explanation.
  • Interactive approaches: Graph-based interactions reason over image scene graphs to support question answering and visual reasoning.The approach learns question-specific relationships before producing answers or descriptions.
  • Interactive approaches: Attribute-based multimodal explanations associate visual features with attributes and counterexamples to improve text-image grounding.The reviewed work uses visual and counter visual examples for class-discriminative explanations.
  • Interactive approaches: Attribute maps and activation maps expose learned features, region contributions, and complementary class cues for interpretability.These visualizations are used to examine how regions and class pairs contribute to explanations.
  • Interactive approaches: Explanation evaluation spans functionally grounded, human-grounded, and application-grounded methods because explanation quality is task-specific and subjective.The categories range from evaluating model functions to assessing human preferences and mental models.
  • Interactive approaches: Automated and human evaluation methods include machine-translation metrics, image-description metrics, crowdsourcing, and student-teacher simulation measures.BLEU-4, METEOR, CIDEr, and SPICE are among the automated metrics discussed.

VII. DIVERSE EXPLAINABILITY REQUIREMENTS

Explainability requirements differ by user role and goal, while multimodal systems face dataset, bias, and modality-balance challenges. The review highlights task-specific datasets and the need to account for how modalities contribute to predictions.

  • Diverse explainability requirements: Model creators may need layer-level explanations for debugging, whereas non-experts often need functional explanations of outputs.Different stakeholders require explanations suited to their understanding and objectives.
  • Diverse explainability requirements: Multimodal explainability has been constrained by limited task-specific datasets, motivating datasets with reasoning annotations and textual justifications.Examples include Raven’s Progressive Matrices, counting-based VQA data, VQA-X, and ACT-X.
  • Diverse explainability requirements: Unequal modality contributions can produce suboptimal results, while imbalance and feature selection can introduce model bias.The review describes bias from modality dominance, crowdsourcing, word embeddings, sentence-level representations, and dataset regularities.
  • Diverse explainability requirements: Explainability should examine cause-and-effect aspects rather than relying only on performance.The review frames this as part of seeking the right reasons for model decisions.

X. ADVERSARIAL ATTACKS ENHANCE

Adversarial perturbations can change multimodal model outputs and can also be used to inspect how models rely on modality-specific patterns. The review connects these analyses to broader needs for faithful, interactive, and human-centered explanations.

  • Adversarial attacks enhance: Adversarial perturbations can produce examples that mislead classifiers into incorrect decisions while remaining difficult for humans to notice.The review also describes masking visual inputs to examine reliance on statistical language patterns.
  • Adversarial attacks enhance: Unimodal explanations provide feature visualizations or post hoc textual justifications but lack complete multimodal introspection.The review identifies this gap as a motivation for further multimodal explainability research.
  • Adversarial attacks enhance: Multimodal explanations may be incomplete or unfaithful to internal model decisions when grounding, reasoning, and context adaptation are inadequate.The review calls for diagnostic capabilities that address inconsistencies and fine-grained feature relations.
  • Adversarial attacks enhance: Multimodal research continues to face challenges in representation, alignment, translation, fusion, co-learning, reasoning, labeled data, architectures, and evaluation metrics.The review also states that no foundational explanation theory has been established and that explanations remain domain-specific.
  • Adversarial attacks enhance: Contrastive and counterfactual explanations are presented as promising because explanations and predictions can be jointly modeled.The review further points toward inherently self-explaining models using integrated visual and textual modalities.
  • Adversarial attacks enhance: The review categorizes multimodal explainability methods, highlights bias and adversarial-defense concerns, and calls for benchmarks and diagnostic datasets.It presents explainability as relevant to trust, transparency, and tracing model flaws.
Loading 2105.07878v2…