PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiViz: Towards Visualizing and Understanding Multimodal Models

Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis-Philippe Morency, Ruslan Salakhutdinov

arXiv:2207.00056v3cs.LGcs.AIcs.CLcs.CVcs.MM

TL;DR

Black-box multimodal models make their internal modeling difficult to understand, motivating methods to visualize their behavior. MULTIVIZ organizes analysis into four stages and combines existing and proposed tools. Across real-world tasks, the authors report applications in prediction simulation, feature interpretation, error analysis, and model debugging.

  • Problem

    Multimodal models are typically black-box neural networks, making it difficult to visualize and understand how they model multimodal information and interactions.

    Source: §1 · p. 1

  • Method

    MULTIVIZ scaffolds analysis into unimodal importance, cross-modal interactions, multimodal representations, and multimodal prediction, combining existing and newly proposed tools.

    Source: §2 · p. 2, §2.2 · p. 3

  • Results

    Across 8 models and 6 real-world tasks, MULTIVIZ supports prediction simulation, interpretable feature concepts, error analysis, and a human-in-the-loop debugging case study.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present MULTIVIZ as a modular, general, human-in-the-loop toolkit for understanding and debugging multimodal models.

    Source: §5 · p. 9

    The four-stage perspective may not fit multimodal translation or crossmodal transfer, which require additional stages for generation or knowledge transfer.

    Source: §6 · p. 10

Abstract

from arXiv · show

The promise of multimodal models for real-world applications has inspired research in visualizing and understanding their internal mechanics with the end goal of empowering stakeholders to visualize model behavior, perform model debugging, and promote trust in machine learning models. However, modern multimodal models are typically black-box neural networks, which makes it challenging to understand their internal mechanics. How can we visualize the internal modeling of multimodal interactions in these models? Our paper aims to fill this gap by proposing MultiViz, a method for analyzing the behavior of multimodal models by scaffolding the problem of interpretability into 4 stages: (1) unimodal importance: how each modality contributes towards downstream modeling and prediction, (2) cross-modal interactions: how different modalities relate with each other, (3) multimodal representations: how unimodal and cross-modal interactions are represented in decision-level features, and (4) multimodal prediction: how decision-level features are composed to make a prediction. MultiViz is designed to operate on diverse modalities, models, tasks, and research areas. Through experiments on 8 trained models across 6 real-world tasks, we show that the complementary stages in MultiViz together enable users to (1) simulate model predictions, (2) assign interpretable concepts to features, (3) perform error analysis on model misclassifications, and (4) use insights from error analysis to debug models. MultiViz is publicly available, will be regularly updated with new interpretation tools and metrics, and welcomes inputs from the community.

1 INTRODUCTION

MULTIVIZ addresses the difficulty of interpreting black-box multimodal models with a four-stage framework for examining their behavior. Experiments across real-world tasks show applications in prediction simulation, feature interpretation, error analysis, and model debugging.

1 INTRODUCTION

  • MULTIVIZ organizes multimodal interpretation into unimodal importance, cross-modal interactions, multimodal representations, and multimodal prediction.The stages examine modality contributions, relationships between modalities, decision-level features, and how those features compose a prediction.Source: §1 · p. 1, Figure 1 · p. 2
  • Black-box neural-network parameterizations make it challenging to understand how multimodal information and interactions are modeled.The paper frames visualization and understanding as steps toward more reliable real-world deployment.Source: §1 · p. 1
  • Across 6 real-world tasks, 6 modalities, and 8 models, experiments show uses in simulating predictions, assigning concepts to features, analyzing errors, and debugging models.The authors describe a human-in-the-loop case study using error-analysis insights to debug a model.Source: §1 · p. 2

2 MULTIVIZ: VISUALIZING AND UNDERSTANDING MULTIMODAL MODELS

MULTIVIZ scaffolds multimodal model analysis into complementary stages, combining existing and proposed tools. Its interface presents stage-specific visualizations and feature-level analysis for selected datasets and models.

2 MULTIVIZ: VISUALIZING AND UNDERSTANDING MULTIMODAL MODELS

  • MULTIVIZ frames interpretation as complementary stages spanning modality inputs, cross-modal relationships, decision-level features, and predictions.The framework assumes datasets pair modality inputs with labels and models produce predictions from those inputs.Source: §2 · p. 2

2.1 UNIMODAL IMPORTANCE (U)

  • Unimodal importance applies gradient and feature-attribution methods to estimate the importance of modality atoms for a chosen output or feature.The UNI module returns importance weights across atoms in the selected modality.Source: §2.1 · pp. 2–3

2.2 CROSS-MODAL INTERACTIONS (C)

  • Second-order gradients isolate non-additive cross-modal interactions, assigning weights to atoms in one modality for a queried atom in another.The method zeros out unimodal terms; unlike EMAP and DIME, its stated motivation is avoiding expectations over modality subsets that may not scale beyond two modalities.Source: §2.2 · pp. 3–4

2.3 MULTIMODAL REPRESENTATIONS

  • Local representation analysis links input regions to a feature on one datapoint, while global analysis examines top-activating datapoints to support concept assignment and error analysis.Both analyses visualize unimodal and cross-modal interactions with respect to the feature.Source: §2.2 · p. 4

2.4 MULTIMODAL PREDICTION (P)

  • A sparse linear model approximates prediction from penultimate-layer features, and its highest nonzero coefficients identify features involved in composing a prediction.Those ranked features can then be visualized using the representation stage.Source: §2.4 · p. 4

2.5 PUTTING EVERYTHING TOGETHER

  • The interactive API lets users select datasets and models, view overview visualizations, and open feature pages for local and global analysis.Overview visualizations include unimodal importance, cross-modal interactions, and prediction weights.Source: §2.5 · p. 4

3 EXPERIMENTS

Across six datasets and eight models, the experiments test whether MULTIVIZ’s complementary stages help people simulate predictions, interpret features, classify errors, and debug models. Results span human studies, a model-debugging case study, and additional analyses of generality and tradeoffs.

3.1 MODEL SIMULATION

  • Full MULTIVIZ gave users the highest VQA 2.0 simulation accuracy, lowest variance, and most consistent agreement; on fusion tasks, adding each stage consistently improved correctness and agreement.Users could simulate predictions whether or not the model’s prediction was correct.Source: Table 2 · p. 6, §3.1 · p. 6
  • At 100 VQA 2.0 datapoints, full MULTIVIZ reached 80% correctness, within the 20-point study’s 81.7 ± 2.9 range; Rℓ + P and Rg + P scored 51.7 ± 12.6 and 71.7 ± 7.6, respectively.The two feature-analysis-only settings also had agreement scores of 0.40 and 0.53, respectively, and underperformed settings combining overall and feature analysis.Source: §3.1 · p. 6

3.3 ERROR ANALYSIS

  • With full MULTIVIZ visualizations, users consistently categorized errors into unimodal perception, cross-modal interaction, or prediction stages.The study used 10 users across two datasets and two models.Source: Table 3 · p. 7, §3.3 · pp. 7–8, Figure 5 · p. 8

3.2 REPRESENTATION INTERPRETATION

  • Users assigned concepts to features more confidently and with higher agreement when they had both local and global visualizations; examples include image-only, language-only, and multimodal concepts.Participants said highlights matching a concept increased confidence, and that the full set of MULTIVIZ stages was needed to reliably interpret examples.Source: §3.2 · p. 7, Figure 4 · p. 7

3.4 A CASE STUDY IN MODEL DEBUGGING

  • In LXMERT fine-tuned on VQA 2.0, two of five highly activated penultimate-layer neurons related to color questions; blue-related points had 5.5% accuracy and accounted for 8.8% of errors.The authors traced the prediction-stage failure to an RGB-to-BGR image-format swap in Faster R-CNN preprocessing in the Hugging Face repository.Source: §3.4 · p. 8, Figure 6 · p. 8

3.5 ADDITIONAL EXPERIMENTS AND TAKEAWAYS MESSAGES

  • MULTIVIZ was also applied to VILT, CLIP, and MDETR; judged cross-modal alignment correlated strongly with task performance, and the framework passed data- and model-randomization checks.Visualizing intermediate layers creates new opportunities but can overwhelm users with images across dimensions and layers.Source: §3.5 · p. 9

4 RELATED WORK

Prior work on multimodal interpretation spans input-level modality importance, cross-modal relationships, and higher-level representations and predictions. Existing approaches often focus on particular stages, modalities, or tasks.

Unimodal importance

  • Input-level work highlights image regions using attention, gradients, or feature attributions such as LIME and Shapley values.These approaches address unimodal importance.Source: §4 · p. 9

Cross-modal interactions

  • Cross-modal studies examine transformer activations, diagnostic inputs, or auxiliary explanation modules; EMAP and M2Lens address unimodal and cross-modal contributions.M2Lens visualizes multimodal models for sentiment analysis.Source: §4 · p. 9

Multimodal representation and prediction

  • Representation and prediction approaches use language syntax or logical statements for compositionality, but are typically limited to certain modalities or tasks.Visualizations have also revealed biases such as unimodal VQA question biases and gender bias in image captioning.Source: §4 · p. 9

5 CONCLUSION

MULTIVIZ proposes a framework for analyzing and visualizing multimodal model behavior. It organizes interpretation into four stages and provides analysis tools across those stages.

5 CONCLUSION

  • MULTIVIZ organizes interpretation into unimodal importance, cross-modal interactions, multimodal representations, and multimodal prediction.The framework combines existing and newly proposed analysis tools and is designed to be modular, general, and human-in-the-loop.Source: §5 · p. 9

6 ETHICS STATEMENT

The paper identifies limitations and cautions for interpreting MULTIVIZ visualizations, including reliability, usability, and task scope. It also notes that interpretability evaluation is difficult and that some multimodal problems need different analysis stages.

Reliability of visualizations

  • Explanation methods may disagree, and existing VQA explanations have not made model responses and failures more predictable to humans.The paper suggests human-in-the-loop interpretation and evaluation as a possible direction for connecting technical solutions with stakeholders.Source: §6 · p. 10

Pitfalls of gradient-based interpretation

  • Gradient-based interpretations have limitations in faithfulness and usefulness, especially warranting further study for cross-modal interactions.Source: §6 · p. 10

The role of cross-modal interactions

  • Good task performance may not require broad cross-modal interaction capture; cross-modal retrieval may rely on a single word-region interaction.Source: §6 · p. 10

User studies

  • The study records annotator information so identities cannot readily be ascertained and collects no identifiable information.Source: §6 · p. 10

Usability

  • More visualization information can overwhelm users, so the paper plans usability work with HCI researchers and careful user studies.Source: §6 · p. 10

Beyond MULTIVIZ stages

  • The proposed stages may not suit every multimodal problem: translation lacks prediction layers, while transfer requires stages for knowledge transfer.Source: §6 · p. 10

Evaluating interpretability

  • Interpretability evaluation is difficult because judgments vary across population subgroups, outputs are high-dimensional, and desiderata change across fields, populations, and time.The authors plan to expand MULTIVIZ with community input, new methods, and evaluation metrics.Source: §6 · p. 10

7 REPRODUCIBILITY STATEMENT

The reproducibility statement points readers to released code, datasets, documentation, and appendices for implementation and experimental details. Those materials describe input attribution, cross-modal interaction analysis, feature interpretation, and prediction decomposition.

7 REPRODUCIBILITY STATEMENT

  • Code, datasets, and documentation are released, with appendices covering visualization approaches, the website and tutorials, dataset preprocessing, and experimental details.The code release also includes human-in-the-loop evaluation scripts and instructions for new datasets and models.Source: §7 · p. 11

A.1 UNIMODAL IMPORTANCE

  • UNI takes a trained model, output or feature, and modality, then returns importance weights across that modality’s atoms.The paper describes LIME as perturbing input atoms, observing prediction changes, and fitting a local linear model.Source: §A.1 · p. 18

A.2 CROSS-MODAL INTERACTIONS

  • EMAP separates unimodal and cross-modal terms, while DIME supports efficient local visualization of the disentangled representations.EMAP quantifies global cross-modal interactions across a dataset; DIME extends it to datapoint-level feature visualization.Source: §A.2 · p. 18
  • Higher-order gradients isolate non-additive cross-modal effects because second-order gradients eliminate unimodal terms and are theoretically necessary and sufficient for recovering interactions.The implementation aggregates a first gradient over one modality atom before taking a second-order gradient across atoms of the other modality.Source: §A.2 · p. 18
  • Cross-modal visualization can condition on one modality atom and return importance weights over atoms in the other modality.The described example uses a word such as “dog” to highlight the corresponding image region.Source: §A.2 · p. 19

A.3 MULTIMODAL REPRESENTATIONS

  • Local representation analysis explains input regions activating a feature, while global analysis examines highly activating datapoints to support concept assignment and error analysis.Global analysis visualizes unimodal and cross-modal inputs across the top activating datapoints.Source: §A.3 · p. 19

A.4 MULTIMODAL PREDICTION

  • Prediction analysis fits a sparse linear model to penultimate-layer features, using selected coefficients and ranked features to show how features compose a prediction.Source: §A.4 · p. 19

A.5 SANITY CHECKS FOR SALIENCY MAPS

MultiViz passes data- and model-randomization sanity checks: its visualizations vary across inputs and models. Its interactive API presents visualizations from multiple analysis stages for selected datasets and models.

A.5 SANITY CHECKS FOR SALIENCY MAPS

  • Visualizations vary across datapoints and labels, capturing input regions, related datapoints, feature concepts, and errors specific to each example.This supports the data-randomization check because outputs change with the data rather than remaining invariant.Source: §A.5 · p. 19
  • Across CLEVR and Flickr-30K, MultiViz produces different results for different models on the same data, explaining performance differences through captured cross-modal interactions.The authors report that this satisfies the model-randomization check.Source: §A.5 · p. 20
  • Together, the data- and model-randomization tests show that MultiViz passes the sanity checks proposed by Adebayo et al. (2018).Source: §A.5 · pp. 19–20

B MULTIVIZ VISUALIZATION TOOL

B.1 MULTIVIZ CODE FRAMEWORK

MultiViz provides a modular Python framework for applying analysis and visualization methods across datasets and models. Its website organizes the resulting views, while its representation layer can be selected to suit visualization needs.

B.1 MULTIVIZ CODE FRAMEWORK

  • The modular Python framework separates dataset, model, analysis, and visualization scripts, and is designed to extend to new datasets, models, and methods.Shared model wrappers expose common functions so the same analysis script can work across different models.Source: §B.1 · pp. 20–21

B.2 THE MULTIVIZ WEBSITE

  • The website organizes a datapoint’s unimodal and cross-modal views alongside prediction features, with feature pages showing local and global analyses.For VQA, the overview uses LIME for unimodal importance and DIME for cross-modal interactions; users can open a feature page from the top-five feature graph.Source: §B.2 · pp. 21–23, Figure 8 · p. 22, Figure 9 · p. 23

B.3 INTERMEDIATE LAYER REPRESENTATION VISUALIZATION

  • The codebase supports representation analysis at any model layer, but the second-last layer is chosen for simpler, linear prediction-stage visualization.Using another layer would make the prediction stage involve multiple layers, weights, and more complex interactions.Source: §B.3 · p. 24, Figure 16 · p. 28

C DATASETS AND MODELS

MultiViz draws on standardized multimodal benchmarks spanning fusion, retrieval, and question answering. The datasets cover varied prediction settings and alignment challenges across real-world and diagnostic tasks.

C DATASETS AND MODELS

C.1.1 MULTIMODAL FUSION

  • Multimodal fusion tasks require combining modalities with different predictive power, identifying higher-order interactions, and sometimes aligning units across time.The section includes CMU-MOSEI, MM-IMDb, and MIMIC-III, covering affective prediction, movie genre classification, and clinical prediction tasks.Source: §C.1.1 · pp. 29–30

C.1.2 MULTIMODAL RETRIEVAL

  • Cross-modal retrieval evaluates whether representations from different modalities can be aligned to retrieve semantically similar data, such as images from text queries.Flickr-30K supports text-to-image reference resolution, entity localization, and bidirectional image-caption retrieval.Source: §C.1.2 · p. 30

C.1.3 MULTIMODAL QUESTION ANSWERING

  • Multimodal question answering tests alignment between question words and entities in visual, video, or embodied inputs.The section includes CLEVR for visual reasoning and VQA 2.0, whose complementary image pairs reduce spurious correlations.Source: §C.1.3 · p. 30

D ADDITIONAL EXPERIMENTS AND DETAILS

Additional experiments evaluate human simulation of model predictions and the local faithfulness of cross-modal interaction visualizations. Results vary by task and model, while the second-order-gradient approach is reported as faster than DIME.

D ADDITIONAL EXPERIMENTS AND DETAILS

  • The second-order-gradient method is faster than DIME because it requires one model run rather than up to 10,000.VQA visualization generation was much slower than visualization for MM-IMDb and CMU-MOSEI, which used second-order gradients instead of DIME.Source: D ADDITIONAL EXPERIMENTS AND DETAILS · p. 31

D.1.1 SETUP

  • For VQA 2.0, the human simulation used 22 validation examples and compared five settings that progressively added analysis stages through the full MultiViz pipeline.Annotators predicted LXMERT outputs using webpages, with only 20 of the 22 examples counted in the study.Source: §D.1.1 · pp. 31–32, §D.1.2 · p. 32

D.1.3 MM-IMDB

  • The simulation experiments used task-specific model and data settings, including single-label analysis for MM-IMDb and binary sentiment classification for CMU-MOSEI.MM-IMDb points came from the test split; CMU-MOSEI labels grouped -1 to -3 as Negative and 0 to 3 as Positive.Source: §D.1.3 · p. 32, §D.1.4 · p. 36

D.1.2 VQA 2.0

  • Across the reported settings, annotators given more information almost always achieved higher average accuracy and inter-rater agreement; settings (4) and (5) achieved perfect accuracy and agreement when including global analysis Rg.Annotators in settings (3), (4), and (5) found the Features page most helpful compared with Overview 31.7%, 61.7%, and 80.0% of the time, respectively.Source: §D.1.2 · p. 32, §D.1.4 · p. 38

D.2.1 CLEVR

  • The CLEVR faithfulness evaluation selected 52 ground-truth phrase–object alignments and measured whether the aligned object ranked among the top one or two bounding boxes.The study used a near-perfect model and a lower-performing late-fusion model.Source: §D.2.1 · pp. 38–39
  • Over 80% of CLEVR alignments were recovered by the top two bounding boxes for near-perfect MDETR, while CNN+LSTM+SA was less likely to recover correct alignments.The authors interpret the MDETR result as evidence that the method is faithful to the model’s prediction process.Source: §D.2.1 · p. 39, Figure 26 · p. 39

D.2.2 FLICKR-30K

  • On Flickr-30K, second-order gradients achieved 44% matching accuracy for ViLT versus 34% for random matching; CLIP achieved 35%.For CLIP, scattered gradients made it difficult to localize one object; the authors identify quantifying intermediate interactions as a future direction.Source: §D.2.2 · p. 39, Figure 27 · p. 39, Figure 28 · p. 40

D.3 REPRESENTATION INTERPRETATION

  • The representation-interpretation study asks whether local and global visualizations let humans assign natural-language concepts consistently to previously uninterpretable features.Source: §D.3 · p. 40

D.3.1 VQA 2.0

The VQA 2.0 study tests whether local and global feature visualizations help annotators interpret representation features. More complete visualizations were associated with more consistent and confident concept assignments.

D.3.1 VQA 2.0

  • The study collected 180 annotations across 12 features and 15 annotators, manually clustered them into 29 concepts, and measured agreement with Krippendorff’s alpha.Each of three groups of five annotators received a different visualization setting.Source: §D.3.1 · pp. 40–41, Figure 32 · p. 42
  • With full MULTIVIZ visualizations, annotators completely agreed on 7 of 12 features, and agreement and confidence increased as they received more information.The authors attribute particular value to global examples and their visualizations.Source: §D.3.1 · pp. 41–42
  • With only one local example, annotators disagreed among concepts including “mirror,” “brushing teeth,” “bathroom,” and “material,” and reported relatively lower confidence.Source: §D.3.1 · p. 42
  • With local and global information but no visualization (Rℓ+ Rg), users used the original seed datapoint plus two similar datapoints; 3/5 identified ‘asking about material,’ while two gave another valid shared concept.Source: §D.3.1 · p. 42
  • All 5 annotators identified the feature as representing “asking about material” when local and global examples included unimodal and cross-modal visualizations.Text importance highlighted “counter,” “countertop,” and “wall,” while image cross-modal interactions highlighted those entities.Source: §D.3.1 · p. 43

D.3.2 CLEVR / D.3.3 MM-IMDB / D.4 ERROR ANALYSIS / D.4.1 SETUP We present three categories of errors:

  • The appendix also shows feature-concept examples for CLEVR and MM-IMDB and describes categorizing model errors as perception, cross-modal interaction, or prediction failures.Ten annotators categorized errors with or without MULTIVIZ visualizations; the study used 24 incorrect VQA examples.Source: §D.3.2 · p. 43, §D.3.3 · p. 43, §D.4.1 · pp. 43–44

D.4.2 VQA 2.0

The error-analysis studies ask annotators to classify model mistakes by stage, with and without MULTIVIZ visualizations. On both VQA 2.0 and CLEVR, annotators were more confident and agreed more often with MULTIVIZ.

D.4.2 VQA 2.0

  • With MULTIVIZ, VQA annotators were more confident and agreed more often when classifying errors; they found second-order gradients on selected words especially helpful for judging cross-modal interactions.They also used unimodal-stage visualizations and FRCNN bounding boxes to assess perception.Source: §D.4.2 · p. 44
  • Among 23 VQA errors, annotators classified an average of 8.8 as perception errors, 6.8 as cross-modal interaction errors, and 7.4 as prediction errors.The authors suggest that unimodal encoders and visual-text alignment are possible improvement directions.Source: §D.4.2 · p. 44
  • MULTIVIZ visualizations helped annotators distinguish correct unimodal importance with correct image-text interaction from cases where the interaction failed.In one example, all annotators judged correctly highlighted “cheese” and “pizza” plus the pizza bounding box as a prediction error; in another, missing interaction highlighting around a man’s jeans indicated an interaction error.Source: §D.4.2 · p. 45

D.4.3 CLEVR

  • On CLEVR, annotators were more confident and agreed more often with MULTIVIZ than without visualizations.Source: §D.4.3 · p. 46
  • Of 10 CLEVR errors, annotators classified an average of 6 as cross-modal interaction errors, suggesting difficulty aligning text phrases with their referents in this late-fusion model.Source: §D.4.3 · p. 46

D.5.1 VQA 2.0

The authors use MULTIVIZ to locate a color-related LXMERT error pattern and select additional training examples. On the targeted color test set, MULTIVIZ-guided data improved accuracy more than random or uncertainty-based sampling.

D.5.1 VQA 2.0

  • Two of five selected penultimate-layer neurons were consistently associated with color questions, and blue-related errors accounted for 8.8% of LXMERT errors.LXMERT accuracy was 5.5% on points where blue was either the correct answer or the model’s prediction; the authors observed that the model could capture unimodal and cross-modal interactions but fail at prediction.Source: §D.5.1 · p. 46
  • The debugging study selected additional examples from an unlabeled pool using features associated with color, non-color features, or uncertainty, then fine-tuned LXMERT’s last layer and evaluated on held-out data.The study compared five selection settings and repeated each setting 10 times.Source: §D.5.1 · p. 47
  • On the targeted color test set, MULTIVIZ selection improved accuracy by 30%, compared with 1.4% for random sampling and 0.0% for uncertainty sampling.Using one color-related feature improved performance by 27%, while using two improved it by 30%.Source: §D.5.1 · p. 47

D.6 SUMMARY OF TAKEAWAY MESSAGES

  • The authors summarize that MULTIVIZ supports feature interpretation, error localization, and debugging, while second-order gradients matched internal alignments over 80% of the time on MDETR using top-2 alignment accuracy.They also report improved human simulation accuracy and agreement with more visualization stages.Source: §D.6 · p. 48

E.1 LIMITATIONS

The authors identify prediction-class structure, gradient availability, and the scale of human testing as limitations. They also describe planned extensions to interpretation tools, evaluation, and stakeholder engagement.

E.1 LIMITATIONS

  • MULTIVIZ works best with a reasonable number of prediction classes: many classes can leave rare answers with zero sparse-model weights, while very few classes can make final-layer features overlap.For yes/no VQA subsets, the authors extended analysis to intermediate-layer features.Source: §E.1 · p. 48
  • MULTIVIZ currently requires categorical model outputs and gradients computable through the model, limiting support for architectures with discrete intermediate steps.The authors propose approximate gradients, such as perturbation or policy gradients, as a possible extension.Source: §E.1 · p. 48
  • The human studies span over 60 hours and close to 100 datapoints, so the authors characterize their conclusions about interpretation and debugging efficacy as preliminary.Participants had or were pursuing STEM graduate degrees and had machine-learning knowledge.Source: §E.1 · p. 48

E.2 NEW CLASSES OF INTERPRETATION TOOLS / E.2.1 CROSS-MODAL INTERACTIONS

  • Planned tool extensions include comparing post-hoc interpretation with interpretable-by-design models and exploring decision trees or other hierarchical prediction processes.Source: §E.2 · p. 49, §E.2.1 · p. 49

E.3 EVALUATING INTERPRETABILITY / E.4 ENGAGEMENT WITH REAL-WORLD STAKEHOLDERS

  • The authors plan to expand interpretability metrics and engage healthcare and affective-computing stakeholders to evaluate tool usefulness on MIMIC and CMU-MOSEI.They note that interpretability evaluation is subjective and its desiderata vary across populations, research fields, and time.Source: §E.3 · p. 49, §E.4 · p. 49
Loading 2207.00056v3…