PaperlayerView exact revision on arXiv

Source-linked AI summary

TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversations

Dylan Slack, Satyapriya Krishna, Himabindu Lakkaraju, Sameer Singh

arXiv:2207.04154v4cs.LGcs.AIcs.CL

TL;DR

Complex ML models are difficult to understand, and practitioners often struggle to choose and interpret explanations. TalkToModel uses natural-language conversations to explain tabular models, and the authors report high-accuracy language understanding and improved model understanding compared with existing systems.

  • Problem

    Complex ML models are hard to understand, and practitioners often struggle to choose explanations, interpret results, and answer follow-up questions.

    Source: §1 · pp. 1–2

  • Method

    TalkToModel enables open-ended natural-language dialogues about tabular datasets and classifiers, parsing utterances into operations that an execution engine runs.

    Source: §1 · p. 2, §4.1 · p. 12

  • Results

    TalkToModel comprehends users with high accuracy and helps users understand model predictions better than existing systems, including domain experts such as healthcare workers.

    Source: §2 · p. 3, §3 · p. 10

  • Takeaways & Limitations

    The authors conclude that natural-language conversations make explainable AI accessible to users from varied backgrounds, including healthcare workers seeking to understand model predictions.

    Source: §3 · p. 10

    Evaluation in doctors’ offices, laboratories, and professional settings where stakeholders use TalkToModel remains future work.

    Source: §3 · p. 10

Abstract

from arXiv · show

Machine Learning (ML) models are increasingly used to make critical decisions in real-world applications, yet they have become more complex, making them harder to understand. To this end, researchers have proposed several techniques to explain model predictions. However, practitioners struggle to use these explainability techniques because they often do not know which one to choose and how to interpret the results of the explanations. In this work, we address these challenges by introducing TalkToModel: an interactive dialogue system for explaining machine learning models through conversations. Specifically, TalkToModel comprises of three key components: 1) a natural language interface for engaging in conversations, making ML model explainability highly accessible, 2) a dialogue engine that adapts to any tabular model and dataset, interprets natural language, maps it to appropriate explanations, and generates text responses, and 3) an execution component that constructs the explanations. We carried out extensive quantitative and human subject evaluations of TalkToModel. Overall, we found the conversational system understands user inputs on novel datasets and models with high accuracy, demonstrating the system's capacity to generalize to new situations. In real-world evaluations with humans, 73% of healthcare workers (e.g., doctors and nurses) agreed they would use TalkToModel over baseline point-and-click systems for explainability in a disease prediction task, and 85% of ML professionals agreed TalkToModel was easier to use for computing explanations. Our findings demonstrate that TalkToModel is more effective for model explainability than existing systems, introducing a new category of explainability tools for practitioners. Code & demo released here: https://github.com/dylan-slack/TalkToModel.

1 Introduction

Practitioners face difficulty choosing and interpreting explanations, while complex models make predictions harder to understand. TalkToModel addresses these challenges with open-ended natural-language conversations about tabular models.

1 Introduction

  • Complex state-of-the-art models are hard to understand, although black-box models can be more flexible and accurate than inherently interpretable alternatives.Post-hoc techniques explain trained-model predictions by fitting locally faithful models or inspecting internal details such as gradients.Source: §1 · p. 1
  • Practitioners struggle to choose explanations, interpret their results, and answer follow-up questions; point-and-click tools still require expertise and lack flexible follow-ups.These difficulties make simple, intuitive interactions a key bottleneck in explainability adoption.Source: §1 · pp. 1–2
  • A general dialogue system must handle diverse model and data questions, preserve natural conversation flow across settings, and generate accurate responses.Prior natural-language explanation prototypes address specific explanations and model classes, limiting their applicability to general conversational settings.Source: §1 · p. 2
  • TalkToModel supports open-ended discussions about any tabular dataset and classifier, including prediction explanations, hypothetical changes, and prediction flipping for individual instances or groups.Users can ask follow-up questions that rely on conversational context.Source: §1 · p. 2
  • Its dialogue engine parses utterances into an SQL-like language, while execution runs operations, selects explanations by fidelity, and returns results through a text interface.The language represents explanations, error analyses, data manipulation, and descriptive text; lightweight fine-tuning supports adaptation to new datasets and models.Source: §1 · pp. 2–3, Figure 1 · p. 3

2 Results

TalkToModel is evaluated for parsing user utterances and helping people understand ML models compared with a dashboard. Fine-tuned T5 parses accurately, and users answer questions faster and more accurately with TalkToModel.

2 Results

Language Understanding

User Study: Utility of Explainability Dialogues

3 Discussion

TalkToModel supports model understanding through natural-language conversations, and the authors report high user-comprehension accuracy and better understanding than existing systems. They identify in-the-wild use, grounded responses, and workflow integration as future research directions.

3 Discussion

  • TalkToModel comprehends users with high accuracy and helps users understand model predictions better than existing systems, particularly healthcare workers using disease-diagnosis models.The authors describe the system as accessible to users from a range of backgrounds.Source: §3 · p. 10
  • Its extensible design and released code, data, and demo are intended to make it straightforward for users and researchers to build on the system.Source: §3 · p. 10
  • Future work should examine in-the-wild use, language-model responses grounded in operation results, and integration into professional workflows.Source: §3 · p. 10

4 Methods

TalkToModel translates user utterances into executable logical forms, runs operations to answer model-understanding questions, and allows designers to extend those operations.

4.1 Text Understanding

  • TalkToModel updates an executable grammar for each dataset, trains an LLM to parse utterances into it, and composes operation results into contextual responses.The responses also provide opportunities for follow-up questions.Source: §4.1 · pp. 11, 13
  • The grammar supports 30/31 prototypical XAI questions, excluding socio-technical questions beyond TalkToModel’s scope.The question bank includes questions about model inputs, outputs, performance, global behavior, individual predictions, and counterfactuals.Source: §4.1 · p. 11
  • Recursive previous-filter and previous-operation grammar operations resolve anaphora and ellipsis by using earlier conversation turns.They let users refer back to prior events or omit information inferable from the conversation.Source: §4.1 · p. 12
  • TalkToModel generates 20,000–40,000 utterance-parse pairs from wildcard templates and dataset details, so users need only provide a dataset to set up a conversation.The authors describe this automated approach as less costly and time-consuming than human annotation.Source: §4.1 · p. 12

4.2 Executing Parses

  • Feature-importance explanations provide per-feature attributions from a data point and model, while counterfactual and interaction explanations address other user questions.The system uses post-hoc explanations that rely on inputs and predictions rather than internal model details.Source: §4.2 · pp. 13–14
  • TalkToModel selects the explanation with the highest faithfulness, measured by prediction sensitivity to perturbing important features and summarized as area under the top-k fudge-score curve.When fidelities are close, explanation stability can break ties.Source: §4.2 · pp. 13–14
  • The execution engine supports filtering, what-if analyses, prediction and error inspection, group statistics, and descriptions of the model and dataset.It can summarize group error patterns by training a shallow decision tree on those errors.Source: §4.2 · pp. 14–15

Extensibility

  • Designers can add or change operations by modifying the grammar, adapting TalkToModel to settings with different user populations or needs.The system is described as modular and straightforward to extend to new settings.Source: Extensibility · p. 15

5 Broader Impact Statement

TalkToModel may be applied in financial, medical, or legal settings, but easier access to model explanations also raises risks around trust and inappropriate use.

5 Broader Impact Statement

  • The authors identify financial, medical, and legal applications as possible settings for TalkToModel.They suggest it could improve model understanding in these situations.Source: §5 · p. 22
  • Making model explanations easier to access could create false trust or lead inexperienced users to apply models inappropriately.The authors say the system includes measures such as qualifying inaccurate explanations and describing the model’s intended purpose.Source: §5 · p. 22

8 Competing Interests

The authors report no competing interests.

8 Competing Interests

  • The authors do not declare any competing interests.Source: §8 · p. 23

A Experimental Details

The appendix outlines additional experimental details and describes how TalkToModel’s grammar is designed and adapted to a dataset.

A Experimental Details

  • The appendix covers additional grammar, parsing-training-data, explanation-selection, crowdsourced revision, and user-study details.Source: A Experimental Details · p. 24

A.1 Grammar Details

  • The grammar uses operations and acceptable argument values, and is simplified toward natural-language-like parses because language models tend to translate such grammars better.For example, a question about the three most important features for people older than 35 becomes “filter age greater than 35 and topk 3.”Source: §A.1 · p. 24
  • For each dataset, grammar feature arguments use its feature names, categorical arguments use observed category values, and numeric arguments use numbers extracted from user utterances.Numeric values written as words, such as “fifty-five,” are also supported.Source: §A.1 · p. 24

A.2 Training Dataset Generation

Training examples are built by instantiating diverse utterance templates, with limits on wildcard values to prevent a few templates from dominating.

A.2 Training Dataset Generation

  • The training set draws on 687 templates spanning the operations permitted by the system.The templates use wildcard slots for feature names and values, classes, explanation types, and filtering expressions.Source: §A.2 · p. 25
  • Numeric values are limited to two per feature, and wildcard values are downsampled to two after enumeration to prevent template imbalance.The authors report that preventing a few templates from dominating improves performance.Source: §A.2 · p. 25
  • One template asks for explanations of predictions on data where a numeric feature exceeds a specified value.The corresponding parse filters on that feature and value, then requests feature-importance explanations.Source: §A.2 · p. 25

A.3 Gold Parsing Dataset

The appendix documents the gold parsing data, user-study task design and compensation, and additional experiments.

A.3 Gold Parsing Dataset

  • Table 1 gives 10 worker-revision sentence examples from each dataset's 50 prompts, while Table 2 reports gold-pair counts by split.The datasets contain relatively more IID than compositional questions.Source: Table 1 · p. 25, §A.3 · p. 25, Table 2 · p. 26

A.4 User Study Questions

  • Participants received similar-difficulty question blocks with different numbers; TalkToModel and dashboard block assignment and ordering were randomized.Participants saw one block for each system.Source: §A.4 · p. 26, Table 3 · p. 26

A.5 User Study Length & Compensation

  • The survey took about 30 minutes; ML professionals received a $20 gift card and Prolific workers averaged $14.74 per hour.Source: §A.5 · p. 26

B Additional Experiments

  • The additional experiments cover explanation selection, training-template quantity, parsing error analysis, and further user-study results.Source: B Additional Experiments · p. 26

B.1 Advantages of Explanation Selection

The appendix benchmarks TalkToModel's explanation-selection method against LIME and SHAP using faithfulness metrics across three datasets. Selection performs better across almost all reported settings, with a German Credit PGU exception.

B.1 Advantages of Explanation Selection

  • Explanation selection performs better than baseline methods across almost every dataset and metric; on German Credit PGU, it is not significantly better than the best LIME setting.The table reports the best statistically significant results; selection is better than SHAP in the German Credit PGU setting.Source: Table 4 · p. 27, §B.1 · p. 27
  • PGI rewards larger prediction changes after perturbing influential features, while PGU rewards smaller changes after perturbing non-influential features.Source: §B.1 · p. 27
  • The comparison covers Diabetes, COMPAS, and German Credit, tests four LIME kernel widths, and averages three explanation runs per data point.PGI uses the most influential 50% of features and PGU the least influential 50%.Source: §B.1 · p. 27

B.2 Effects of the number of training templates

The appendix tests how the number of training templates affects parsing accuracy by repeatedly retraining T5-Base on Diabetes with randomly selected template subsets.

B.2 Effects of the number of training templates

  • T5-Base was retrained five times at each of five template-set sizes, from 20% to 100% of the original set.Each run randomly down-sampled the prompts before retraining.Source: §B.2 · p. 27, Figure 1 · p. 28
  • Accuracy improves as more templates are used; compositional gains appear to level off, while IID gains do not.The reported trend is shown for the Diabetes dataset.Source: Figure 1 · p. 28, §B.2 · p. 28

B.3 Parsing Error Analysis

The error analysis suggests few-shot GPT-J’s parsing errors stem from model capacity to learn from examples, rather than nearest-neighbor selection. The study used GPT-J models up to 6 billion parameters, so larger models might perform better.

B.3 Parsing Error Analysis

  • For 10-shot GPT-J, prompts included the operations needed to parse utterances, suggesting model capacity—not example selection—likely explains the errors.The broader comparison found fine-tuned T5 models performed considerably better than few-shot GPT-J.Source: §B.3 · pp. 28–29
  • Fine-tuned T5 models performed considerably better than few-shot GPT-J, motivating an analysis of why the few-shot approach underperformed.The error analysis tested whether nearest-neighbor selection failed to supply the needed parsing operations.Source: §B.3 · p. 28
  • The study used GPT-J models up to 6 billion parameters; the authors suggest larger models might achieve better results.This possibility is not established by the evaluated models.Source: §B.3 · p. 29

B.4 User Study Results: Per Question Likert Scores

Medical workers generally preferred TalkToModel while answering questions, and their trust ratings rose over the survey; ML professionals’ interface preferences were more mixed. The appendix also documents question-grammar coverage and screenshots of TalkToModel and its dashboard baseline.

B.4 User Study Results: Per Question Likert Scores

  • Medical workers clearly preferred TalkToModel while answering questions, and their trust ratings rose from “somewhat agree” to “agree” by the survey’s end.ML professionals were more mixed in interface preference, despite greater accuracy with TalkToModel and similar confidence and trust ratings.Source: §B.4 · p. 29, Figure 2 · p. 30, Figure 3 · p. 31

B.5 XAI Question Bank

  • The TalkToModel grammar parsed 30/31 core prototypical XAI questions, excluding socio-technical questions, and supports resolving some coreference through a previous_filter operation.Supplementary tables present prototypical questions, their parses, and explanations of those parses.Source: §B.5 · p. 29, Table 6 · p. 32, Table 7 · p. 33

C Interface Screenshots

  • The appendix introduces screenshots of both interfaces, with the dashboard described as a point-and-click baseline for understanding ML models.Its pages include global feature importances and individual predictions, which require navigating to the page and entering a data-point index.Source: C Interface Screenshots · p. 34, §C.2 · p. 36, Figure 7 · p. 36

C.1 TalkToModel Interface

  • TalkToModel’s interface is a text chat where users enter prompts, receive responses, pin messages, and can request a random question from a chosen category.Examples show a context-aware “why?” discussion and use for inspecting incorrectly predicted instances.Source: §C.1 · p. 34, Figure 4 · p. 34, Figure 5 · p. 34

C.2 Dashboard Interface

  • The dashboard opens on global feature importances and offers tabs for predictions, metrics, feature importances, and what-if scenarios.Inspecting an individual prediction requires opening that page and entering the data-point index.Source: Figure 7 · p. 36, Figure 8 · p. 36, §C.2 · p. 36
Loading 2207.04154v4…