PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Explanations Through LLMs: Beyond Traditional User Studies

Francesco Bombassei De Bona, Gabriele Dominici, Tim Miller, Marc Langheinrich, Martin Gjoreski

arXiv:2410.17781v1cs.AI

TL;DR

Traditional XAI user studies are costly, time consuming, variable, and difficult to scale, while existing metrics do not consistently capture end-user understanding. This paper replicates a study of counterfactual and causal explanations with seven LLMs under varied memory and response-generation settings. LLMs reproduce many human-study conclusions, but alignment depends on the model, memory, and aggregation or variability conditions.

  • Problem

    XAI evaluation lacks standardized qualitative procedures and universal metrics that capture end-user understanding, while traditional user studies are costly and difficult to scale.

    Source: §1 · pp. 1–2

  • Method

    The paper replaces human participants with seven LLMs to replicate a user study comparing counterfactual and causal explanations under different memory and inference settings.

    Source: §1 · p. 2, §4.1.2 · p. 5

  • Results

    LLMs replicated most original conclusions, with Qwen 2 72B and GPT-4o Mini matching human conclusions in 6 of 9 cases, while absolute ratings still diverged from human responses.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    LLMs show potential as scalable and cost-effective complements or alternatives for qualitative XAI evaluation, provided models and experimental settings are selected carefully.

    Source: §1 · p. 2, §6 · p. 9

    The study relies on one public user study involving two similar XAI methods, limiting generalization across explanation techniques, domains, and forms of human assessment.

    Source: §5 · p. 9

Abstract

from arXiv · show

As AI becomes fundamental in sectors like healthcare, explainable AI (XAI) tools are essential for trust and transparency. However, traditional user studies used to evaluate these tools are often costly, time consuming, and difficult to scale. In this paper, we explore the use of Large Language Models (LLMs) to replicate human participants to help streamline XAI evaluation. We reproduce a user study comparing counterfactual and causal explanations, replicating human participants with seven LLMs under various settings. Our results show that (i) LLMs can replicate most conclusions from the original study, (ii) different LLMs yield varying levels of alignment in the results, and (iii) experimental factors such as LLM memory and output variability affect alignment with human responses. These initial findings suggest that LLMs could provide a scalable and cost-effective way to simplify qualitative XAI evaluation.

1 Introduction

XAI evaluation is increasingly important as opaque AI systems enter critical domains, but traditional qualitative studies remain costly, inconsistent, and difficult to scale. The paper proposes using LLMs to replicate human evaluations of counterfactual and causal explanations.

  • Opaque AI decisions in critical sectors such as healthcare increase the need for XAI tools that support trust, understanding, transparency, and accountability.Source: §1 · p. 1
  • Quantitative metrics for XAI lack a universal standard and may miss whether explanations help end users understand model decisions.Source: §1 · p. 1
  • Traditional user studies are costly, time consuming, variable, and difficult to scale or reproduce, limiting qualitative XAI evaluation.Source: §1 · p. 2
  • LLMs can provide preliminary XAI evaluations before expert validation, reducing recruitment demands and helping refine tools in domains such as healthcare.Source: §1 · p. 2
  • Across seven LLMs, the study finds that most original conclusions can be replicated, but alignment varies by model and by memory or output-variability settings.Source: §1 · p. 2

2 Traditional user studies to evaluate XAI tools

Traditional XAI user studies simulate end-user interaction through tasks, predictions, questionnaires, and trust assessments, but their diverse designs lack standardized evaluation procedures. They also require substantial resources and can produce variable, weakly generalizable results.

  • XAI user studies use varied activities, including predicting model outputs, rating explanation usefulness, and measuring trust before and after explanations.Source: §2 · pp. 2–3
  • Different studies compare aggregated outcomes across explanation types, settings, or no-explanation baselines to assess effectiveness, actionability, or trust.Source: §2 · p. 3
  • The field lacks a universally accepted structure or standardized metrics for conducting qualitative XAI user studies.Source: §2 · p. 3
  • Recruiting participants and simulating realistic settings are resource intensive, especially when domain experts such as clinicians are required.Source: §2 · p. 3
  • Small samples and variation in user experience or interpretation can produce inconsistent findings and make replication across institutions prohibitively expensive or time consuming.Source: §2 · p. 3

3 Can LLMs evaluate XAI tools?

The paper proposes querying LLMs as scalable substitutes or complements to human participants in qualitative XAI evaluation. Their usefulness depends on alignment with human preferences and on choices such as prompting, memory, and aggregation.

  • Using LLMs alongside specialized models and XAI tools is motivated by their contextual and conversational abilities, especially in niche domains such as healthcare.Source: §3 · p. 3
  • LLMs could replace or complement human participants by enabling repeated, inexpensive studies with large-scale data collection and analysis.Source: §3 · pp. 3–4
  • Useful LLM evaluations require alignment with human preferences, including agreement on comparative judgments and specific ratings.Source: §3 · p. 4
  • Model choice is consequential because training and capabilities produce different degrees of general and absolute alignment with human responses.Source: §3 · p. 4
  • Prompts, prompting techniques, conversation memory, and instruction order can alter output coherence, relevance, variability, and alignment with study goals.Source: §3 · p. 4

4 Experiments

The experiments transpose a human study of causal and counterfactual explanations to seven LLMs, testing alignment with human conclusions under different models, memory settings, and aggregation strategies. Results show partial replication, model-dependent alignment, and effects from memory and response variability.

4.1.1 User study

  • The replicated human study compares causal and counterfactual explanations across high- and low-familiarity scenarios using helpfulness, prediction, and confidence tasks.Participants rated explanation helpfulness, predicted model outputs, and reported confidence; the study included sixteen cases for each of the first two tasks.Source: §4.1.1 · pp. 4–5

4.1.2 Estimating human preferences with LLMs

  • Each LLM was treated as a participant under the same explanation and familiarity conditions, with multiple inference runs either averaged or treated as separate users.The aggregation approaches respectively stabilize mean responses or preserve run-level variability resembling individual human differences.Source: §4.1.2 · pp. 5–6

4.1.3 Metrics and statistical tests

  • The evaluation reproduces the original study’s mean comparisons and two-way ANOVA over familiarity, explanation type, and their interaction to measure absolute and general alignment.Normality assumptions were only partially satisfied for Mistral-Nemo-Instruct-2407, Mistral-7B-Instruct-v0.3, and GPT-4o-Mini, so their statistical outcomes require caution.Source: §4.1.3 · p. 6

4.2 Results

  • Six of nine statistical outcomes were replicated by Qwen 2 72B, with perfect general alignment on prediction tasks but weaker alignment for confidence questions.The model’s absolute responses still differed from human responses, particularly for confidence-related judgments.Source: §4.2 · p. 7, Figure 2 · p. 7
  • Absolute alignment was strongest for helpfulness and weaker for accuracy, especially in unfamiliar settings, while confidence showed the largest errors.Figure 3 compares MSE across helpfulness, accuracy, and confidence under familiar and unfamiliar causal and counterfactual conditions.Source: §4.2 · p. 7, Figure 3 · p. 7
  • Different LLMs produced different levels of general alignment, with Qwen 2 72B and GPT-4o Mini matching human conclusions in 6 of 9 cases.LLaMA 8B outperformed LLaMA 70B, whereas Qwen 2 72B outperformed Qwen 2 7B, so model size was not uniformly predictive across architectures.Source: §4.2 · p. 7, Figure 2 · p. 7
  • Memory generally improved concordance with human judgments but also introduced variability because responses incorporated prior interactions differently.Models without memory behaved more uniformly, while memory-based performance depended on how effectively prior context was used.Source: §4.2 · p. 8, Figure 4 · p. 8
  • Aggregation increased alignment in 11 of 14 cases, while non-aggregated responses—especially without memory—diverged more from the original user study.Aggregated models with memory maintained more consistent performance, whereas individual runs showed greater variability in confidence and prediction accuracy.Source: Figure 4 · p. 8, §4.2 · p. 9

5 Limitations

The study’s conclusions are constrained by its narrow evidence base, possible contamination from prior training data, and dependence on the evaluated LLMs. These limitations restrict generalization and require reassessment as models and alignment methods evolve.

  • A single user study covering two similar XAI methods limits how broadly the findings can generalize across explanation techniques and contexts.The authors note that XAI spans many techniques and application domains.Source: §5 · p. 9
  • Prior inclusion of the original study’s tasks or responses in LLM training could bias outputs and compromise result validity.The authors identify independence from prior study knowledge as a critical issue.Source: §5 · p. 9
  • Results may change with newer LLMs and alignment methods, so performance and human-preference alignment require repeated evaluation over time.Future models may differ in behavior, reasoning, and alignment capabilities.Source: §5 · p. 9

6 Conclusions

The paper investigates LLMs as complements to user studies for evaluating XAI explanations and reports promising initial results. LLMs reproduced most traditional-study conclusions, but alignment varied across models and depended on memory and response variability.

  • LLMs replicated most conclusions from the traditional user study, indicating potential as scalable and cost-effective alternatives for XAI evaluation.The study compared counterfactual and causal explanations.Source: §6 · p. 9
  • Different LLM architectures and capabilities produced varying outcomes, making model selection relevant to specific evaluation tasks.Source: §6 · p. 9
  • Memory use and response-generation variability significantly affected alignment between LLM outputs and human preferences.Source: §6 · p. 9

A Implementation details

The implementation used local Hugging Face models alongside GPT 4o Mini through the OpenAI API. Local inference used multi-GPU hardware and CUDA acceleration, while the full experiment required approximately 76 hours.

Loading 2410.17781v1…