PaperlayerView exact revision on arXiv

Source-linked AI summary

Are we measuring trust correctly in explainability, interpretability, and transparency research?

Tim Miller

arXiv:2209.00651v1cs.HCcs.AI

TL;DR

XIT trust evaluations can treat increased trust as positive without establishing whether the model merits trust, and rating scales measure perceived rather than demonstrated trust. The paper discusses evaluation methods that manipulate model trustworthiness and assess whether XIT calibrates perceived and demonstrated trust. Its scope includes laboratory and field studies, but the approach requires manipulating trustworthiness.

  • Problem

    XIT studies may report trust ratings without establishing whether trust is warranted, while scales measure perceived rather than demonstrated trust.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    The paper discusses evaluating XIT interventions by manipulating model trustworthiness and measuring calibrated perceived and demonstrated trust.

    Source: §1 · p. 2

  • Results

    The paper argues that evaluating calibrated trust requires varying model trustworthiness and testing whether XIT calibrates trust for more- and less-trustworthy systems.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluation design generalizes to field studies, but measuring the effect of XIT on trust requires manipulating trustworthiness.

    Source: §6 · p. 10

    Laboratory incentives can simulate risk, but missing a small bonus is not comparable to risks such as invasive medical treatment.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

This paper presents an argument for why we are not measuring trust sufficiently in explainability, interpretability, and transparency research. Most studies ask participants to complete a trust scale to rate their trust of a model that has been explained/interpreted. If the trust is increased, we consider this a positive. However, there are two issues with this. First, we usually have no way of knowing whether participants should trust the model. Trust should surely decrease if a model is of poor quality. Second, these scales measure perceived trust rather than demonstrated trust. This paper showcases three methods that do a good job at measuring perceived and demonstrated trust. It is intended to be starting point for discussion on this topic, rather than to be the final say. The author invites critique and discussion.

1 INTRODUCTION

As machine use has expanded, trust research has often relied on participants’ ratings, which measure perceived rather than demonstrated trust. This paper argues that XIT evaluation should test whether interventions calibrate both perceived trust and demonstrated trust against model trustworthiness.

1 INTRODUCTION

  • As machine tasks have expanded, trust has become a concern across computer science and human factors.Earlier work on automation highlighted problems of over- and under-reliance.Source: §1 · p. 1
  • Trust evaluation should focus on properly calibrated trust because both over- and under-reliance can be problematic.The paper describes harm when a person’s trust or distrust does not align with whether a machine should be trusted.Source: §1 · p. 1
  • Perceived trust ratings are not the same as demonstrated trust, and existing trust-fall methods are not designed to measure XIT effects.The paper notes that perceived trust can affect appropriation, adoption, and reliance, while demonstrated trust involves a choice to rely.Source: §1 · p. 2
  • XIT evaluation should vary model trustworthiness and test whether interventions calibrate trust for more- and less-trustworthy systems.The authors argue that model trustworthiness cannot be known a priori, so evaluation should manipulate it.Source: §1 · p. 2
  • The paper aims to clarify XIT trust-measurement problems, share evaluation methods, and invite discussion of other approaches.The evaluation methods discussed are drawn from existing literature.Source: §1 · p. 2

2 RELATED WORK

Prior work measures trust through both participant ratings and reliance choices, but the reviewed approaches do not by themselves establish whether XIT interventions calibrate trust. The paper frames existing scales and demonstrated-trust measures as components of a broader evaluation process.

2 RELATED WORK

2.1 Measuring Perceived Trust

  • Trust scales, including one aimed at explainable AI, measure perceived trust but do not themselves measure the effect of an XIT intervention.The paper says such scales can be used within a larger evaluation process.Source: §2.1 · pp. 2–3

2.2 Measuring Demonstrated Trust

  • The trust fall game measures demonstrated trust through participants’ choice to rely on a machine’s answer or solve a problem themselves.Participants first form a mental model by observing or interacting with the machine.Source: §2.2 · p. 3
  • The trust fall game alone cannot show whether an XIT intervention works as intended because the appropriate level of trust in the agent is unknown.The paper notes that without knowing whether participants should trust or distrust the agent, the intervention’s effect cannot be judged.Source: §2.2 · p. 3

3 TRUST IN HUMAN-MACHINE SCENARIOS

The paper distinguishes trust from machine trustworthiness and frames human-machine trust around a person’s belief that a machine will uphold a contract in a context, together with acceptance of vulnerability. It argues that evaluation should seek calibrated trust rather than trust regardless of trustworthiness.

3 TRUST IN HUMAN-MACHINE SCENARIOS

  • Interpersonal trust is described as willingness to be vulnerable based on an expectation that another party will perform an important action.Source: §3 · p. 3
  • Human-machine trust concerns whether a person believes a machine will uphold a contract in a context and accepts vulnerability to its actions.The paper identifies context and what the machine is trusted to do as key elements.Source: §3 · pp. 3–4
  • A machine is trustworthy to a contract if it can maintain that contract, separating trustworthiness from a person’s trust or distrust.Source: §3 · p. 4
  • The goal is calibrated trust: avoiding unwarranted trust and unwarranted distrust, rather than maximizing trust regardless of context.Source: §3 · p. 4
  • The account of trust includes vulnerability and expectations, with the outcome important to the trustor; warranted trust and distrust are tied to the trustee’s trustworthiness.Source: §3 · p. 4
  • The paper’s model is simplified: it does not capture many realities of trust, which is tentative, dynamic, and not binary.The authors nevertheless consider it useful for showing that XIT research can fail to measure basic notions of trust.Source: §3 · p. 4

4 REQUIREMENTS FOR EVALUATING WARRANTED/UNWARRANTED TRUST/DISTRUST

The paper sets requirements for evaluating trust in XIT: assess tasks with measurable outcomes and participant risk, offer reliance choices when measuring demonstrated trust, and manipulate trustworthiness. Manipulating trustworthiness is necessary to judge whether an intervention calibrates trust.

4 REQUIREMENTS FOR EVALUATING WARRANTED/UNWARRANTED TRUST/DISTRUST

  • Participants must perform tasks with measurable outcomes.The paper says the tasks must have an outcome.Source: §4 · p. 5
  • Participants must be aware of, and vulnerable to, a downside from unwarranted trust or distrust.The paper says trust requires vulnerability to risk.Source: §4 · p. 5
  • To measure demonstrated trust, evaluations must let participants choose whether to rely on the machine.The paper distinguishes this requirement from measuring perceived trust alone.Source: §4 · p. 5
  • Evaluations must manipulate the machine’s known or estimated trustworthiness to test whether an XIT intervention calibrates trust.Without that manipulation, a higher-trust result does not establish that trust is warranted.Source: §4 · p. 5

5 MEASURING TRUST

The paper presents three protocols for evaluating how XIT interventions affect trust, extending the trust-fall idea. It distinguishes perceived trust from reliance and emphasizes calibrating trust against known differences in model trustworthiness.

5.1 Approaches to Measuring Trust

  • Questionnaires and interviews measure perceived trust, while choosing whether to rely on a system demonstrates trust; neither is the universally best measure.The paper recommends considering both and ideally checking whether perceived trust aligns with reliance.Source: §5.1 · pp. 5–6

5.2.1 Within-subject design.

  • In a within-subject design, participants learn about agents with differing trustworthiness, then choose which to rely on without seeing their test answers.Counterbalance agent order; choices across tasks provide reliance scores, and a survey can also measure perceived trust.Source: §5.2.1 · pp. 6–7, Figure 1 · p. 6

5.2.2 Between-subject design.

  • In a between-subject design, each participant sees one agent and chooses between relying on it or doing the task themselves.This removes ordering effects; a related design supports mediation analysis of trustworthiness, perceived trust, and reliance.Source: §5.2.2 · pp. 7–8, Figure 2 · p. 7

5.2.3 Manipulating trust at the instance level.

  • Instance-level designs manipulate task quality for one agent and score agreement between participant and agent against known ground truth.This allows finer-grained calibration analysis, but agreement is not reliance; a modified protocol can instead offer an agent-versus-self choice without showing the explanation or decision.Source: §5.2.3 · pp. 8–9

5.3 Vulnerability in laboratory experiments

  • Trust measurements require participants to face risk from agent failure; otherwise, reliance alone does not establish trust.The paper discusses performance bonuses, gamified task points, and ethically managed deception as ways to introduce or simulate stakes.Source: §5.3 · pp. 9–10
  • Low-stakes financial bonuses and game points can serve as risk proxies, but findings should not be generalized to high-stakes decisions.The paper cautions that small bonus risks are not comparable to invasive medical decisions and that game stakes are low.Source: §5.3 · pp. 9–10

6 DISCUSSION

The proposed trust-evaluation design requires intervening on model trustworthiness, including for perceived-trust measures; the authors also note its limits and the need to study trust over time. They invite critique and further methods.

6 DISCUSSION

  • Trustworthiness manipulation is needed for all three measures: questionnaires or surveys, interviews or focus groups, and reliance.The methods are not restricted to laboratory experiments and generalize to field studies.Source: §6 · p. 10
  • Without manipulating trustworthiness, the authors argue, evaluations do not truly measure trust in XIT methods.They believe such evaluations cannot be used in field studies where trustworthiness cannot be manipulated.Source: §6 · p. 10
  • Manipulating the underlying agent’s trustworthiness is also needed to assess whether an XIT method correctly affects perceived trust.Source: §6 · p. 10
  • Trust varies over time even in the presence of XIT methods, making longitudinal evaluation valuable but difficult.The authors call for longitudinal studies to measure trust and other crucial factors.Source: §6 · p. 10
  • The position paper invites critique, changes to the proposed methods, and new methods to advance discussion of XIT trust evaluation.Source: §6 · p. 10
Loading 2209.00651v1…