Source-linked AI summary
Designing for Responsible Trust in AI Systems: A Communication Perspective
Q. Vera Liao, S. Shyam Sundar
TL;DR
The paper addresses limited understanding of how people form trust in AI and how different users can make inaccurate trust judgments. It introduces MATCH, a communication-centered conceptual model of trustworthiness cues and their processing, while highlighting user vulnerabilities and unresolved understanding of such cues.
Problem
Trust-in-AI discourse has insufficiently examined human trust judgments, although the same AI technology can be judged differently and some users can form inaccurate judgments.
Method
The paper develops MATCH, a communication-centered conceptual model describing AI trustworthiness cues and how people process them to make trust judgments.
Results
MATCH brings a communication perspective to trust in AI and highlights that trustworthiness cues can help calibrate user trust while users’ cognitive processes have potential limitations.
Takeaways & Limitations
The model is intended to support understanding of responsible trust in AI by considering communication and the differing ways users process trustworthiness cues.
Takeaways & Limitations
The paper identifies limited understanding of what constitutes trustworthiness cues in AI systems and notes that some user groups are more vulnerable to harms such as over-trust.
Abstract
from arXiv · showhide
Current literature and public discourse on "trust in AI" are often focused on the principles underlying trustworthy AI, with insufficient attention paid to how people develop trust. Given that AI systems differ in their level of trustworthiness, two open questions come to the fore: how should AI trustworthiness be responsibly communicated to ensure appropriate and equitable trust judgments by different users, and how can we protect users from deceptive attempts to earn their trust? We draw from communication theories and literature on trust in technologies to develop a conceptual model called MATCH, which describes how trustworthiness is communicated in AI systems through trustworthiness cues and how those cues are processed by people to make trust judgments. Besides AI-generated content, we highlight transparency and interaction as AI systems' affordances that present a wide range of trustworthiness cues to users. By bringing to light the variety of users' cognitive processes to make trust judgments and their potential limitations, we urge technology creators to make conscious decisions in choosing reliable trustworthiness cues for target users and, as an industry, to regulate this space and prevent malicious use. Towards these goals, we define the concepts of warranted trustworthiness cues and expensive trustworthiness cues, and propose a checklist of requirements to help technology creators identify appropriate cues to use. We present a hypothetical use case to illustrate how practitioners can use MATCH to design AI systems responsibly, and discuss future directions for research and industry efforts aimed at promoting responsible trust in AI.
1 INTRODUCTION
The paper argues that responsible trust in AI requires attention not only to model trustworthiness, but also to how trustworthiness is communicated and judged by diverse users. MATCH frames AI-system design around trustworthiness cues, their cognitive processing, and safeguards against inappropriate or deceptively induced trust.
- The problem: Trust in AI is a human judgment shaped by communicated cues, so the same technology can produce different and inaccurate trust judgments across people.These judgments affect how users interact with and rely on AI systems.
- The problem: Technically sound explanations can still produce harmful over-trust and over-reliance, particularly among AI novices and users in cognitively constrained settings.The paper also identifies certain personality traits as associated with vulnerability to these harms.
- MATCH: MATCH models trust communication through underlying trustworthiness attributes, AI-system affordances, trustworthiness cues, and users’ cognitive processing.It draws on communication and human-factors literature and focuses on systems rather than standalone models.
- MATCH: AI-generated content, transparency, and interaction provide distinct affordances through which systems present trustworthiness cues to users.Transparency enables trust judgments but does not inherently warrant trust, while interaction design shapes trust beyond the system’s content.
- Responsible cue design: Users may process trustworthiness cues analytically or through heuristics, enabling fast judgments that can sometimes be flawed.The paper therefore urges creators to account for target users’ cognitive processes when designing cues.
- Responsible cue design: Warranted cues should support calibrated trust and truthful communication, while expensive cues impose creator-side costs intended to deter malicious deception.The paper proposes a checklist and calls for regulation, user empowerment, and social, organizational, and industrial mechanisms.
2 BACKGROUND AND RELATED WORK
Prior work characterizes trustworthy AI, institutional safeguards, trust formation, and technology affordances, but leaves the cognitive and communicative pathways to user trust insufficiently integrated. MATCH synthesizes these perspectives into a model linking trustworthiness attributes, information cues, cognitive processing, and contextual influences.
- Trustworthy AI and trust in AI: Trustworthy-AI research operationalizes principles through dimensions and technologies including ability, benevolence, integrity, fairness, explainability, auditability, safety, robustness, privacy, and transparency.Related frameworks map ethical principles onto qualities or technologies intended to support trustworthy AI.
- Trustworthy AI and trust in AI: Institutional and regulatory work emphasizes governance, independent oversight, auditable documentation, legal or ethical compliance, and public control.These mechanisms are presented as ways to construct signals of trustworthy AI and support public trust.
- Trustworthy AI and trust in AI: Existing work often remains detached from the cognitive mechanisms through which people make trust judgments and from designed system aspects that promote user trust.This creates a gap between implementing trustworthiness principles and promoting trust in use.
- Trust in technologies: Trust-in-automation research describes trust as mediated by displayed information, cognitive processing, reliance behavior, and individual, organizational, and cultural context.Trust can be processed analytically, analogically, or affectively, while reliance is influenced by workload, risk, time constraints, and system configuration.
- Trust in technologies: Appropriate trust includes calibration, resolution, and specificity, and depends on effective communication of system trustworthiness.Information displays shape trust judgments, while existing trust also affects how users select and interpret information.
- MATCH: MATCH synthesizes trustworthiness foundations, mediating information cues, dual-process cognition, and affordance-cue-heuristic theory to describe trust judgments in AI systems.The model treats technology affordances as capable of cueing trust-related heuristics.
3 MATCH: A CONCEPTUAL MODEL OF USER TRUST IN AI
MATCH models AI trust as a communication process in which model attributes are expressed through system affordances and trustworthiness cues, then processed through users’ cognitive mechanisms. It emphasizes that trust judgments can be noisy, heuristic-driven, and user-dependent, motivating warranted and expensive cues for responsible communication.
- MATCH structure: MATCH separates model trustworthiness attributes, system affordances that communicate cues, and users’ cognitive processing of those cues.The model’s name refers to model attributes, affordances, trustworthiness cues, and heuristics.
- Scope: The model focuses on trust in underlying AI models as an attitude, excluding institutional trust, trust in AI as a technology, and behavioral reliance.The paper acknowledges that individual, environmental, organizational, and cultural contexts also shape trust judgments and reliance.
- Model trustworthiness attributes: AI trustworthiness is organized around ability, intention benevolence, and process integrity.Examples include performance, fairness, robustness, improvability, intended use, compliance, and appropriate operational processes.
- Affordances and cues: Trustworthiness cues are information within a system that can contribute to trust judgments, including AI-generated content, transparency, and interaction.Transparency can help users reflect on multiple bases of trust, while interaction can provide cues even when features are disassociated from the underlying model.
- Cognitive processing: Users may process cues systematically or through trust-related heuristics, and these processes can produce errors, over-trust, or unequal trust judgments across user groups.The paper links over-trust from some explainable-AI features to explainability fashion and highlights greater risks for AI novices and cognitively constrained users.
- Responsible cue design: Warranted cues should support well-calibrated judgments, while expensive cues impose costs that are difficult to bear without the corresponding trustworthiness quality.The paper presents these concepts as bases for selecting and regulating cues against deceptive use.
4 USE CASE: USING MATCH TO DESIGN FOR APPROPRIATE TRUST IN AN AI SYMPTOM CHECKER
The HealthChecker use case applies MATCH to select and evaluate trustworthiness cues for patients and doctors. It shows that cue calibration depends on users’ abilities and heuristics, and that interface features can create inappropriate trust even when unrelated to model trustworthiness.
- Use case and users: HealthChecker is a symptom-checking AI app that suggests diagnoses for common diseases, serving patients and primary doctors as distinct user groups.The personas represent an AI-novice patient and a medically expert doctor who is moderately familiar and cautious with AI.
- Step 1: Model attributes: The designers assess model attributes including performance, fairness, robustness, improvability, intended use, privacy, and diagnostic process integrity.These attributes are considered in the first MATCH step, before selecting communication cues.
- Step 2: Trustworthiness cues: MATCH maps cues across AI-generated content, transparency, and interaction, including diagnosis suggestions, normative metrics, explanations, documentation, customization, and social feedback.The mappings connect cues to ability, process integrity, intention benevolence, and model-extrinsic evidence.
- Step 2: Trustworthiness cues: The designers identify sleek visual design and chatbot interaction as irrelevant cues that nevertheless contributed significantly to users’ trust.These features cannot be mapped to model trustworthiness attributes.
- Step 3: Cue calibration: The designers recommend iteratively examining cues with target users and using uncertainty information or multiple candidate diagnoses to mitigate unfounded heuristics.Think-aloud studies are proposed to empirically assess users’ processing and trust judgments.
- Step 3: Cue calibration: Diagnosis suggestions have high calibration likelihood for Jessie but low calibration likelihood and high over-trust risk for Eric.Jessie can systematically assess recommendation quality, whereas Eric may rely on machine or positive-confirmation heuristics.
- Summary and guidance: Practitioners can use MATCH’s four-step analysis to design systems that responsibly communicate true trustworthiness after identifying and understanding prototypical user groups.The analysis also supports retrospective investigation of inappropriate trust and specifies responsibilities for appropriate and equitable trust.
5 DISCUSSION: TOWARDS RESPONSIBLE TRUST IN AI
The discussion frames responsible AI trust as a communication and design problem: creators must select truthful cues that support well-calibrated judgments while addressing users’ cognitive limitations and deceptive design.
- MATCH brings a communication perspective to AI trust by connecting trustworthiness attributes, communicated cues, and users’ trust-judgment processes.The model draws on communication and human-factors literatures and decouples model trustworthiness from its communication.
- Responsible cue design requires truthfully and comprehensively communicating model attributes and using cues that target users are likely to process into well-calibrated judgments.The discussion defines these as two essential requirement sets for technology creators.
- Transparency and interaction can increase trust through intention- or process-based pathways even when a model’s ability should not be relied upon.This offers an alternative explanation for why transparency features may raise trust without necessarily indicating greater model ability.
- Because cues can be processed analytically or through heuristics, creators should study how different users attend to and process cues when judging trust.Empirical work should test cues across models with different trustworthiness levels to identify poorly calibrated judgments and reduce response-bias concerns.
- User empowerment can combine critical-assessment guidance with independent tools that expose cues creators may downplay or hide.Suggested tools include trustworthiness checklists, heuristic-awareness guidance, and augmenting visualizations of supporting evidence.
- Social, organizational, and regulatory mechanisms can provide supporting evidence through governance, provenance, track record, and other-user feedback, but responsible implementation requires truthful communication.The discussion also advocates regulating creators’ use of trustworthiness cues and developing broader assurance ecosystems.