A Systematic Review on Fostering Appropriate Trust in Human-AI Interaction

1 INTRODUCTION

• Over- and under-trust in AI are associated with misuse and disuse, which the paper links to risks of harm and negative consequences.The introduction cites examples including biased AI recruiting, a railroad accident, and facial recognition used to target communities.Source: §1 · pp. 1–2
• Prior approaches to fostering appropriate trust include confidence scores, explanations, alarms, warning signals, and uncertainty communication.The introduction also describes efforts to adjust trust to reflect a system’s trustworthiness.Source: §1 · p. 2
• The field lacks a comprehensive, cohesive account of appropriate trust, and consensus on its definition remains elusive across differing perspectives on trust, trustworthiness, and reliance.The introduction also notes confusion between appropriate trust and appropriate reliance.Source: §1 · p. 2
• The review examines appropriate trust’s history, definitions, related concepts, measures, methods, tasks, intervention results, and challenges.It uses a Belief, Intentions, and Actions (BIA) mapping to highlight commonalities and differences among related concepts.Source: §1 · pp. 2–3
• The paper contributes a BIA-based mapping, a synthesis of definitions and approaches, and research opportunities grounded in its analysis.The synthesis covers measures, tasks, and methods for building appropriate trust.Source: §1 · p. 3

2 BACKGROUND AND HISTORY OF APPROPRIATE TRUST

• The field evolved from automated machines and decision aids to robots and AI teammates, with early calibration defined as matching user reliance to system reliability over time.The section situates this history within a timeline of research from 1987 to 2022.Source: §2 · p. 3, Figure 1 · p. 4

2.1 1980-1990s: Over- and under-trust in automation

• Early automation research identified over-trust in incorrect advice and under-trust in control-system information, prompting training and concern about blind reliance.Researchers also linked over- and under-trust to operators’ limited understanding of system performance and highlighted misuse, disuse, and overuse as risks.Source: §2.1 · pp. 4–5

2.2 1990s: Introduction of HCI as a field and focus on appropriate trust

• Muir’s 1987 trust-dynamics model and later work examined calibration; system information improved confidence in reliability but had little effect on advice-taking, supporting user-centered design.Lee and Moray described calibration as correspondence between a person’s trust and automation capability; later work connected trust calibration to reducing automation dis- and misuse.Source: §2.2 · p. 5

2.3 2000s: Emergence of appropriate trust as a key topic of research

• Lee and See’s 2004 seminal article provided the first conceptual model relating calibration, resolution, and automation capability as factors in defining appropriate trust in automation.Source: §2.3 · p. 5

2.4 Parallel Developments: Influential Domains

• Psychology and philosophy shaped appropriate-trust research; psychology frames it as rational assessment of the risks and benefits of trusting, while disciplines share the goal of accurate trust assessment.Evans et al. found that older children were more sensitive to trustee characteristics and more discerning about when to trust.Source: §2.4 · p. 6
• In philosophy, appropriate trust is linked to epistemic responsibility and reasonable reliance on another’s goodwill, competence, and reliability.Jones and Hawley also emphasize calibrating trust in light of past experience, social norms, and situational factors.Source: §2.4 · p. 6
• Research interest in trust calibration developed more slowly than interest in promoting trust in automation, partly because trust is multidimensional and automation systems are complex.The review describes this as a contrast in research interest, not as a measured comparison.Source: §2.4 · p. 7
• Research interest in appropriate trust has grown drastically, likely driven by the increasing cognitive complexity of AI and ubiquity of interpersonal interactions, as well as organizational interest.The authors provide an overview of studies from 2012 through June 2022.Source: §2.4 · p. 7

3 SYSTEMATIC REVIEW METHODOLOGY

• The review followed a protocol with inclusion and exclusion criteria and used PRISMA guidelines to identify relevant articles.The paper states that the PRISMA process is illustrated in Figure 3.Source: §3 · p. 7

3.1 Search String

• An exploratory ACM/IEEE search for “appropriate trust” or “calibrated trust” articles from the last five years informed iterative keyword combinations; the timeline was broadened to the last decade (2012–2022).Source: §3.1 · pp. 7–8

3.2 Selection Criteria

• After two screening stages, 65 articles met the review criteria, including human involvement and a definition or measurable conceptualization of appropriate trust or a related concept.The process narrowed 1,697 search results to 169 articles before full-text examination and fine-grained screening.Source: §3.2 · p. 9

3.3 Corpus Overview and Analysis

• The final corpus contains 65 papers, with Human Factors and HRI as the most represented venues at eight papers each; 2022 coverage is incomplete.CHI and FAccT each contributed four papers, and the four named venues account for 47.3% of the corpus.Source: §3.3 · pp. 9–10, Figure 4 · p. 10

4 DEFINITIONS AND RELATED CONCEPTS

• The reviewed literature uses multiple definitions and overlapping terms, with calibrated trust (16 articles), appropriate reliance (8), and warranted trust (6) most common.Table 1 lists definitions that often differ, motivating the authors’ unified mapping.Source: §4 · p. 10

4.1 A Belief-Intentions-Actions (BIA) Mapping

• The BIA mapping separates trust-related concepts into beliefs, intentions, and actions, and represents both the human and AI agent.It places human trust beliefs about the agent’s trustworthiness at the belief level.Source: §4.1 · p. 12
• Appropriate trust is defined as a human’s belief about the AI agent’s trustworthiness matching the agent’s actual trustworthiness.The mapping groups existing definitions using similarities such as system performance or reliability, beliefs, and calculations.Source: §4.1 · p. 13
• The mapping is not exhaustive, and Table 1 terms do not always correspond one-to-one with its categories because authors define the same terms differently.The authors describe the framework as an early single framework for associating and categorizing concepts, intended to help reduce community discord.Source: §4.1 · pp. 13–14
• Calibrated trust involves correcting over- or under-trust over time, while appropriate trust is the maintained calibrated state but may occur without calibration.The review also describes calibrated trust as introducing dynamic trust and trust variations into interaction.Source: §4.1 · p. 13
• Warranted trust requires a causal relation between agent trustworthiness and human trust, whereas contractual trust concerns an explicit agreement and does not itself imply appropriateness.The review also distinguishes well-placed and responsible trust as intentions about how to act toward the agent.Source: §4.1 · pp. 13–14

• The review examines how included papers define and measure appropriate trust.Source: §5 · p. 14
• It also surveys the domains and tasks used in the included studies.Source: §5 · p. 14
• The section reviews methods for building appropriate trust and the results achieved.Source: §5 · p. 14

5.1.1 Perceived Trust.

• About 40% of papers matched system capabilities with users’ trust beliefs, most commonly manipulating system trustworthiness and using self-report scales.Miller [110] argues that manipulation is crucial for assessing calibration; static reliability matching may not capture trust’s dynamic nature.Source: §5.1.1 · p. 15

5.1.2 Demonstrated Trust.

• Only about 26% of studies used behavioral measures; approaches included agreement, switching to the AI’s prediction, and comparing behavior with an ideal trust level.Switching was usually combined with other measures, while ideal behavior could be estimated from the task and AI reliability.Source: §5.1.2 · p. 15

5.1.3 Mixed Approach.

• About 20% of studies combined self-report and behavioral measures, either pairing trust ratings with decisions or tracking calibration over time.Some studies found subjective trust ratings and reliance behavior diverged, underscoring that the measures can capture different aspects of trust.Source: §5.1.3 · p. 16

5.1.4 Synopsis.

• Appropriate-trust measures typically compare human trust with system trustworthiness or use agreement metrics, but a single measure may not capture trust’s multidimensionality.Behavioral measures can be context-specific, while subjective measures may reflect individual biases or reluctance to disclose true feelings.Source: §5.1.4 · pp. 16–17
• The review recommends examining combinations beyond perceived- and demonstrated-trust measures, since the most suitable mix may depend on the human-AI collaboration.Using validated questionnaires alongside behavioral measures may offer more insight than either alone.Source: §5.1.4 · p. 17
• Measures may include personality traits, past experiences, social norms, and cultural values.Source: §5.1.4 · p. 17
• Measures can differ across contexts and populations.Source: §5.1.4 · p. 17
• Studies should explicitly define and describe task context and study domains for measurement.Source: §5.1.4 · p. 17

5.2 Tasks

• Military operations, transport, and domain-agnostic applications are the most common domains; automated driving and prediction/classification tasks each appear 14 times.Reconnaissance appears 8 times, and military collaboration is a particularly preferred case.Source: §5.2 · p. 17
• Only three studies used non-controlled methods such as think-aloud sessions, co-design, or interviews, suggesting limited user and stakeholder input into design.Those studies targeted medical, mobility, and military experts.Source: §5.2 · p. 17

5.2.1 Risk.

• 78.3% of studies included risk, usually simulated through points gained or lost (45.7%) or performance-based bonuses (21.7%).Only one study used a task that was risky in the experimental setting itself; other studies relied on real-world task risk or did not discuss risk.Source: §5.2.1 · pp. 17–18

5.2.2 Dynamism.

• 63% of studies measured trust more than once, while a third measured it only once; some automated-driving studies derived continuous trust measures from driver behavior.Most reviewed studies were laboratory-based with simple tasks or theoretical models, leaving generalization to complex, dynamic real-world situations uncertain.Source: §5.2.2 · p. 18

5.2.3 Participant Expertise.

• 67.4% of studies recruited non-experts; all automated-driving studies recruited licensed drivers, but only three non-driving studies recruited experts.The review recommends closer alignment between participant expertise and expected end-user expertise.Source: §5.2.3 · p. 18

5.3 Methods for building appropriate trust (How to achieve it?)

• The review identifies four method categories: system transparency; cognition and perception; models, guidelines, theories and frameworks; and relational framing and continuum of trust.Source: §5.3 · p. 19

5.3.1 Improving System Transparency.

• About 52% of reviewed articles used transparency to inform users about AI capabilities and limitations, including explanations that can adapt to confidence and user behavior.Adaptive explanations were described as appearing for confident predictions and being absent for low-confidence ones to avoid over-trust; this approach was reported to support trust calibration.Source: §5.3.1 · p. 20
• Confidence scores (n = 12) were described as aligning users’ perceptions with actual AI trustworthiness and as effective for trust calibration, but may be poorly calibrated and do not necessarily improve AI-assisted decision-making.Source: §5.3.1 · p. 20
• Transparency also included verbal assurances and olfactory reliability displays, which communicated system promises, intent, or changing reliability to support trust calibration.The review also describes task- and context-focused situational-awareness communication; warnings about situational awareness reduced average trust-miscalibration periods by approximately 40%.Source: §5.3.1 · pp. 20–21

5.3.2 Cognition and Perception.

• Cognition and perception methods, used in 21% of reviewed papers, included training and performance observation to help users form more accurate mental models of AI.Training that emphasized system shortcomings was reported to calibrate expectations and trust; observing performance helped participants learn when to rely on AI confidence scores.Source: §5.3.2 · p. 21
• Nudges and cognitive forcing interventions gave users time to reflect and were reported to calibrate trust or reduce over-reliance compared with simple explainable-AI approaches.The review also reports that increasing automation’s humanness improved trust calibration, with compliance matching the aid’s actual reliability.Source: §5.3.2 · p. 21

5.3.3 Models and Guidelines.

• Theoretical models examine how actual and perceived AI trustworthiness relate, identifying performance, reliability, transparency, feedback, and context as influences on users’ perceptions.Source: §5.3.3 · p. 22
• Models also address communicating trustworthiness cues and enabling real-time trust calibration, including a proposed model of machine competence.The passage names the MATCH model for responsible trust and describes transparency and interaction as affordances for trustworthiness cues.Source: §5.3.3 · p. 22
• Most human-centred AI guidelines aim to build users’ trust rather than appropriate trust, and only the Google PAIR guidebook addresses trust calibration.Its considerations emphasize transparency and effective communication, including explaining system limits and risks of trusting false positives or negatives.Source: §5.3.3 · p. 22
• Trust models consider performance, reliability, transparency, feedback, and context; MATCH emphasizes transparency and interaction in trustworthiness cues.Real-time calibration models can adjust trust in response to task difficulty and system performance.Source: §5.3.3 · p. 22
• Selecting and adapting guidelines to the AI system’s context, domain, goals, and stakeholders is important because comprehensive models are complex and the many guidelines can overwhelm.Guidelines may focus on distinct concerns such as ethics, explainability, or fairness.Source: §5.3.3 · p. 23

5.3.4 Continuum of trust.

• Distrust can support calibrated trust, and trust decays without evidence of trustworthiness or un-trustworthiness.Perceived trust necessity and cognitive ability are identified as important antecedents for detecting trustworthiness cues.Source: §5.3.4 · p. 23
• Only 14% of reviewed papers examine the trust continuum, while studies use calibration points and critical states to identify when intervention may be needed.Researchers also infer over- or under-trust from reliance behavior rather than subjective trust measures.Source: §5.3.4 · p. 23

5.4 Results of calibration interventions

• Most reviewed papers supported transparency as facilitating appropriate trust, but explanations varied: some increased acceptance regardless of correctness, while only some improved calibration.The review also notes that confidence scores’ effects depended on whether people had enough unique knowledge to complement AI errors.Source: §5.4 · p. 24
• System interaction changes affected perceived trustworthiness, while nudging and cognitive forcing functions target cognitive and perceptual mechanisms involved in trust calibration.The review describes appearance-related strategies as challenging and says that trust’s effects on human–AI team performance and trust repair remain uncertain.Source: §5.4 · pp. 24–25
• Cognitive forcing functions reduced over-reliance on AI but were associated with worse human+AI team performance than AI alone; calibration training was linked to more robust trust over time.Nudging may make users more receptive and reflective, while the review recommends interventions that promote analytical thinking.Source: §5.4 · p. 24
• Human-like appearance sometimes increased perceived trustworthiness or benevolence, but did not reliably calibrate trust, leaving its appropriate use unresolved.One cited study found no trust difference despite greater perceived benevolence.Source: §5.4 · pp. 24–25
• Trust-continuum approaches helped calibrate trust in some situations, but cues did not necessarily improve team performance, and continuum methods also failed to reliably improve performance or repair trust.Calibration points and critical states prompted trust adjustment; miscalibration could also shape how new trustors approached an entity.Source: §5.4 · p. 25
• The review identifies empirical study of trust development over time across contexts, and its effects on human decision-making, as an opportunity for future research.It also calls for controlled studies testing how user expertise, task complexity, and explanation type influence calibration.Source: §5.4 · pp. 24–25
• Interventions produced mixed results: some supported appropriate trust, but did not consistently improve human–AI team performance or repair trust, leaving no one-size-fits-all solution.The authors recommend carefully defining appropriate trust and specifying how it is conceptualized and measured.Source: §5.4 · p. 25

6.1.1 Discord and diversity in understanding appropriate trust.

• Definitions of appropriate trust vary with research context, disciplinary perspective, and differing interpretations of the “right amount” of trust.The authors link this diversity to trust’s multidimensional nature and its association with attitudes or subjective beliefs.Source: §6.1.1 · p. 26
• Trust calibration is a process, whereas appropriate trust is the maintained state of calibrated trust across interactions.The authors propose a framework that considers system capability, trustworthiness, beliefs, task requirements, and context.Source: §6.1.1 · p. 26

6.1.2 Prominent focus on system’s capabilities in definitions.

• Definitions and research often foreground system ability, but the authors argue that trustworthiness also involves integrity, benevolence, process, and purpose.They suggest studying perceived integrity first and benevolence as the relationship develops, while continuing to consider ability.Source: §6.1.2 · pp. 26–27

6.1.3 Adequately Measuring Appropriate Trust.

• Some reviewed task designs included little or no risk, although the authors argue that trust research should account for vulnerability and participants’ personal stakes.They state that participants should experience or face the possibility of AI failure.Source: §6.1.3 · p. 27
• Studies often examine over-trust while overlooking under-trust; the authors argue both require equal consideration.They warn that a skewed focus on either scenario can lead to sub-optimal outcomes.Source: §6.1.3 · p. 27
• Behavioral experiments account for 70% of experiment designs, but their ability to capture trust fully is unclear given simplified settings and limited generalizability.The authors also question whether single- or multiple-item questionnaires robustly capture changes in trust over time.Source: §6.1.3 · p. 27
• Reliance is sometimes used as a proxy for trust or treated as equivalent, though the review distinguishes trust from discrete engagement or disengagement with AI.The authors also flag agreement with model predictions and situational awareness as potentially ambiguous proxies.Source: §6.1.3 · p. 27

6.2 Open Questions

• AI accuracy alone cannot define over- or under-trust, because trust also involves perceived reliability, transparency, multiple interactions, and beliefs about users’ interests.The authors caution that reliance on an AI output is not equivalent to trust.Source: §6.2 · p. 28
• A comprehensive trust evaluation should combine accuracy with transparency, interpretability, adaptability, longitudinal interaction, user feedback, and cognitive and emotional factors.The authors suggest this broader view can help track when trust moves toward over-trust or under-trust.Source: §6.2 · p. 28
• For non-binary decisions, the review found no articles clarifying how to calculate appropriate trust, and continuous outcomes make over- and under-trust boundaries difficult to set.The authors suggest considering task characteristics and assigning probabilities to possible outcomes.Source: §6.2 · p. 28
• The review found little work on how changing AI performance and task frequency or regularity shape trust gain and loss, limiting understanding of long-term real-world dynamics.The authors call for research on how task frequency and regularity affect trust dynamics.Source: §6.2 · p. 28

6.3 Novel perspectives

• A relational perspective links trust calibration to system-level outcomes such as cooperation and resilience, and frames the goal as supporting trusting through automation responsivity.The authors say automation responsivity also requires attention to human values, societal norms, and conflicts.Source: §6.3 · p. 29
• The authors suggest studying trust mis-calibration as a way to understand appropriate trust, including shifts in trust tactics in dynamic settings and interactions with multiple AI systems.They note that such trust-strategy changes are missing from human-AI studies they reviewed.Source: §6.3 · p. 29

6.4 Limitations

• The review searched a limited set of disciplines and English-language studies, so relevant research may have been missed and language bias may limit generalizability.The authors suggest future reviews include studies in other languages.Source: §6.4 · p. 29
• The BIA mapping is only one possible way to organize concepts related to appropriate trust, and the authors call for multidisciplinary mapping work.They also note that the search period ran from 2012 through June 2022, leaving later papers outside the review.Source: §6.4 · p. 29

• Different forms of appropriate trust are often only partly explained, and terms such as optimal, well-placed, meaningful, and justified trust can be unclear or used interchangeably with appropriate reliance.Source: §6.5 · p. 30
• The review recommends defining appropriate trust, distinguishing it from related concepts, and using the trust–reliance distinction proposed by Hoff & Bashir.Source: §6.5 · p. 30
• 38% of articles in the final corpus conceptualize appropriate trust or related concepts through trustworthiness, motivating attention to less-studied dimensions such as integrity and benevolence.Source: §6.5 · p. 31
• The review recommends measuring dynamic trust in practice and using situational reference points to maintain goal alignment in binary decision tasks.It notes potential issues when AI model targets diverge from human goals.Source: §6.5 · p. 31
• Single-focus approaches require ground truth, which is often unavailable, limiting their suitability for studying appropriate trust in human–AI interaction.Source: §6.5 · p. 31
• Around 37% of reviewed articles report the effect of improving system transparency for establishing appropriate trust; the review recommends post-experiment surveys or interviews about AI trustworthiness.Source: §6.5 · p. 31
• 43% of included articles report that the designed task’s objective directly influenced results; the review recommends controlling participants’ initial expectations about AI.Source: §6.5 · p. 31
• The review cautions that results may differ depending on whether the objective is AI-agent trustworthiness or accuracy, and that task design may influence human trust.Source: §6.5 · p. 31
• The review flags under-trust in AI systems and reports that 45% of user-study articles focused on detecting over-trust.Source: §6.5 · p. 31
• The authors recommend investigating under-trust in AI by adopting methods from the social sciences and psychology.Source: §6.5 · p. 31
• Around 10% of articles in the corpus followed established guidelines, and the authors recommend adopting such guidelines when designing user studies.Source: §6.5 · p. 31

7 CONCLUSION

• The review finds no single definition of appropriate trust and ambiguity around related concepts such as warranted trust and appropriate reliance.Source: §7 · p. 32
• The proposed Belief, Intentions, and Actions mapping highlights commonalities and differences among concepts related to appropriate trust.Source: §7 · p. 32
• The review identifies three common measurement techniques—Perceived, Demonstrated, and Mixed—and multiple domains with associated tasks.Source: §7 · p. 32
• Four common approaches to building appropriate trust are transparency, perception, guidelines, and studying the continuum of trust.Source: §7 · p. 32
• The review identifies challenges including diverse definitions and a strong focus on capability, and offers recommendations based on trends and challenges.The authors present these gaps as opportunities for future research.Source: §7 · p. 32