Source-linked AI summary
Trust in AI: Progress, Challenges, and Future Directions
Saleh Afroogh, Ali Akbari, Evan Malone, Mohammadali Kargar, Hananeh Alambeigi
TL;DR
Trust in AI matters for adoption, yet research lacks a systematic account of its concepts, dimensions, metrics, and distrust drivers. This paper systematically reviews the literature, develops a taxonomy of technical and axiological trustworthiness measures, and identifies unevenly studied areas and open questions for future research.
Problem
The literature lacks a systematic review of trust and distrust in AI’s concepts, dimensions, impacts, metrics, and underlying considerations.
Method
The paper conducts a systematic literature review covering trust conceptions, human-AI interaction, technology acceptance, trustworthiness metrics, measurements, and trust makers and breakers.
Results
The review finds that trust-related AI research has developed unevenly across four major classes, leaving some areas with little or no attention.
Takeaways & Limitations
Future work should address underexplored areas and develop context-sensitive approaches for calibrating trust, including in AI-AI interaction.
Takeaways & Limitations
Trust-calibration research for AI-AI interaction remains limited, with transparency and explainability potentially less relevant than security and reliability.
Abstract
from arXiv · showhide
The increasing use of artificial intelligence (AI) systems in our daily life through various applications, services, and products explains the significance of trust/distrust in AI from a user perspective. AI-driven systems (as opposed to other technologies) have ubiquitously diffused in our life not only as some beneficial tools to be used by human agents but also are going to be substitutive agents on our behalf, or manipulative minds that would influence human thought, decision, and agency. Trust/distrust in AI plays the role of a regulator and could significantly control the level of this diffusion, as trust can increase, and distrust may reduce the rate of adoption of AI. Recently, varieties of studies have paid attention to the variant dimension of trust/distrust in AI, and its relevant considerations. In this systematic literature review, after conceptualization of trust in the current AI literature review, we will investigate trust in different types of human-Machine interaction, and its impact on technology acceptance in different domains. In addition to that, we propose a taxonomy of technical (i.e., safety, accuracy, robustness) and non-technical axiological (i.e., ethical, legal, and mixed) trustworthiness metrics, and some trustworthy measurements. Moreover, we examine some major trust-breakers in AI (e.g., autonomy and dignity threat), and trust makers; and propose some future directions and probable solutions for the transition to a trustworthy AI.
1. INTRODUCTION
Trust is central to AI acceptance because it shapes behavior, interaction, and adoption, while distrust can limit diffusion. This systematic literature review addresses fragmented understandings of trust and distrust in AI across technical, ethical, legal, and contextual dimensions.
- Motivation: Trust influences behavior, interaction, and technology acceptance, while AI’s broad diffusion makes trust essential to its adoption.AI performs human-like tasks and has become integral to everyday applications, services, and products.
- Motivation: Trust in AI regulates diffusion by reflecting people’s willingness to accept systems, rely on their suggestions and decisions, share tasks, provide information, and support the technology.AI development and adoption depend on satisfying stakeholder and user expectations and needs.
- Conceptual scope: Trustworthiness extends beyond ethics to AI performance, transparency, explainability, and compliance, while AI’s learning and proactive, unexpected, or incomprehensible behavior distinguishes it from other automated systems.Influential trust factors can be human-based, context-based, or technology-based.
- Motivation: Distrust hinders AI adoption across domains, with black-box behavior creating insufficient trust in manufacturing because ordinary users struggle to understand AI.The review frames distrust as a practical barrier despite AI’s potential in sectors such as manufacturing and medical imaging.
- Contribution: The study conducts a systematic literature review to organize trust and distrust dimensions, conceptualizations, relationships, and possible resolutions across existing research.It responds to a fragmented literature spanning studies, reports, and case studies in different domains.
2. METHODOLOGY
The review used an inclusive, systematic approach guided by PRISMA because no dedicated database for trust in AI exists. It combined manual and keyword-based Google Scholar searches, yielding 329 target papers after screening and duplicate removal.
- Search and review protocol: The review followed a PRISMA-based protocol to examine English-language academic papers, reports, case studies, and AI trust frameworks.The protocol was developed because no database specifically dedicated to trust in AI was available.
- Search strategy: Manual searching identified 19 trust-in-AI papers after duplicate removal, supplemented by keyword-based searches of Google Scholar.The keyword searches covered combinations of “trust” or “trustworthy” with “AI” or “Artificial Intelligence.”
- Screening and selection: The searches produced 336 relevant papers from 1,205 retrieved records, after semantic-keyword relevance screening.Duplicate papers were subsequently eliminated from the analysis.
- Screening and selection: The final systematic-review sample contained 329 target papers selected using stated inclusion and exclusion criteria, including publication in academic journals.The passage also indicates that topical dominance or substantial topical coverage was part of the selection criteria, although the criterion is truncated in the supplied text.
3. FINDINGS
The qualitative review organized the selected literature into major themes covering interaction, acceptance, explainability, empathy, fairness, technical trustworthiness, evaluation, trust frameworks, and trust makers or breakers. These findings span technical, ethical, legal, social, and human-agency dimensions of trust in AI.
- Trust in AI interactions: The review examined trust across human-machine, machine-human, and machine-machine interactions, including robo-advisors, autonomous vehicles, adversarial attacks, certification, and regulation.Four researchers developed eight major codes as the basis for categorizing the reviewed literature.
- AI technology acceptance: Trust and distrust were linked to AI acceptance across domains including healthcare, electronic markets, investment advice, personal assistants, and chatbots.The literature also addressed deception, racist or genocidal ideologies among developers, managerial endorsement, and cognitive versus emotional trust.
- Explainability, empathy, privacy, and fairness: Explainability and transparency research addressed opaque AI concepts, decision rationale, interpretability, transparency levels, and safeguards against overtrusting automated systems.Related themes included empathy, privacy, fairness, accountability, and the conditions supporting trust in AI systems.
- Technical trust metrics: Technical trust metrics focused on safety, accuracy, robustness, reliability, security, lineage, system accuracy, cybersecurity, and calibrating trust in high-stakes decisions.The literature also considered distrust, overtrust, malicious nodes, Bayesian trust models, and human-multi-robot teams.
- Trustworthiness measurement: Trustworthiness evaluation used psycho-physiological, empirical, theoretical, qualitative, experimental, and model-based methods, alongside certificates, audits, multidimensional metrics, and risk assessments.Reported criteria included performance, situation awareness, resilience, efficiency, data protection, predictability, and believability.
- Trust breakers and makers: Trust breakers included surveillance, manipulation, bias, privacy and security risks, loss of human control, autonomy and dignity threats, and unpredictable futures; trust makers included transparency, oversight, accountability, explainability, regulation, and expert endorsement.Trust-building approaches also incorporated AI personality, anthropomorphism, reputation, context, team and individual factors, human agency, and local justifications.
3.1. Trust in Human-Machine interaction: typology and parameters
Trust in human-machine interaction concerns how people anticipate AI systems’ correctness and confidence while accepting vulnerability, rather than primarily judging intentionality or benevolence. Its formation depends on perceived ability, observable decision processes, human traits, and domain-specific conditions, shaping AI acceptance across healthcare, finance, assistants, manufacturing, and robotics.
- Trust in Human-Machine interaction: typology and parameters: Trust in AI requires anticipating whether a model’s decision is correct and confident, linking trust to vulnerability, risk, uncertainty, and collaboration.Unlike general interpersonal trust, AI trust focuses on expected decision performance rather than necessarily anticipating the system’s behavior.
- Trust in Human-Machine interaction: typology and parameters: Perceived trustworthiness depends on AI ability—shaped by input data, problem representation, and algorithms—and increases when its observable decision process matches user priors.Actual trustworthiness belongs to the AI system, whereas perceived trustworthiness belongs to the user; trust and trustworthiness can therefore diverge.
- Trust in Human-Machine interaction: typology and parameters: Human-machine interaction places people as trustors and AI systems as trustees in applications including medical diagnosis, robo-advice, and autonomous driving.Human emotions and perceived AI humanness also influence trust, with users differing on whether limited empathy is beneficial or concerning.
- Trust in Human-Machine interaction: typology and parameters: Trust and adoption are also conditioned by interaction context: trust management filters information in AI-AI vehicle networks, voice-assistant adoption depends on interaction quality and trust, and organizations mediate trust through leaders, peers, and institutional influences.Social-robot trustworthiness combines robot characteristics and performance with human traits, needs, expertise, workload, prior experience, and situational awareness.
- Trust in Human-Machine interaction: typology and parameters: Trust shapes AI acceptance across domains: healthcare emphasizes transparency and explainability, while financial users often prefer human advice for complex or social situations,,,,.Healthcare users also value doctors’ holistic perspective and social interaction, whereas only half of people trust algorithmic investment advice.
3.2. Trustworthy AI and its metrics: Technical, and non-technical metrics · 3.2.1. Trust & explainability / transparency / interpretability
The paper frames trustworthy AI through seven non-technical and three technical metrics, emphasizing that opacity challenges trust while transparency, explainability, and interpretability address distinct aspects of understanding AI decisions. It also highlights the lack of consensus on defining and quantifying interpretability.
- 3.2. Trustworthy AI and its metrics: Technical, and non-technical metrics: Trustworthy AI is presented as necessary for AI systems in healthcare and other industries to be reliable, safe, and ethical.
- 3.2. Trustworthy AI and its metrics: Technical, and non-technical metrics: Trust-building measures in other domains include standard definitions, complaint systems, independent ratings, formal software validation, trust metrics, and roadmap development,,, [110],,;;; [115];.
- 3.2. Trustworthy AI and its metrics: Technical, and non-technical metrics: The review organizes trustworthy AI around 7 non-technical and 3 technical metrics, alongside measurement models and trustworthiness frameworks.
- 3.2.1. Trust & explainability / transparency / interpretability: Transparency is an ethical principle supporting trust, but it is distinct from explainability: transparency concerns how clearly a system reached an answer, whereas explanations pursue broader goals,,.
- 3.2.1. Trust & explainability / transparency / interpretability: Opaque deep neural networks limit understanding of how inputs become outputs, challenging explainability and users’ perceived trustworthiness despite their outstanding performance,,.
- 3.2.1. Trust & explainability / transparency / interpretability: Interpretability seeks to make the relationship between inputs and outputs understandable to stakeholders, sharing goals with but remaining distinct from explainability [150],, [152].
- 3.2.1. Trust & explainability / transparency / interpretability: Interpretability can begin before model selection through exploratory analysis, visualization, and feature engineering, and should reflect the needs of operators, executors, and examiners,,, [167].
- 3.2.1. Trust & explainability / transparency / interpretability: No consensual definition or quantification method for AI interpretability has yet been established, although the field is expected to reach readiness [152],,, [170].
3.2.2. Trust & empathy in AI
Empathy is presented as crucial to building trust between human users and AI systems, involving non-judgmental understanding of others’ feelings and subjective experiences. AI-based treatments and online services prompt reconsideration of empathy and trust in patient-doctor relations, with this link discussed in prior studies.
- Trust & empathy in AI: Empathy is considered crucial for building trust between human users and AI systems, involving non-judgmental understanding of others’ feelings (Wiesman 1996).It is also defined as the ability to simulate how others subjectively experience a situation (Gamer et al, 2010).
- Trust & empathy in AI: AI-based treatments and online communities motivate reconsidering empathy and trust in patient-doctor relations, while studies examine their link in online and AI-based services (,; Bock et al,; Yoon and Lee).
3.2.3. Trust and privacy · 3.2.4. Trust and fairness in AI
Trust in AI is shaped by tradeoffs between privacy and trust, while fairness concerns make unbiased treatment across user groups central to trustworthy AI. The review discusses context-dependent trust levels, technical privacy safeguards, and data or design failures that can produce algorithmic unfairness.
- 3.2.3. Trust and privacy: Higher privacy concerns are associated with lower trust, revealing a fundamental tradeoff between protecting personal information and trusting AI systems.
- 3.2.3. Trust and privacy: Privacy concerns self-determination over when, how, and how much personal or personally identifiable information is communicated.
- 3.2.3. Trust and privacy: Because required trust varies by context, AI applications may need high, medium, or low privacy levels.
- 3.2.3. Trust and privacy: Proposed technical responses to privacy–trust tensions include homomorphic-encryption-based secure computation, blockchain implementations, cooperative-information controls, and anonymous authentication with attack tracking.
- 3.2.4. Trust and fairness in AI: Algorithmic discrimination and bias hinder trust in AI, whereas perceived fairness or justice enhances it.
- 3.2.4. Trust and fairness in AI: Users should be assured that AI systems act justly and without bias toward all groups; fairness means equal or equitable treatment in AI decisions.
- 3.2.4. Trust and fairness in AI: Algorithmic unfairness can result from failing to develop AI systems with fair training data or fair design.
3.2.5. Trust and accountability in AI
Research on trust and accountability in AI emphasizes legal and institutional foundations alongside model reliability. Accountability research primarily “offloads” responsibility into defined stakeholder rights, developer obligations, enforcement mechanisms, and the right to explanation,,,,,.
- Legal and institutional trust: A robust legal framework is identified as necessary for establishing and maintaining trust in AI,,.Public trust depends not only on model reliability and individual recommendations but also on trust in institutions.
- Accountability in AI: Accountability research centers on “offloading”: clarifying human stakeholders’ rights, AI developers’ obligations, and how governments, developers, and watchdogs enforce them,,.The right to explanation is central to this accountability project,, [220],,,,,,, [227],.
3.2.6. Trust and technical metrics (safety, accuracy, robustness)
Trust in AI is framed through technical properties including reliability, safety, security, accuracy-related information quality, lineage, and robustness;. Proposed approaches emphasize calibrated reliance through interpretability and uncertainty awareness, while transparency mechanisms and formal models aim to strengthen accountability and trustworthy deployment,.
- 3.2.6. Trust and technical metrics (safety, accuracy, robustness): Technical trustworthiness in AI includes reliability, safety encompassing fairness and explainability, security, and lineage;.
- 3.2.6. Trust and technical metrics (safety, accuracy, robustness): Trust calibration for high-stakes law, medicine, and military decisions requires interpretable and uncertainty-aware AI systems.
- 3.2.6. Trust and technical metrics (safety, accuracy, robustness): Decision models can jointly account for trust and information quality, while FactSheets and supplier declarations promote transparency and accountability,,.
- 3.2.6. Trust and technical metrics (safety, accuracy, robustness): Blockchain-based graph approaches support trusted knowledge sharing by isolating malicious nodes and preventing knowledge pollution.
- 3.2.6. Trust and technical metrics (safety, accuracy, robustness): Formal trust models examine attacker behavior, robustness properties, hidden observations, Bayesian human–multi-robot trust, and human–robot factors,,,.Trust can improve AI design, development, and deployment, but cybersecurity does not guarantee trust in AI.
3.2.7. Evaluating and measuring/ trustworthiness certificate in AI
Trustworthiness in AI is evaluated through psycho-physiological, empirical, theoretical, and behavioral approaches using questionnaires, experiments, qualitative assessments, and performance measures. Research also develops trustworthiness metrics, certification processes, and standards to assess reliability, safety, transparency, and related properties.
- Evaluation approaches: Trust in human-AI systems is assessed through psycho-physiological and empirical methods, supported by theory and techniques including questionnaires, experimental protocols, and qualitative evaluations.These approaches aim to establish accuracy and safety guidelines for AI-assisted decision-making, although challenges can affect evaluation validity and accuracy.
- Trustworthiness metrics: Trustworthiness frameworks examine attributes such as reliability, safety, resilience, agility, vulnerability, and errors to establish confidence thresholds for AI systems.Ontology-based approaches consider relationships among vulnerabilities and errors, while quality evaluation incorporates vulnerability and risk assessments.
- Human-AI interaction: Trust evaluation also includes user signals, satisfaction, interaction preferences, and transparency among designers, testers, and users.A psycho-physiological model seeks accurate user-trust signals, while participants preferred physical interaction and embodiment over voice control in one human-AI interaction study.
- Trust measurement: Recommended AI trust scales emphasize predictability, reliability, efficiency, and believability, while assessment distinguishes trust in outputs from reliance on machine advice.Behavioral and physiological measures include individual and team performance, team situation awareness, and process measures.
- Trustworthiness certification: Certification processes are proposed to support algorithmic auditing, customizable AI certification, and education addressing AI safety concerns.Applying ISO standards to AI quality and security management in specific processes is presented as a way to enhance reliability and safety.
3.2.8. Trustworthy AI Frameworks
This section surveys trustworthy AI frameworks emphasizing accountability, privacy, explainability, consent, ethical interaction, and formal models of warranted trust across application domains. Figure 3 summarizes the range of frameworks proposed for trustworthy AI.
- Trustworthy AI Frameworks: Banavar centers trustworthy AI on secure, morally sound systems cultivated through repeated interaction, with accountability, adaptability, data integration, and privacy safeguards.The framework treats ethical considerations as central to human–AI interaction, consistent with the ethical framework discussed in and adopted in.
- Trustworthy AI Frameworks: Explainable frameworks use guidelines, user feedback, pattern identification, and result examination to connect research transparency with practical trust in air traffic management, medicine, logistics, and healthcare.These approaches address settings where AI processes are difficult to understand or trace and aim to improve acceptance of human–AI decisions.
- Trustworthy AI Frameworks: Trust models span human–AI decision similarity, conversational-agent antecedents, culturally informed interaction, ethical chat-system training, and user training for technology acceptance.The surveyed frameworks apply Naïve Bayes classification to behavioral patterns and autoregressive models to dialogue and negotiation datasets,,.
- Trustworthy AI Frameworks: Consent-based and metric-driven frameworks address explainability and trustworthiness challenges in contact tracing, big-data analysis, and public-health-emergency research.These proposals aim to support decisions about accepting AI outcomes through explicit consent and trustworthy metrics.
- Trustworthy AI Frameworks: Sociological and regulatory frameworks formalize warranted trust around user vulnerability and predictability, while European guidance links trustworthy AI to policy, rights, and consumer protection,.Related reviews also distinguish human from AI labor and address robot autonomy and the principles required for trustworthy systems,.
- Trustworthy AI Frameworks: Figure 3 presents the different frameworks that can be employed in trustworthy AI.The figure provides an overview of the framework landscape surveyed in this section.
3.3. Distrust in AI and Scary AI
Distrust in AI is driven by fears that increasingly capable systems may be untrustworthy under competition and by three major distrust-maker classes: surveillance and manipulation, autonomy and dignity threats, and unpredictable futures. Reducing distrust is difficult because transparency is necessary but explanations may be poorly understood, unattractive to users, or time-sensitive in healthcare.
- General sources of distrust: AI distrust reflects difficulty justifying trust in systems not trained or instructed for competitive circumstances, alongside broader fears about hypothetical AGI,,.AGI refers to hypothetical systems with broad, human-like plastic intelligence rather than task-specific algorithms.
- Major distrust makers: The section identifies three major distrust-maker classes: surveillance and manipulation, human autonomy and dignity threats, and unpredictable futures.These classes organize the section’s account of distrust in AI systems.
- Surveillance and manipulation: Surveillance distrust arises from privacy risks, possible data theft, and distrust of the companies or governments deploying AI and the reasons behind algorithmic recommendations,,,.Distrust can target both claims about how AI is used and the algorithm’s outputs and rationale.
- Human autonomy and dignity threat: Autonomy and dignity concerns center on AI reproducing bias and inequity or supplanting human agency, with agency loss among the highest-rated concerns of AI-distrustful people.These concerns reflect skepticism about AI’s effects on human agency and dignity, even though people may sometimes trust AI more than humans in healthcare and governance,.
- Unpredictable futures: Unpredictability fuels both global fears about AGI’s societal consequences and local concerns about whether such systems will possess appropriate values [291],,.The passage distinguishes global from local forms of unpredictability concern.
- Challenges to reducing distrust: Reducing distrust requires transparency [227],, but users may not engage with explanations, and their usefulness can be time-sensitive in healthcare,.The section notes that what transparency should entail remains unclear.
3.4. Trust makers: building/increasing trust in AI
Trust in AI can be increased through technical and axiological methods, but their effectiveness depends on human, system, and application-context factors. Explainability, communication, documentation, regulation, and interaction design can build trust, while poor explanations, limited accessibility, and ethics-washing can undermine it.
- Influential factors: Trust is influenced by human, AI-related, and context-related factors, including expertise, culture, personal traits, performance, reliability, and AI personality,,.Gender has conflicting effects,, whereas education and age are not important for building trust.
- Technical trust-building: Explainability can build trust, especially in critical domains, but global explanations improve later understanding while local justifications are effective yet time-sensitive.Explanations that are too detailed can confuse users, while insufficient detail fails to explain the system.
- Technical trust-building: Interactive visualization and result communication can increase trust, alongside adequate performance, interpretability, fairness, and bias-aware design and evaluation.Trust depends not only on system properties but also on how the system communicates its results to people.
- Axiological trust-building: Supplier declarations of conformity increase trust by documenting development, training, deployment, testing, intended use, non-tested scenarios, and ethical concerns [329].This transparency mainly benefits expert users and may disadvantage patients with low literacy backgrounds.
- Governance and accountability: Regulation and accountability are proposed as foundations for public trust, while ethics-washing—overstating corporate ethics to evade regulation—can damage trust, [333].Roszel et al. proposed 20 guidelines addressing efficacy, reliability, safety, and responsibility [333].
- Domain-specific implementation: Trust-building methods must fit application contexts: virtual agents integrated into explainable-AI interactions can increase trust, while human–agent value similarity shapes trust judgments,.Chatbot guidelines were developed from surveys of end-users and experts, and marketing research emphasizes psychological factors affecting acceptance of AI-generated information.
4. DISCUSSION
The discussion presents trust in AI as essential but distinct from technical trustworthiness, shaped by psychological, ethical, social, and domain-specific factors. It identifies unresolved challenges in accountability, measurement, privacy, transparency, fairness, and trust equity, while outlining directions for more integrated assessment and intervention.
- Conceptual foundations: Trust is essential for AI acceptance, but trustworthiness concerns technical ability whereas trust can also arise from non-technical factors such as reputation and documentation.AI trust operates across human–AI, AI–human, and AI–AI interactions, each with requirements beyond common trust factors.
- Trust determinants: Understanding trust requires relating performance, explainability, transparency, compliance, ethics, bias, discrimination, and privacy within each application domain.The discussion identifies understanding relationships among these factors as an unmet need for improving trust and technology adoption.
- Trust limitations: Excessive transparency can enable malicious users to game AI systems, while trust-damaging decisions by major AI companies can undermine public trust despite academic explainability efforts.The discussion characterizes trust as reciprocal: users must trust AI, and AI must also gain users’ trust.
- Trust limitations: Empathy may build trust while requiring private data, creating a privacy tradeoff in applications such as empathic AI treatment and voice imitation.Privacy encroachments can ultimately undermine the trust that empathy initially strengthens.
- Trust and reliability: Trust is subjective confidence, whereas reliability is objective functional performance; therefore, persuasive design can produce overtrust, while reliability alone may not motivate private-information sharing.Fairness can support objective trustworthiness, but perceived fairness has separate requirements and is also positively related to trust.
- Accountability and future directions: AI accountability remains difficult because systems make decisions and potentially provide explanations, while developers retain responsibility for accountability.Trust metrics also face challenges from AI’s changing behavior, cultural dependence, and limitations of questionnaires, surveys, and protocols; future assessment should combine empirical, psychophysiological, theoretical, and multilevel approaches, alongside a trust equity framework.
5. CONCLUDING REMARKS AND FUTURE DIRECTIONS
Trustworthy AI requires comprehensive, adaptable governance that accounts for AI-related, human-related, and context-dependent factors. Future research should address calibrated trust, AI–AI interaction, cultural and technological dynamism, and underexplored areas identified in the literature.
- Concluding remarks: Trust in AI should inform AI theory, robot design, human–AI interaction, and training for designers and users.The review identifies trust-related factors as metrics for developing AI technologies, while noting that isolated metrics are insufficient.
- Future directions: A comprehensive system should monitor algorithmic development and update principles as AI changes and as cultural contexts differ.Such a system should notify lawmakers when principles need modification for new technological or cultural conditions.
- Concluding remarks: Building trust requires evaluating AI-related, human-related, and context-related factors, with transparency, explainability, and performance especially important across applications.Some application-dependent and human-related axiological factors may enhance trust without improving the AI’s underlying trustworthiness.
- Future directions: Standards and regulations overseen by trustworthy agencies could promote calibrated trust and reduce overtrust or undertrust, including among people with limited technical knowledge.The review identifies a research gap in models for calibrating trust during AI–AI interaction, which has distinctive characteristics.
- Future directions: Figure 7 shows uneven attention across trust semantics, metrics, and measurement, leaving sparsely studied areas as fertile topics for future research.The conclusion is based on the reviewed papers and quantitative analysis across four major classes of trust-related AI research.