PaperlayerView exact revision on arXiv

Source-linked AI summary

The Role of Domain Expertise in User Trust and the Impact of First Impressions with Intelligent Systems

Mahsan Nourani, Joanie T. King, Eric D. Ragan

arXiv:2008.09100v1cs.HCcs.AIcs.LG

TL;DR

Users’ experience with system errors can shape their sense of accuracy, and the role of domain expertise in those first impressions is not well established. This study tests trust over time using an explainable arthropod classifier with different output orders. Early errors were associated with lower trust among experts, while novices trusted the low-accuracy system across orders.

  • Problem

    Because users infer system accuracy from experience, the ordering of errors may shape trust, but how domain expertise relates to those first impressions is underexplored.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    A controlled study had novice and domain-experienced participants review an explainable arthropod classifier under different orders of correct and incorrect outputs, with trust measured over time.

    Source: §1 · p. 1, §3.1 · p. 3, §3.5 · p. 5

  • Results

    Experts’ trust depended on early system performance, with negative first impressions associated with low trust, while novices trusted the low-accuracy system regardless of error order.

    Source: §5.1 · p. 7, §6 · p. 8, §5.3 · p. 8

  • Takeaways & Limitations

    First impressions and error detection can shape experts’ trust and willingness to use a system, while novices may over-trust when they cannot identify errors.

    Source: §5.1 · p. 7, §5.3 · p. 8, §6 · p. 8

    The study used classification tasks with objectively defined correct predictions, so findings require verification on exploratory or complex tasks where errors may be hard to define or detect.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

Domain-specific intelligent systems are meant to help system users in their decision-making process. Many systems aim to simultaneously support different users with varying levels of domain expertise, but prior domain knowledge can affect user trust and confidence in detecting system errors. While it is also known that user trust can be influenced by first impressions with intelligent systems, our research explores the relationship between ordering bias and domain expertise when encountering errors in intelligent systems. In this paper, we present a controlled user study to explore the role of domain knowledge in establishing trust and susceptibility to the influence of first impressions on user trust. Participants reviewed an explainable image classifier with a constant accuracy and two different orders of observing system errors (observing errors in the beginning of usage vs. in the end). Our findings indicate that encountering errors early-on can cause negative first impressions for domain experts, negatively impacting their trust over the course of interactions. However, encountering correct outputs early helps more knowledgable users to dynamically adjust their trust based on their observations of system performance. In contrast, novice users suffer from over-reliance due to their lack of proper knowledge to detect errors.

1 Introduction

Users form views of intelligent systems through experience, so the timing of errors may shape trust differently for people with different levels of domain knowledge. The study examines trust over time under different orders of system performance.

1 Introduction

  • Domain expertise can affect initial trust and error detection, while little domain knowledge has been associated with over-reliance on automation.Explanations are proposed as a way to improve transparency and help users understand system predictions.Source: §1 · p. 1
  • Users develop a sense of system accuracy through experience, making the timing of inevitable errors important to their impressions.Early observations may shape perceived accuracy before users develop an appropriate mental model.Source: §1 · p. 1
  • The study compares domain-experienced and novice users who encounter the same image-classifier outputs in different orders, measuring trust and its calibration over time.The two error-order scenarios place errors either at the beginning or the end of usage.Source: §1 · pp. 1–2

2 Related Work

Prior work considers trust calibration, automation bias, and domain expertise, but the relationship between expertise and the ordering of system outputs remains underexplored. This study focuses on how that ordering relates to trust over time.

2 Related Work

  • Trust in automation concerns whether users can rely on and predict system results, and lost human-machine trust can be difficult to restore.People are reported to be more forgiving of humans than machines when trust is violated.Source: §2 · p. 2
  • Trust research compares system accuracy, explanation fidelity, and users’ changing trust, with prior studies finding accuracy can matter more than explainability (Papenm ier et al. 2019).Measures include explicit questionnaires, agreement with wrong outputs, repeated trust ratings, and perceived system accuracy.Source: §2 · p. 2
  • Research links novice users’ over-trust and over-reliance on system predictions to automation bias (Mosier and Skitka 1999; Bussone, Stumpf, and O’Sullivan 2015).Mistrust can also lead users to underestimate a system and rely on themselves instead.Source: §2 · p. 2
  • Prior work studied expertise across domains, but novice and expert labels vary with the domain and task; one study of expertise relative to model knowledge reported inconclusive results.The cited literature includes medical, data science, visual analytics, and aviation settings.Source: §2 · p. 2
  • Earlier first-impression research examined people, systems, and agents, but the ordering of different model outputs received little attention; prior work also left domain expertise and trust changes over time unexamined.Dietvorst et al. (2015) found users may avoid an algorithm after observing an error; the current study extends prior first-impression work by considering expertise and trust changes.Source: §2 · pp. 2–3

3 Experiment

The experiment tests how domain expertise and the order of correct and incorrect system outputs relate to first impressions, trust over time, and perceived accuracy. Participants reviewed an arthropod classifier and reported trust during and after the task.

3 Experiment

  • The study uses a simulated multi-class image-classification task to examine how domain knowledge and the order of observed errors relate to user trust.The experiment section covers study design, goals, and participants.Source: §3 · p. 3

3.1 Research Goals and Hypotheses

  • The hypotheses predict that output order will shape first impressions mainly for experts, that positive first impressions will correspond to higher expert trust, and that experts will adjust trust over time.The proposed contrast is that novices may maintain high trust and be more susceptible to automation bias.Source: §3.1 · p. 3

3.2 Experimental Design

  • The 2x2 between-subjects design varies expertise and output order, with correct outputs first or misclassifications first while accuracy and trials stay constant.Entomology was selected because novices could partly identify errors while experts were expected to do better; participants completed 40 trials at 47.5% accuracy.Source: §3.1 · p. 3, §3.2 · pp. 3–4

3.3 Dataset

  • The dataset contains 40 U.S. arthropod macro images spanning easy- and hard-to-detect examples, with expert-generated labels and saliency-map explanations.The simulated classifier’s 47.5% accuracy came from 19 correct labels and 21 false labels.Source: Figure 1 · p. 3, §3.3 · p. 4

3.4 Participants

  • The study recruited 116 participants, separating experienced users with at least one year of university entomology coursework from novices with little or no familiarity.A familiarity-rating comparison found the experienced group rated its entomology familiarity significantly higher than novices, F(1, 107) = 712.99, p < 0.001.Source: §3.4 · p. 4

3.5 Study Procedure and Measures

  • Participants rated each label and explanation, reported trust after every five trials, then completed a trust questionnaire, estimated system accuracy, and answered free-response questions.The per-trial ratings used a 5-point Likert scale; trust ratings used a 7-point scale.Source: §3.5 · p. 5

4 Results

Across the study, trust depended on domain expertise and the order in which participants saw correct outputs and errors. Expert participants’ trust changed with observed performance, while novices overestimated accuracy and reported higher trust overall.

4.1 Data Pre-processing

  • After exclusions for data errors or inattention, 111 participants remained, with 27–28 in each expertise-by-order condition.Five participants’ data were removed.Source: §4.1 · p. 5

4.2 Average Self-reported Trust

  • For experienced participants, average trust depended on output order: correct-first participants reported significantly higher trust than wrong-first participants.The effect of output order was significant only for knowledgeable participants; domain knowledge and output order also had a significant interaction.Source: §4.2 · p. 5

4.3 Changes in Trust over Time

  • Experienced correct-first participants’ trust declined as system accuracy fell, while wrong-first participants began with lower trust and changed less; novices showed no significant change differences.Figure 4 summarizes first-to-last trust change; Figure 5 plots trust across time steps.Source: Figure 4 · p. 5, §4.3 · p. 6, Figure 5 · p. 6
  • Among experienced participants, 21 of 28 correct-first participants reported trust falling, while 8 of 27 wrong-first participants reported a slight increase.Eleven of 27 wrong-first participants said they did not trust the system at all, and five reported declining trust despite improving accuracy later.Source: §4.3 · p. 6, Table 2 · p. 6

4.4 Post-Study Questionnaire

  • Novices significantly overestimated system accuracy, while experienced participants’ accuracy estimates varied with output order; novices also reported significantly higher questionnaire trust.Figure 6 summarizes perceived-accuracy error and questionnaire trust; Table 3 reports the corresponding results.Source: §4.4 · p. 7, Figure 6 · p. 7, Table 3 · p. 7

5 Discussion

The discussion finds that expertise shapes how first impressions influence trust, then considers design implications and limits on extending the findings across users, tasks, and representations.

5.1 Interpreting the Results

  • Novices over-trusted the low-accuracy system, while experts’ trust and perceptions of accuracy depended on their first impressions.Experts with positive first impressions adjusted trust as they observed performance; novices consistently overestimated accuracy.Source: §5.1 · p. 7
  • Experts with positive first impressions adjusted trust more over time, while those with negative impressions began with lower trust that remained low.The positive-first-impression group showed a significantly greater magnitude of trust change.Source: §5.1 · p. 7
  • Experts with negative first impressions reported they would not use the system in the future, raising concern about continued real-world use (Dietvorst, Simmons, and Massey 2015).The discussion notes that trust is easier to lose than to reestablish.Source: §5.1 · p. 7

5.2 Implications for Intelligent System Designers

  • A system designed for varied expertise risks under- and over-reliance: experts may be skeptical, while novices may miss problems and show automation bias.Suggested measures include introductory sessions on system strengths and shortcomings and examples of both correct and incorrect outputs; the showcasing approach needs further study.Source: §5.2 · pp. 7–8

5.3 Limitations and Future Work Opportunities

  • The novice findings assume little or no domain knowledge; novices able to judge errors correctly might show the expert pattern.The authors note that assigned expertise can vary with the domain and task.Source: §5.3 · p. 8
  • Open-ended free responses limit how specifically the study can distinguish characteristics of trust and mistrust, despite alignment with quantitative results.Self-reported trust ratings also have limits because participants may distinguish Likert-scale values differently.Source: §5.3 · p. 8
  • The study used a task with defined correct predictions; exploratory tasks may have errors that are difficult or impossible to detect or define.The discussion gives missing-data predictions and recommendation suggestions as examples of this boundary.Source: §5.3 · p. 8
  • Findings reflect instance-level observations with simple output representations; generalization to deeper expert analysis of the model as a whole remains for future study.The authors propose examining higher-level representations or visualizations.Source: §5.3 · p. 8

6 Conclusion

The conclusion reports a controlled entomology experiment on expertise, first impressions, and trust calibration. It finds that early errors corresponded to lower trust and reluctance to use the system, while positive impressions accompanied later calibration.

  • Only participants with domain knowledge formed first impressions; early errors lowered trust and willingness to use or rely on the system, while positive impressions accompanied calibration and future use.Source: §6 · p. 8
  • The controlled study had domain-knowledgeable and novice participants review a low-accuracy simulated arthropod-classification system.Source: §6 · p. 8
  • Trust was rated during the study and measured implicitly and explicitly at the end.Source: §6 · p. 8
Loading 2008.09100v1…