PaperlayerView exact revision on arXiv

Source-linked AI summary

Take It, Leave It, or Fix It: Measuring Productivity and Trust in Human-AI Collaboration

Crystal Qian, James Wexler

arXiv:2402.18498v2cs.HC

TL;DR

The study asks whether conversational AI improves productivity and trust, given uncertainty about its effects and potential for automation bias. It compares Bard access with traditional resources in a study of software engineers, finding that outcomes depend on context and user expertise rather than AI access being uniformly better.

  • Problem

    It remains unclear whether conversational AI strictly improves productivity, given imperfect accuracy, biases, and possible automation bias or effort substitution by users.

    Source: §1 · p. 1

  • Method

    The authors studied 76 software engineers completing an occupation-specific programming exam with and without Bard, comparing resource access and measuring productivity and trusting behaviors.

    Source: §1 · p. 1

  • Results

    AI’s effects on productivity and trust depended on context and the user, and access to AI was not strictly better than no access.

    Source: §8 · p. 10

  • Takeaways & Limitations

    Conversational AI may augment worker productivity but was not used in a way that completely replaced human effort or traditional resources.

    Source: §8 · p. 10

    The findings may not generalize beyond software engineers at a US-based technology company, whose AI attitudes and machine-learning literacy may differ from laypeople’s.

    Source: §6 · p. 9

Abstract

from arXiv · show

Although recent developments in generative AI have greatly enhanced the capabilities of conversational agents such as Google's Gemini (formerly Bard) or OpenAI's ChatGPT, it's unclear whether the usage of these agents aids users across various contexts. To better understand how access to conversational AI affects productivity and trust, we conducted a mixed-methods, task-based user study, observing 76 software engineers (N=76) as they completed a programming exam with and without access to Bard. Effects on performance, efficiency, satisfaction, and trust vary depending on user expertise, question type (open-ended "solve" vs. definitive "search" questions), and measurement type (demonstrated vs. self-reported). Our findings include evidence of automation complacency, increased reliance on the AI over the course of the task, and increased performance for novices on "solve"-type questions when using the AI. We discuss common behaviors, design recommendations, and impact considerations to improve collaborations with conversational AI.

1 INTRODUCTION

Conversational AI may improve productivity, but its imperfect accuracy and users’ automation biases leave its effects uncertain. This study examines how access to Bard relates to productivity and trust across expertise levels and question types.

1 INTRODUCTION

  • The study observed 76 software engineers completing a programming exam with and without Bard, examining productivity and trusting behaviors.The authors also discuss common behaviors, design implications, and ethical considerations for intelligent systems.Source: §1 · pp. 1–2
  • Participants increasingly relied on conversational AI over the exam, despite mixed measured and perceived productivity outcomes.Users reported greater productivity and efficiency despite spending more time on the task, and reliance increased even as reported trust fell.Source: §1 · p. 1
  • AI’s effects varied by context and expertise, with novices particularly improving on open-ended solve questions.The study compares access to Bard with traditional resources and considers behavioral and self-reported outcomes.Source: §1 · p. 1
  • Experts were more likely than novices to reject AI, which could reduce their performance gains from using it to correct mistakes.Participants also substituted effort to AI and reported reduced cognitive load.Source: §1 · p. 2

2 RELATED WORK

Prior work reports that conversational agents are increasingly used across professional and academic settings, while outcomes and user interactions vary with ability and task context.

2.1 Usage of conversational agents

  • Conversational agents are increasingly adopted in professional and academic settings, including for high-impact decisions, despite accuracy declining on harder tasks.Prior work finds ChatGPT and GPT-4 perform sufficiently well on diverse analytical NLP tasks, with accuracy decreasing as difficulty rises.Source: §2.1 · p. 2

2.2 Variation in interactions by user ability

  • User ability may shape trust and reliance: experts may rationally reject assistance, while novices may overestimate their ability or be less discerning of errors.Prior work also reports higher risk of consuming unreliable agent content among people with lower literacy and education.Source: §2.2 · p. 2

2.3 Variation in interactions by context

  • Task context changes outcomes: ChatGPT improved writing performance for low performers, whereas Copilot did not improve programming success or completion time.Prior studies also report unverified copy-pasting and reduced visual attention to generated code; imperfect assistance can nonetheless improve performance.Source: §2.3 · p. 2

3 EXPERIMENT DESIGN

The study used a moderated, within-participant programming exam to compare access to Bard and traditional resources, while measuring expertise and analyzing participant commentary.

3.1 Procedure

  • 76 engineers completed a moderated, hour-long study with a ten-question exam, screen sharing, think-aloud commentary, and pre- and post-task assessments.Participants were recruited from full-time software engineers at Google; 76 completed the study.Source: §3.1 · p. 2

3.2 Task design

  • The Java readability exam tested style-guide judgments through ten multiple-choice questions, divided into randomized Bard-first and Bard-last sections with two passes per question.Search questions mapped directly to the style guide; solve questions required open-ended critique of code and could not be answered directly from documentation. Participants could add Bard or non-LLM resources on the second pass.Source: §3.2 · pp. 2–3

3.3 Expertise measurement

  • Expertise was measured using both objective and self-reported percentile ranks, with objective rank based on engineering statistics and experience.The study refers to participants with higher objective expertise percentile ranks as experts relative to novices.Source: §3.3 · p. 3

3.4 Thematic analysis

  • The authors thematically analyzed think-aloud transcripts and post-task survey responses, using codes from prior research, pilot studies, and initial participant data.Reported behavior themes were saturated by codes appearing in at least ten user sessions.Source: §3.4 · p. 3

4 RESULTS

The results show that productivity and trust varied with expertise, question type, and whether outcomes were measured through behavior or self-report. Bard could improve novices’ performance on solve questions, while participants spent longer with it and increasingly depended on it over the exam.

4.1 Productivity

  • Measured expertise was positively related to final exam score, and the study operationalized productivity as performance, time spent, and post-task satisfaction.Performance was the total score after the second pass; efficiency was total exam time.Source: §4.1 · p. 3, §4.1.1 · p. 4

4.1.1 Performance.

  • Bard had no significant effect on search-question scores, while Book improved them especially for experts; Bard improved solve-question performance for novices, not experts.Book resources did not provide answers to solve questions, leaving participants more reliant on their expertise.Source: §4.1.1 · p. 4

4.1.2 Efficiency.

  • Participants averaged 39.6 minutes on the exam, yet reported feeling more efficient with Bard despite spending significantly more time using it.Longer Bard use was linked to slower responses, less specific visual direction, and more verbose interactions.Source: §4.1.2 · pp. 4–5

4.1.3 Satisfaction.

  • Participants felt significantly more productive with Bard than Book, with novices agreeing more than experts, but satisfaction and frustration did not significantly change.The survey was administered after the task.Source: §4.1.3 · p. 5, Table 2 · p. 5

4.2 Trust

  • The study assessed trust through both actions and self-reports, treating resource use as reliance and rejection as distrust.A participant could correctly or incorrectly trust a resource, depending on the resulting score.Source: §4.2 · p. 6, §4.2.1 · p. 6, Table 3 · p. 6

4.2.1 Demonstrated measures of trust.

  • All participants, particularly novices, increasingly depended on Bard during the exam despite reporting that they wanted to decrease its use; dependence on Book did not change over time.Experts depended on Book more than novices.Source: §4.2.1 · p. 6
  • Novices were more likely than experts to distrust Book, while experts were more likely to distrust Bard; relying on Book benefited search questions, but experts became less likely to use Bard to correct wrong answers.For solve questions, distrusting either resource had no score effect; participants of all expertise levels were equally likely to incorrectly trust Bard.Source: §4.2.1 · p. 6

4.2.2 Perceived measures of trust.

  • After the task, participants expressed significantly less trust in Bard than before it, a change the authors suggest may reflect trust calibration after exposure and feedback.Pre-task sentiment toward Bard was mostly neutral.Source: §4.2.2 · p. 6

5 DISCUSSION

The discussion explains why Bard did not uniformly improve performance or satisfaction, despite perceived efficiency gains, and describes how expertise and question context shaped resource use. It connects these patterns to design implications for appropriate trust, user-specific support, and source attribution.

5.1 Why might using the AI hurt performance?

  • Adding Bard did not strictly improve scores, and consulting additional resources sometimes changed correct answers to incorrect ones, particularly penalizing experts who were often correct initially.Participants did not seem to recognize that, on this objectively scored task, consulting resources after deriving a correct answer could only lead them astray.Source: §5.1 · p. 7

5.2 Why do participants perceive that they are more productive and efficient with the AI?

  • Participants perceived greater productivity and efficiency despite spending more time with Bard, alongside easier, more passive interactions and effort substitution.Blindly copy-pasting questions and reduced monitoring were discussed as possible automation complacency, given that AI access did not strictly improve productivity.Source: §5.2 · pp. 7–8

5.3 Why is there no change in satisfaction or frustration?

  • Despite perceived productivity gains, participants were not more satisfied or less frustrated; the discussion points to inappropriate trust and defensive blame after Bard-related mistakes.Participants were more likely to blame Bard or their own interpretation and prompting when they missed a question using Bard than when using Book resources.Source: §5.3 · p. 8

5.4 How do participants use resources?

  • Participants most often copied questions directly into Bard, while Book use typically involved searching documentation or other resources for keywords from the question.Participants sometimes added context to prompts, whereas they generally did not paste full solve-type questions into search engines.Source: §5.4 · p. 8

5.5 How do participants pick what to use?

  • Resource choice correlated with question type and expertise; novices often consulted Bard first, and AI suggestions helped them identify where to look in documentation.Participants also assumed Bard might not know company-internal conventions, and none checked whether the style guide was publicly available or whether Bard knew it.Source: §5.5 · pp. 8–9

5.6 Design implications for more effective conversational AI

  • The optimal resource strategy depends on task context and user expertise, while ex-post feedback can make that strategy difficult to determine.Although Bard could outperform the average user when questions and answers were directly copied into it, participants did not use that strategy, perhaps because it required more effort.Source: §5.6 · p. 9

5.6.1 Design for appropriate trust.

  • The authors recommend communicating uncertainty, avoiding designs that create a conversational partner, and considering output customization by user expertise.They note that uncertainty cues may reduce undue overreliance but can also increase cognitive effort and decrease satisfaction.Source: §5.6.1 · p. 9

5.6.2 Cite sources.

  • Participants valued curated, deliberately written resources and were skeptical of Bard’s generated output; source attribution and appropriate citations could help bridge that perceived disconnect.Participants also sought answers with sources, and the authors suggest that citations in conversational output could support more appropriate trust.Source: §5.6.2 · p. 9

6 LIMITATIONS AND FUTURE WORK

The study’s findings are bounded by its setting, agent, and task design. The authors call for replication and longer-term study to assess how well the results generalize.

6 LIMITATIONS AND FUTURE WORK

  • The sample comprised software engineers at a US-based technology company, whose AI attitudes and machine-learning literacy may differ from laypeople; restrictions on company data may also limit familiarity with conversational AI.The authors note these differences may affect performance and behavior.Source: §6 · p. 9
  • The results may be idiosyncratic to Bard, so replication with other agents or a standardized behavioral task suite could test generalizability.Other AI studies have reported similar productivity and trust findings [2].Source: §6 · p. 9
  • Participants used the system for less than an hour, and moderation and flat compensation may have shaped behavior; the authors suggest longitudinal study, follow-up, and higher-stakes or piecewise incentives.Longer exposure and feedback may be needed to study trust formation.Source: §6 · p. 9

7 IMPACT CONSIDERATIONS

The authors frame conversational AI as potentially productivity-enhancing, but its effects vary by user and can include risks from overreliance and incorrect outputs. They recommend designing for calibrated trust and communicating confidence.

7 IMPACT CONSIDERATIONS

  • Appropriately calibrated trust and use in applicable settings could allow conversational AI to increase productivity, but adoption is not always beneficial.The authors note that access can be detrimental in specific contexts.Source: §7 · pp. 9–10
  • Experts tended to distrust automated assistance, while novices were more likely to adopt it and could be more susceptible to influence; novice adoption may have potential to equalize productivity.All participants showed automation complacency, according to the authors.Source: §7 · pp. 9–10
  • The authors recommend designing for appropriate trust, displaying confidence, caution with conversational-partner framing, user customization, and source citation.Source: §7 · p. 10
  • Participants across expertise levels could change correct answers to incorrect ones after AI exposure, underscoring risks when conversational AI informs consequential decisions.The authors contrast the limited impact of errors on a company coding exam with possible use in medical and public-health decisions.Source: §7 · p. 10

8 CONCLUSION

The study finds that conversational AI’s effects on productivity and trust depend on context and user, so access is not uniformly better than no access. Its mixed-method results reveal divergence between perceived and measured outcomes and motivate attention to human behavior and misleading information.

8 CONCLUSION

  • In a study of 76 software engineers completing an occupation-specific exam with and without conversational AI, effects on productivity and trust depended on context and user, and AI access was not strictly better.Source: §8 · p. 10
  • Participants perceived efficiency gains despite taking more time with AI, and reported less trust while increasingly depending on its outputs.The authors use these divergences to motivate mixed-method approaches to interpreting behavior.Source: §8 · p. 10
  • The authors argue system design should account for confirmation bias and automation complacency and minimize potentially misleading information, given users’ increasing dependence, particularly novices’.Source: §8 · p. 10
  • The paper contributes empirical evidence from a real-world human-AI interaction scenario and calls for further work on more productive and trustworthy conversational AI systems.Source: §8 · p. 10

A.1 Pre-Task Survey

The pre-task survey asked about participants’ LLM-tool use and support, programming-language familiarity, knowledge resources, and self-described engineering experience and productivity.

A.1 Pre-Task Survey

  • Participants rated developer satisfaction, perceived work productivity, focus or flow, code quality, and engineering velocity.Source: §A.1 · p. 13
  • The survey asked which programming languages participants had used most in the prior three months, their language familiarity, and how well different knowledge resources supported development.Listed resources included documentation, chat-based LLMs, forums, search tools, and discussion spaces.Source: §A.1 · p. 13
  • Participants reported how often they used LLM-assisted tools at work and how well those tools supported development and other tasks.Source: §A.1 · p. 13

A.2 Post-Task Survey

The post-task survey collected free-form accounts of participants’ approaches to Bard and non-LLM resources, including how they verified answers. Sample statistics draw on company data and pre-task self-reports.

A.2 Post-Task Survey

  • Participants were asked how they used Bard and non-LLM resources to answer and verify questions, which non-LLM resources helped, and for any other study comments.Source: §A.2 · p. 13
  • Figure 6 reports categorical statistics for the 76 participants.Source: Figure 6 · p. 14

B DESCRIPTIVE STATISTICS

C EXPERTISE PERCENTILE RANKS

The study constructed objective and self-reported expertise measures and converted them to percentile ranks; objective expertise was the primary measure, while self-reported expertise had low predictive power for outcomes of interest.

C EXPERTISE PERCENTILE RANKS

  • The objective expertise measure weighted Java readability certification and status first, normalized Java changelists second, and most recent Java submission third, using other-language measures as tiebreakers.Source: C EXPERTISE PERCENTILE RANKS · p. 14
  • Self-reported expertise weighted Java experience, engineering productivity, and LLM expertise, and both expertise measures were transformed into percentile ranks to reduce outlier influence.Source: C EXPERTISE PERCENTILE RANKS · p. 14
  • The objective expertise statistics are summarized in Table 7; readability status distinguishes people who took the exam from those who withdrew after failing to take it.Source: C EXPERTISE PERCENTILE RANKS · p. 14, Table 7 · p. 15
  • Figure 7 presents pre- and post-task survey response distributions; its caption identifies Cronbach’s alpha as a reliability measure and marks significant distribution differences from a chi-square independence test.Source: Figure 7 · p. 15
  • Results were largely robust across individual and aggregated objective expertise measures, whereas self-reported expertise had low predictive power for outcomes of interest.Each participant’s expertise position is relative to others in the primary analysis sample.Source: C EXPERTISE PERCENTILE RANKS · p. 15

D TRUST IN AUTOMATED SYSTEMS SURVEY

The Trust in Automated Systems survey was administered before and after the task using five-point bipolar Likert responses; comparative satisfaction responses were mapped to numeric values for analysis.

D TRUST IN AUTOMATED SYSTEMS SURVEY

Loading 2402.18498v2…