Investigating and Designing for Trust in AI-powered Code Generation Tools

1 INTRODUCTION

• As prompt-based AI code-generation tools gain popularity, a key design challenge is helping developers evaluate their trustworthiness for responsible, reliable experiences.These tools can generate multiple lines of code in real time from a prompt within an IDE.Source: §1 · p. 1

2.1 Trust in AI

• Trust in AI is a subjective attitude, distinct from reliance or compliance, that varies with context and experiences and is affected by institutional investment, others’ endorsement, and task risk.Lack of trust can prevent workflow adoption even when system performance is superior; blind trust in high-stakes tasks can lead users to overlook AI mistakes or risks [48] [49].Source: §2.1 · pp. 2–3
• Trust affordances use interface cues to convey system trustworthiness, with design aimed at aligning users’ trust with actual trustworthiness [36] [13] [30].Prior research identifies ability, benevolence, and integrity as trust-shaping system properties [32].Source: §2.1 · p. 3
• Interface approaches include explaining AI outputs, exposing model mechanisms, enabling user control, and delaying outputs, but transparency findings are mixed [58] [37] [34] [14].Confidence scores may mislead users about output quality [4]; user control has been associated with greater trust in co-creation and writing tasks [34], while delayed output can reduce over-trust [14].Source: §2.1 · p. 3

2.2 Generative AI in software engineering: AI-powered code generation tools

• LLM-based generators suggest code from prompts and project context, enable acceptance or editing of snippets, and challenge developers’ mental models and code review [5] [49] [50].Unlike rule-based completion, they can synthesize longer, contextually relevant code; their interaction paradigm is new for developers [44] [64].Source: §2.2 · p. 4
• Generative AI code-generation tools warrant more nuanced trust consideration than traditional developer-support tools because their mechanisms are more opaque and outputs harder to anticipate, requiring caution about potential risks [48].An ethnography of deterministic code-generation tools identified 16 trust factors, which might change for AI-powered tools as system properties and social and organizational usage contexts shift.Source: §2.2 · p. 4
• The authors respond with retrospective interviews about real-world use and a design-probe study exploring trust affordances based on transparency and user control.The study setting involved real consequences from undesirable AI outcomes [s4.p4#404ec0].Source: §2.2 · p. 4

3.1 Methods: Retrospective Interview Study

• Study 1 interviewed 17 developers about real-world AI code-generation use, using critical incidents and retrospective interviews to examine trust judgments.Participants collected notable experiences in advance and discussed them during interviews.Source: §3 · p. 4, §3.1.1 · pp. 4–5

3.2.1 Ability: AI tools’ practical benefits.

• Developers valued AI tools for practical benefits such as saved time and code contributions, especially when suggestions provided a useful foundation they could modify.Trust could erode when verification and correction took more time than the suggestions saved.Source: §3.2.1 · pp. 5–6

3.2.2 Benevolence: alignment of goals between AI and developer.

• Developers trusted tools they perceived as aligned with their goals, but intrusive suggestions and fears of reduced learning or career growth undermined that trust.Their goals included immediate task completion as well as respecting preferences and supporting longer-term learning and career aspirations.Source: §3.2.2 · p. 6

3.2.3 Integrity: the model mechanisms.

• Understanding and agreeing with an AI tool’s mechanisms supported trust, while missing information about privacy, security, and code provenance impeded evaluation.Developers wanted transparency about how the tool worked and what checks ensured code was appropriately copyrighted.Source: §3.2.3 · p. 6

3.2.4 Situational factors: the stake and complexity of tasks.

• Trust varied with task stakes and complexity: developers limited AI to suggestive use in high-impact work and distrusted it for open-ended or specially constrained tasks.Some trusted AI for proof-of-concept projects but not production; concerns also included accessibility, responsiveness, and code explainability.Source: §3.2.4 · p. 6

3.3.1 Biased trust attitudes due to lack of reliable source of information on AI ability.

• Without reliable information about AI performance across contexts, developers relied on personal trial and error, making trust judgments inefficient and vulnerable to bias.Negative experiences could outweigh positive ones, and expectations drawn from conventional autocomplete tools could set an unrealistic bar for generative suggestions.Source: §3.3.1 · p. 7

3.3.2 Ineffective and inefficient evaluation of AI output.

• Inadequate support for assessing suggestion quality left developers relying on time-consuming manual checks and external tools that disrupted their programming workflow.One participant spent half an hour finding an extra bracket in a long suggestion; switching between coding and review was described as mentally draining.Source: §3.3.2 · p. 7

3.3.3 Lack of mechanisms to align AI with developers’ goals and preferences.

• Current interaction paradigms made it difficult for developers to communicate their goals and preferences, including when they wanted suggestions, leading to misaligned or intrusive outputs.Prompting could interrupt programming flow, and developers also wanted ways to express long-term learning goals and control suggestion timing.Source: §3.3.3 · pp. 7–8

4.1 Developing design concepts and visual stimuli.

• The team iteratively developed low-fidelity Copilot-in-Visual-Studio-Code-style visuals from Study 1 challenges, interviews, literature, and feedback to elicit reactions to design concepts.The study explored concepts as trust affordances rather than quantitatively evaluating any specific design.Source: §4.1 · p. 8, §4 · p. 8

4.2 Study procedure, participants, and data analysis.

• Twelve Copilot-experienced developers completed 60-minute design probes; researchers elicited feedback on animated concepts and analyzed recordings for helpfulness, risks, and improvements.The sample included developers with 4 to 45 years of programming experience; sessions used deductive thematic analysis.Source: §4.2 · p. 11

4.3.1 Demonstrating AI’s practical benefits via usage statistics.

• Usage dashboards helped developers align expectations with Copilot’s abilities and calibrate trust, but participants raised concerns about interpreting statistics and workplace surveillance.Participants valued acceptance-rate and time-saved statistics; they also wanted more granular performance breakdowns and actionable tips.Source: §4.3.1 · pp. 11–12

4.3.2 Offering quality indicators to support evaluation of AI suggestions.

• Confidence and familiarity indicators helped developers assess suggestions and target validation, but scores lacked context and could bias acceptance or rejection.The concepts included solution-level confidence, token-level uncertainty highlights, and file-level familiarity signals.Source: §4.1.2 · p. 9, Figure 2 · p. 10, §4.3.2 · p. 12

4.3.3 Communicating developers’ intention with control mechanisms.

• Project controls and context sliders helped developers align suggestions with preferences, improve predictability, and form concrete expectations, but excessive customization could burden users.The controls let developers set project goals and preferences, adjust suggestion scope and validation, and select file or snippet context; some wanted clearer examples and presets.Source: §4.1.2 · p. 10, §4.3.3 · pp. 12–13

5.1 Trust in generative AI tools

• Developers’ trust reflects perceived ability, benevolence, integrity, and situational factors, while weak interface trust affordances leave them relying on experience-based intuitions that can bias trust.The authors argue that GUI remains important for calibrated trust; usability challenges can contribute to overreliance or prevent users from fully benefiting from generative AI.Source: §5.1 · p. 13

5.2.1 Encourage structured reflection on AI tool’s performance and applicability in specific contexts.

• Personalized performance statistics helped developers form accurate expectations, but dashboards may need guidance or actionable suggestions to prompt reflection and support effective tool use.The statistics displayed AI performance across contexts, helping users understand tool utility; interpreting comparisons may require data literacy.Source: §5.2.1 · p. 14

5.2.2 Support evaluation of AI output using context-aware quality indicators.

• Confidence indicators at token, solution, and file levels helped developers evaluate suggestions, but should be customizable and embedded in an explained evaluation framework to guard against overreliance.The authors suggest indicators could explain low-confidence code and include social transparency, such as community acceptability, rather than relying only on potentially misleading numerical scores.Source: §5.2.2 · p. 14

5.2.3 Afford users to convey short- and long-term goals and preferences.

• Controls over suggestion timing, characteristics, and local context can align AI with user goals, but should prioritize meaningful choices, offer simple defaults, and clarify their effects.Persona-based grouping was helpful; onboarding that demonstrates control options and progressively clarifies them may address users’ difficulty understanding those options.Source: §5.2.3 · pp. 14–15

5.3 Limitations and future work

• The authors call for experimental prototype evaluation and broader future studies, noting limits from qualitative methods, a male-skewed sample, one company’s Study 1, and rapidly changing tools.Interviews were collected in July–August 2022; future work should examine women and gender-minority developers and trust across communities and organizational settings as tools evolve.Source: §5.3 · p. 15

• Table 1 describes Study 1 participants, listing years of programming experience and self-reported job titles.Source: Table 1 · p. 20
• Table 2 describes Study 2 participants and identifies years of programming experience as a participant-information column.Source: Table 2 · p. 20

B.1 Example interview questions

• Interviewers asked about participants’ programming projects, tasks, and typical development activities.Source: §B.1 · p. 20
• The guide asked participants about their experience with AI-powered code-generation tools.Source: §B.1 · p. 20
• Participants were asked to describe significant moments, their tasks and tool interactions, and how those interactions affected their trust.Source: §B.1 · p. 20
• The questions explored how participants defined trust in the tool and where they thought that trust came from.Source: §B.1 · p. 21
• Participants considered which tasks they trusted or distrusted the tool to do, including moments when later reflection reversed those judgments.Source: §B.1 · p. 21
• The guide asked how perceptions of trust changed over time and how the tool’s design could be improved to support greater trust.Source: §B.1 · p. 21

• Participants were asked to share 1–3 significant moments each day over the next few days, including moments of appreciation, frustration, or uncertainty about using the tool.They could describe each instance in one or two sentences and include a screenshot or code snippet in the study chat.Source: §B.2 · p. 21
• Researchers planned to send daily morning reminders and asked participants to report when they did not use AI tools that day.Source: §B.2 · p. 21
• An interview would be scheduled after participants completed the preparation phase by collecting several significant moments.Source: §B.2 · p. 21

C.1 Example design probe questions

• Participants were asked to critique the prototypes rather than their own performance, focus on design concepts rather than feasibility or usability, and imagine placeholder code in their daily workflow.They were also invited to think aloud while viewing the prototypes.Source: §C.1 · p. 21
• Before the mockups, interview questions covered participants’ tool experience, definitions of trust, how they trusted AI, expectation and workflow challenges, and desired interaction changes.The final question asked how participants would redesign interaction to better judge when to trust the tool.Source: §C.1 · p. 21
• For each mockup, researchers asked how participants would use it, whether it could help them judge when to trust Copilot, which mockup was most helpful, and what features to add.They presented and briefly explained mockups one by one.Source: §C.1 · p. 22

D CODE BOOK FOR STUDY 1

• Table 3 presents the codebook for the inductive thematic analysis in Study 1.Source: D CODE BOOK FOR STUDY 1 · p. 22

E CODE BOOK FOR STUDY 2

• Table 4 presents the codebook for the thematic analysis in Study 2.Source: E CODE BOOK FOR STUDY 2 · p. 22

F FEATURES OF GITHUB COPILOT AND TABNINE AS OF JULY 2022

• In July 2022, Copilot and Tabnine provided editor-based code completion; Copilot used OpenAI Codex, suggested lines or blocks from comments and preceding code, and supported multiple languages and frameworks.Neither tool then had a chat interface; Copilot Chat and natural-language questions about selected code snippets became available after the study, as did Tabnine chat.Source: F FEATURES OF GITHUB COPILOT AND… · p. 22

G DESIGN CONCEPTS SHOWN IN THE STUDY

• One prototype group focused on control mechanisms.The paper identifies these as Group 1.Source: Figure 5 · p. 26
• A second prototype group presented quality indicators of AI suggestions.The paper identifies these as Group 2.Source: Figure 6 · p. 27
• A third prototype group presented a usage-statistics dashboard.The paper identifies this as Group 3.Source: Figure 7 · p. 28