Source-linked AI summary
How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering
Rudrajit Choudhuri, Dylan Liu, Igor Steinmacher, Marco Gerosa, Anita Sarma
TL;DR
The paper addresses limited evidence about conversational generative AI for supporting students in software engineering learning. In a between-subjects study of 22 students, it compared ChatGPT with non-generative-AI resources and found no statistical differences in productivity or self-efficacy, but increased frustration.
Problem
Evidence remains limited on how conversational generative AI can support learning advanced software engineering topics involving contextual decisions, subjectivity, and trade-offs.
Source: §1 · p. 1Method
A between-subjects study of 22 students compared ChatGPT assistance on software engineering tasks with traditional resources.
Source: §1 · p. 2Results
The study found no statistical differences in productivity or self-efficacy between ChatGPT and traditional resources, while ChatGPT increased participants’ frustration.
Source: §1 · p. 2, §7 · p. 11Takeaways & Limitations
The authors call for caution in learning use and recommend designing for better user interactions in light of the tool’s pitfalls and guideline violations.
Source: §1 · p. 2, §7 · p. 11Time-boxing may have left participants insufficient time to complete tasks, although it was applied to both groups and does not affect between-group comparisons.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Conversational Generative AI (convo-genAI) is revolutionizing Software Engineering (SE) as engineers and academics embrace this technology in their work. However, there is a gap in understanding the current potential and pitfalls of this technology, specifically in supporting students in SE tasks. In this work, we evaluate through a between-subjects study (N=22) the effectiveness of ChatGPT, a convo-genAI platform, in assisting students in SE tasks. Our study did not find statistical differences in participants' productivity or self-efficacy when using ChatGPT as compared to traditional resources, but we found significantly increased frustration levels. Our study also revealed 5 distinct faults arising from violations of Human-AI interaction guidelines, which led to 7 different (negative) consequences on participants.
1 INTRODUCTION
The paper tests whether ChatGPT helps students with software engineering tasks, where evidence about conversational AI for advanced SE learning is limited. It reports no statistical difference in task performance or self-efficacy, increased frustration, and faults linked to Human-AI interaction guideline violations.
1 INTRODUCTION
- Research on conversational AI had not established how it could support advanced SE learning, whose objectives involve contextual decisions, subjectivity, and trade-offs.Earlier work addressed introductory CS, algorithmic programming problems, or improving AI technology and output.Source: §1 · p. 1
- A between-subjects study compared students using ChatGPT exclusively with students using resources other than generative AI to assess help with SE tasks.The study also examined ChatGPT’s faults, their causes, and their consequences for participants.Source: §1 · pp. 1–2
- The study found no statistical differences in task performance or overall self-efficacy, while ChatGPT use increased participants’ frustration.The evaluation involved three software engineering tasks and 22 students.Source: §1 · p. 2
- Qualitative analysis identified five fault categories and seven consequences; participants commonly reported unclear capabilities, inefficient correction, poor scoping under uncertainty, and limited decision transparency.The analysis linked the guideline violations to the faults.Source: §1 · p. 2
- The authors argue that inconsistent behavior and output can frustrate learners and cause cognitive overload, motivating better interaction design and caution when using these tools for learning.They also urge educators, students, and self-learning professionals to consider the consequences of using these tools to acquire knowledge and skills.Source: §1 · p. 2
2 METHOD
The study used a controlled, between-subjects design to compare ChatGPT with non-generative-AI resources across software engineering tasks. It measured task performance and student perceptions, and used structured reviews and guideline-based analysis to examine AI faults and participant experiences.
2 METHOD
- The controlled, between-subjects experiment compared an Experimental group using ChatGPT (GPT-4) exclusively with a Control group using online resources except generative AI.The team refined the protocol through sandbox sessions before the experiment.Source: §2 · p. 2
2.1 Task Design
- Tasks aligned with course learning objectives and covered API use, debugging, code quality, and version management through functionality fixes, code-smell removal, and repository contributions.Researchers designed moderately challenging tasks for novice SE students and validated their complexity in sandbox sessions.Source: §2.1 · p. 3
2.2 RQs, Metrics and Instruments
- The study used multiple metrics and instruments to assess student perceptions and ChatGPT’s effects, organized around the research questions.Table 1 relates the metrics and instruments to the RQs.Source: §2.2 · p. 3, Table 1 · p. 3
2.2.1 RQ1: How effective is convo-genAI in helping students in soft-
- RQ1 measures included NASA-TLX cognitive-load ratings, task correctness, before-after self-efficacy ratings, and Likert-scale continuance-intention responses.Time to completion was excluded because tasks were time-boxed and both groups used their allotted time.Source: §2.2.1 · pp. 3–4
2.2.2 RQ2: What are the current pitfalls in convo-genAI?.
- For RQ2, researchers used AAR/AI reviews and Microsoft’s Human-AI interaction guidelines to examine ChatGPT faults, their causes, and effects on participants.The adapted AAR/AI process repeated its inner-loop questions after each task; participants also rated adapted guideline statements.Source: §2.2.2 · pp. 4–5
2.3 Sandboxing
- Six sandbox sessions exposed burdensome AAR/AI questions and logistical challenges, prompting clearer mixed-format questions, an in-person setting, and 80-minute sessions with time-boxed tasks.Researchers also omitted two guidelines judged inapplicable to the study context.Source: §2.3 · p. 5
2.4 Lab Study
- The three-week lab study recruited undergraduate students in university SE courses; 22 participants were randomly assigned, completed three tasks, and took part in sessions lasting about 80 minutes.Participants completed consent and questionnaires, while the Experimental group also answered AAR/AI questions before and after each task.Source: §2.4 · p. 5
3 RESULTS
Across three software engineering tasks, ChatGPT users showed significantly higher frustration, but no statistically significant productivity or overall self-efficacy advantage over controls. The analysis identified five fault types linked to Human-AI guideline violations and seven participant consequences.
3.1 RQ1: Effectiveness
- ChatGPT users reported significantly higher frustration than controls (U=101, p=0.008, δ=0.669), while overall productivity and self-efficacy showed no statistically significant advantage.Task-specific productivity varied: ChatGPT users had lower productivity on Task 1 (δ=-0.339) and higher productivity on Task 2 (δ=0.421), without an overall productivity difference.Source: §3.1 · pp. 5–6, Table 4 · p. 6
- Participants reported mixed continuance intentions: 45.5% responded positively and 45.5% negatively, while 9.1% were undecided.Responses covered both plans to use ChatGPT for learning and recommendations to friends.Source: §3.1 · p. 6, Figure 3 · p. 6
3.2.1 Faults made by ChatGPT.
- The five fault categories were limited niche-topic advice, failure to comprehend the problem, incomplete assistance, hallucination, and wrong guidance.Participants associated problem-comprehension failures and incomplete assistance with increased mental load and frustration; hallucinations and wrong guidance required identifying and resolving errors.Source: §3.2.1 · p. 7
- Limited niche-topic advice was sometimes incorrect, and participants linked it to lower Task 1 productivity among ChatGPT users.Participants described difficulty with specialized libraries and ambiguous concepts.Source: §3.1 · p. 6, §3.2.1 · p. 7
3.2.2 Causes of these faults and their consequences.
- Participants reported that all violated Guideline 2 and Guideline 10; 73% reported Guideline 11 violations, and about 54% reported violations of Guidelines 1 and 9.The authors linked capability and limitation mismatches to faults, while unclear uncertainty and decision-making were associated with hallucination and wrong guidance.Source: Figure 4 · p. 7, §3.2.2 · p. 8
- Faults cascaded: limited topic knowledge or misunderstanding could be followed by hallucination and wrong guidance.Seven consequences fell into uncertainty, reflections, and actions; participants doubted correctness, found ChatGPT unhelpful or doubted themselves, and used experience, cherry-picked, or modified responses.Source: §3.2.2 · p. 8
- Participants found ChatGPT unhelpful when it gave limited, incomplete, or wrong advice, and incomplete assistance could prompt self-doubt.Some participants relied on their own experience, while others selected parts of responses or modified them to reach a solution.Source: §3.2.2 · pp. 8–9
4 DISCUSSION: RECOMMENDATION
The discussion argues for designing generative AI as a scaffolding agent rather than a source of ready-made answers. It recommends clearer interaction and pedagogical support, while noting differences in reported guideline violations by gender.
4 DISCUSSION: RECOMMENDATION
- The authors argue that generative AI should be tailored as a scaffolding learning agent, since ready-made answers may diminish critical thinking and affect learning.In this study, self-efficacy decreased in some cases, including understanding Python code.Source: §4 · p. 9
- Future agents should better interpret student intentions and expectations, personalize assistance, and use pedagogical scaffolds to explain problem solving and handle underperforming scenarios.Suggested supports include templates, heuristics, and human intervention.Source: §4 · p. 9
- An iterative participatory design approach is recommended to address violations involving capability and limitation disclosure, correction, uncertainty scoping, and decision-making transparency.Participants’ uncertainty led them to rely on their own experience, cherry-pick, and modify responses, which may have added mental workload and help explain the lack of a positive productivity impact.Source: §4 · p. 9
- Reported guideline violations differed by gender: all women reported a transparency violation, while majorities of men reported violations involving personalization and customization.The authors suggest differing information-processing styles and limited interface tinkering as possible explanations.Source: §4 · p. 9
- Future research should examine how uncertainty relates to cognitive styles and use GenderMag during iterative AI design to address possible inclusivity issues.The discussion notes that tools that do not accommodate cognitive styles may also fail to be inclusive of gender.Source: §4 · pp. 9–10
5 RELATED WORK
Prior work covers educational chatbots, generative AI for programming and instructional materials, and studies of AI’s effects on learners. This paper focuses on conversational generative AI for software engineering students.
5 RELATED WORK
- Educational chatbots have been proposed for out-of-class software engineering support, but are often limited to educator-anticipated queries.Such systems can function like programmed search for frequent questions.Source: §5 · p. 10
- Generative AI research includes programming performance, learning exercises, learner-created materials, interactive course content, and prompt engineering.Reported student-impact findings include improved coding skills in one study, compromised code comprehension in another, and no efficiency or accuracy improvement in a third.Source: §5 · p. 10
- This work complements prior research by examining support for students specifically in software engineering.Source: §5 · p. 10
6 THREATS TO VALIDITY
The study reports threats to construct, internal, and external validity, while describing mitigation efforts and acknowledging residual risks. Its Python-based tasks and small sample constrain generalizability.
6 THREATS TO VALIDITY
- Questions may be misinterpreted and produce incorrect measurements despite literature-based instruments and sandbox refinement.The team refined instruments and protocol in sandbox sessions, but acknowledged this residual construct-validity risk.Source: §6 · p. 10
- Self-selection, exhaustion, distraction, task selection, participant assessment, and desirability bias remain internal-validity threats despite mitigation.Sessions were limited to 80 minutes and tasks were time-boxed, which could leave insufficient completion time; equal time-boxing was not considered to affect between-group comparisons.Source: §6 · p. 10
- Task design drew on instructor insights and course materials and was sandboxed, while fault analysis depended on participants’ ability to assess ChatGPT interactions.Participants averaged 3.75 years of experience with ChatGPT and Python.Source: §6 · p. 10
- Qualitative interpretation may affect validity, so researchers compared analyses with established codes and refined categories through frequent meetings until unanimous agreement.The authors describe these procedures as efforts to ensure consistency.Source: §6 · p. 10
- Python tasks and 22 participants limit generalizability; findings may not extend to other languages, universities, convo-genAI systems, work contexts, or interaction styles.The authors traded breadth for depth in the Python context and recruited across multiple SE classes, distributing participants evenly by demographics.Source: §6 · p. 10
7 CONCLUSION
The study found no statistical differences in productivity or self-efficacy, but ChatGPT increased frustration and was associated with uncertainty and occasional self-doubt. The authors see design opportunities in careful co-design, while warning that novices may struggle with complex or confusing responses.
7 CONCLUSION
- No statistical differences emerged in productivity or self-efficacy, while ChatGPT increased frustration and was associated with uncertainty and occasional self-doubt.The authors attribute the latter experiences to a lack of transparency and clarity in ChatGPT’s behavior and communication.Source: §7 · p. 11
- ChatGPT may give good answers in straightforward cases but tends toward incorrect or confusing responses in complex scenarios, which novices may struggle with or learn from incorrectly.The authors contrast expert developers’ ability to navigate responses with novices’ potential difficulty.Source: §7 · p. 11
- The findings offer design insights, and the authors foresee careful co-design as a way for genAI to support novices learning software engineering.They plan to implement and evaluate genAI-embedded pedagogical tools that foster critical thinking and support learning.Source: §7 · p. 11