Source-linked AI summary
The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Programming Tasks
Md Istiak Hossain Shihab, Christopher Hundhausen, Ahsun Tariq, Summit Haque, Yunhan Qiao, Brian Mulanda
TL;DR
The effects of GenAI on students adding code to unfamiliar codebases remain little understood, although these tasks require understanding and modifying software written by others. This study compares Copilot and no-Copilot brownfield work, reporting a 34.9% reduction in task completion time with GenAI assistance.
Problem
How GenAI affects students working on brownfield tasks remains little understood, despite the challenge of understanding and modifying unfamiliar code written by others.
Source: §1 · pp. 1–2Method
A within-subjects study compared students’ brownfield programming with and without Copilot, combining performance measures, second-by-second behavioral analysis, and exit interviews.
Source: §1 · p. 2, §3.1 · p. 4Results
34.9% reduction in task completion time was reported with GenAI assistance in brownfield programming tasks.
Source: §5.1 · p. 10Takeaways & Limitations
The findings point to teaching strategies that help students evaluate and integrate GenAI suggestions while developing brownfield programming skills.
Source: §1 · p. 2The study included 10 students from one university, so results require extreme caution when generalized beyond that population.
Source: §6.2 · p. 12
Abstract
from arXiv · showhide
When graduates of computing degree programs enter the software industry, they will most likely join teams working on legacy code bases developed by people other than themselves. In these so-called brownfield software development settings, generative artificial intelligence (GenAI) coding assistants like GitHub Copilot are rapidly transforming software development practices, yet the impact of GenAI on student programmers performing brownfield development tasks remains underexplored. This paper investigates how GitHub Copilot influences undergraduate students' programming performance, behaviors, and understanding when completing brownfield programming tasks in which they add new code to an unfamiliar code base. We conducted a controlled experiment in which 10 undergraduate computer science students completed highly similar brownfield development tasks with and without Copilot in a legacy web application. Using a mixed-methods approach combining performance analysis, behavioral analysis, and exit interviews, we found that students completed tasks 35% faster (p < 0.05) and made 50% more solution progress p (< 0.05) when using Copilot. Moreover, our analysis revealed that, when using Copilot, students spent 11% less time manually writing code (p < 0.05), and 12% less time conducting web searches (p < 0.05), providing evidence of a fundamental shift in how they engaged in programming. In exit interviews, students reported concerns about not understanding how or why Copilot suggestions work. This research suggests the need for computing educators to develop new pedagogical approaches that leverage GenAI assistants' benefits while fostering reflection on how and why GenAI suggestions address brownfield programming tasks. Complete study results and analysis are presented at https://ghcopilot-icer.github.io/.
1 Introduction
The influence of GenAI on students adding code to unfamiliar codebases remains little understood, despite the difficulty of understanding and modifying software written by others. The study asks how Copilot affects students’ effectiveness, programming processes, and usage patterns in brownfield tasks.
1 Introduction
- Students’ use of GenAI in brownfield programming remains underexplored, although these tasks require modifying unfamiliar code written by others.Unlike greenfield assignments that start from scratch, brownfield work requires integrating new code into existing codebases that can be difficult to understand and modify.Source: §1 · p. 1
- Students must evaluate and integrate suggestions that may not align with an existing codebase’s coding patterns and practices.Understanding those decisions is described as crucial to developing teaching strategies for brownfield skills needed in software jobs.Source: §1 · p. 2
- The study asks how Copilot influences student effectiveness, performance, programming processes, and usage patterns in brownfield tasks.The authors conducted a within-subjects experiment with No Copilot and Copilot conditions and report differences in efficiency, process, adoption strategies, and understanding concerns.Source: §1 · p. 2
2 Related Work
Prior research describes GenAI’s integration into software development and examines productivity, adoption, and educational effects. This study extends work on student interaction patterns to brownfield tasks and detailed behavioral analysis.
2.1 Large Language Models in Software Development
- GenAI tools integrate conversational assistance, code completion, and code explanations into development environments, supporting collaborative human–AI programming [40].GitHub Copilot, ChatGPT, and Gemini [44] are cited as examples of tools integrated into contemporary IDEs.Source: §2.1 · p. 2
2.2 GenAI Coding Assistants: Performance and Adoption
- Studies report productivity gains with Copilot, while other work describes delegated tasks, usage patterns, and limitations including weak requirements alignment and difficult control [26] [28].Scientists also use LLMs to navigate unfamiliar languages and libraries [27], and developers report project-context and policy barriers to adoption.Source: §2.2 · p. 2
2.3 GenAI Coding Assistants in Computer Science Education
- Education research reports varied novice use and concerns about retention, topic understanding, and superficial comprehension, alongside perceived utility and course redesign [34] [42].Other studies report that 91% of participants felt Gemini assistance exceeded educational support expectations, while some novices may benefit early in their learning.Source: §2.3 · p. 3
2.4 Student Interaction Patterns with GenAI Coding Tools
- In greenfield assignments, GenAI benefits varied with students’ metacognitive skills; strong skills were associated with enhanced performance, while difficulties were associated with negative effects.Prior work also describes shepherding and straying with Copilot, one-shot prompting, and links between student characteristics and usage patterns.Source: §2.4 · p. 3
- This study extends prior work by examining brownfield rather than greenfield tasks and analyzing student programming behavior second by second.Brownfield tasks require adding code to unfamiliar codebases written by others.Source: §2.4 · p. 3
3 Method
The study compared students’ brownfield programming with and without Copilot using a within-subjects design. It measured task outcomes, coded behavior second by second, and gathered interview accounts.
3 Method
- Participants completed No Copilot tasks first, then highly similar Copilot tasks; task order was counterbalanced, but treatment order was not.The authors chose the sequence to observe initial programming without GenAI assistance.Source: §3.1 · p. 3
- The experiment used a within-subjects, mixed-methods design with No Copilot as control and Copilot as the experimental condition.Participants completed front-end web development tasks in both conditions.Source: §3.1 · pp. 3–4
- Performance was measured by task completion time and tests passed, while programming behaviors were categorized second by second.Exit interviews elicited participants’ perceptions of their programming processes.Source: §3.1 · pp. 3–4
3.2 Pilot Testing
A 13-participant pilot refined the study materials and procedures, including task equivalence, automated testing, and interview and timing protocols.
3.2 Pilot Testing
- The 13 pilot participants came from the main study’s population but were excluded from the final sample.Pilot testing was conducted before the main study to refine materials, tasks, and procedures.Source: §3.2 · p. 4
- The researchers replaced search-bar and sorting tasks with Add Distance and Add Picture after finding the original tasks insufficiently similar in complexity and scope.The replacement tasks were intended to offer better task equivalence while retaining authentic brownfield challenges.Source: §3.2 · p. 4
- Pilot work also strengthened automated test suites and the experimenter protocol, streamlined interview questions, and optimized study timing.Timing was adjusted to balance thorough data collection with participant engagement.Source: §3.2 · p. 4
3.3 Participants
The study recruited 10 undergraduate computer science students with prior web-development experience and little or no experience with GenAI programming assistants. Participants were screened for relevant coursework, project experience, and year of study.
3.3 Participants
- Recruitment used upper-division course postings at Oregon State University, and participants received a $50 prepaid card.The university had over 4,000 undergraduate computer science students enrolled in its programs.Source: §3.3 · p. 4
- Eligibility required third- or fourth-year standing, an A or A- in a front-end web-development course, and a substantial outside project.Applicants also needed little or no prior experience using a GenAI programming assistant.Source: §3.3 · p. 4
- 10 students participated: five traditional and five post-baccalaureate computer science majors, with eight having no prior GitHub Copilot experience.The other two had used Copilot only once or a few times; participants were nine male and one female, with a mean age of 26.7.Source: §3.3 · p. 4
3.4 Materials and Tasks
Students implemented two highly similar features in an unfamiliar legacy web application, with task instructions, automated tests, a Copilot tutorial, and a standardized computing environment. The materials supported comparison of feature work with and without Copilot.
3.4.1 Legacy Code Base.
- The 3,818-line legacy application used vanilla JavaScript, HTML, and CSS, with localStorage rather than a back-end database.It supported login, user settings, and user-data operations.Source: §3.4.1 · p. 4
3.4.2 Feature Implementations.
- The two feature implementations were designed for similar cognitive demands, technical complexity, and scope, with comparable implementation-property counts.The compared properties included lines of code, statements, variables, control structures, and operations.Source: §3.4.2 · p. 4
- Both features used three parallel tasks: UI construction, interactive behavior, and data persistence; one added distance entry and unit conversion, the other profile-picture selection.The Add Picture feature included default and custom image options.Source: §3.4.2 · p. 4
3.4.3 Test Suites.
- Automated suites tested four cases for each feature’s first two tasks and five for its third, checking UI rendering, behavior, and persistence.Passing the tests provided consistent verification criteria for task solutions.Source: §3.4.3 · p. 4
3.4.4 Participant Instructions.
- Participants received instructions about the study, its two-hour allocation, and think-aloud protocol, and were told Copilot—not the participants—was being tested.They were also given an overview of the session.Source: §3.4.4 · p. 4
3.4.5 Tutorial.
- The tutorial introduced the application, code base, and Copilot through app exploration and an About-dialog warm-up task.Participants practiced code completion and chat, but received no further usage advice; a newer direct-insertion feature was not taught.Source: §3.4.5 · pp. 4–5
3.4.6 Task Description Documents.
- Task documents specified screenshots, project files, localStorage use, feature requirements, web-search permission, and when to run tests and finish.Copilot was disabled for the No Copilot treatment and available for the Copilot treatment.Source: §3.4.6 · p. 5
3.4.7 Experimental Computing Environment.
- Sessions used a Linux virtual machine with Visual Studio Code and Copilot, Chrome, Node.js, Git, and participant-specific source repositories.The environment also included study instructions, tasks, and tutorial documents.Source: §3.4.7 · pp. 5–6
3.5 Procedure
Each student completed both feature tasks, first without Copilot and then with it, while feature order was counterbalanced. Sessions included a tutorial and timed implementation periods with testing.
3.5 Procedure
- All participants completed No Copilot before Copilot, while the order of the two feature implementations was counterbalanced.The sequence was chosen to prevent Copilot from influencing initial task performance and to control task-order and task-treatment effects.Source: §3.5 · p. 6
- The 120-minute Zoom session included consent, demographics, environment setup, a 15-minute tutorial, and work on two feature implementations.Participants first completed a screening questionnaire about web-development and GenAI-assistant experience.Source: §3.5 · p. 6
- For each task, participants read instructions aloud, implemented code, could run automated tests, and committed their completed work.They had up to 30 minutes per feature, excluding instruction reading; timing paused between tasks and after successful test completion.Source: §3.5 · p. 6
3.6 Data Collection and Analysis
The researchers combined performance measures, coded programming behaviors, and interview transcripts, cross-checking test progress against committed solutions. They developed and reliability-checked a fine-grained scheme for analyzing activity over time.
3.6 Data Collection and Analysis
- The mixed-methods analysis used task time, passed test cases, coded behaviors, exit interviews, and committed solutions checked against automated tests.Video recordings provided quantitative and qualitative data about participants’ coding outcomes, processes, and perceptions.Source: §3.6 · p. 6
3.6.1 Programming Process Coding Scheme.
- An inductively developed coding scheme distinguished high-level activities from specific behaviors such as manual edits and accepted or pasted Copilot code.Three analysts independently coded 15% of the video data; Krippendorff’s alpha was 0.91 for primary categories and 0.85 for secondary categories.Source: §3.6.1 · p. 6
4 Results
Copilot improved brownfield task performance, while students also shifted how they wrote code, sought information, and integrated testing into their workflows. These gains coexisted with concerns about understanding and integrating Copilot-generated code.
4.1.1 Task Efficiency.
- 34.9% less time was needed to complete Task 1 with Copilot, a statistically significant difference (p = 0.04).Participants averaged 837.0 seconds with Copilot and 1285.9 seconds without it.Source: §4.1.1 · pp. 6–7
4.1.2 Solution Completeness and Correctness.
- With Copilot, participants passed 6.30 tests on average versus 4.20 without it, indicating more feature progress and correct partial implementations.The difference was statistically significant (p = 0.01).Source: §4.1.2 · p. 7
4.2.1 Programming Activity Distribution.
- Programming activity distributions differed significantly between conditions, with less time spent writing code and viewing web content and more time writing prompts and viewing Copilot responses.The overall distribution difference was significant (p = 0.004); the four named activity differences remained significant after correction.Source: §4.2.1 · pp. 7–8, Table 4 · p. 8
4.2.2 Code Writing Behavior.
- Manual entry fell from 95.6% of coding time without Copilot to 52.5% with it, while students also pasted, inserted, accepted, and modified Copilot code.With Copilot, participants spent 21.3% of coding time pasting chat responses, 9.3% inserting code, 6.6% accepting suggestions, and 3.7% modifying suggestions.Source: §4.2.2 · p. 8
4.2.3 Workflow Transitions.
- Copilot users developed a frequent prompt→read response→implement cycle, unlike the read→understand→implement workflow without Copilot; testing was more integrated across activities.The transition matrices differed significantly (p < 0.0001).Source: §4.2.3 · p. 8
4.3 Copilot Usage Strategies (RQ3)
- Lower performers pasted Copilot chat code more often, while higher performers tended toward more selective use; students also raised concerns about understanding code and its fit.The selective-use trends were not statistically significant and require confirmation with larger samples.Source: §4.3 · p. 9
5 Discussion and Implications
Copilot improved students’ brownfield-task performance and reshaped their programming workflows. More successful students integrated generated code selectively, motivating educational approaches that balance assistance with active reflection.
5.1 RQ1: Copilot’s Impact on Programming Performance
- 34.9% reduction in task completion time and 26.2% improvement in test pass rates were reported for students using Copilot on brownfield tasks.The study extends prior productivity findings to undergraduate students working in legacy code bases.Source: §5.1 · p. 9
- The discussion links reduced extraneous cognitive load with a risk that unreflective acceptance of suggestions may reduce learning opportunities.The authors distinguish syntax and boilerplate effort from the cognitive work of learning and problem-solving.Source: §5.1 · p. 10
5.2 RQ2: Copilot’s Impact on Programming Processes
- Copilot use coincided with 10.63% less manual code writing and 11.6% less web-search time, alongside a mix of coding methods.The authors describe Copilot as both a code generator and an information resource.Source: §5.2 · p. 10, Figure 3 · p. 10, Figure 2 · p. 10
- Workflow shifted from read → understand → implement without Copilot to prompt → read response → implement with it.The authors relate this shift to “interactive steering,” in which programmers guide and evaluate generated solutions.Source: §5.2 · pp. 10–11, Figure 4 · p. 11
5.3 RQ3: Differences in how Copilot is Appropriated
- Top performers pasted 83.9% less Copilot code and favored granular inline suggestions over whole code blocks.The discussion connects this selective integration with metacognitive skills and critical evaluation.Source: §5.3 · p. 11, Figure 5 · p. 12
5.4 Implications for Computing Education
- Educators are urged to teach prompt engineering, comprehension, and critical evaluation while keeping students actively engaged with suggestions.The proposed learning outcomes particularly address brownfield integration, where codebase understanding and consistency with existing conventions matter.Source: §5.4 · p. 11
- The authors recommend scaffolding reflection on suggestion relevance and correctness to balance GenAI assistance with learning.They advocate nuanced incorporation of GenAI rather than simply restricting or embracing coding assistants.Source: §5.4 · pp. 11–12
6 Threats to Validity
The study identifies threats involving inconsistent Copilot versions, unmitigated treatment-order learning, and video-based behavioral coding. Its small, selective single-university sample and narrow technical scope constrain generalizability.
6.1 Internal Threats to Validity
- One of 10 participants used Copilot Edits for 40% of code-writing time, so version differences could have influenced that participant’s outcomes and process.The extension auto-updated during data collection, and only one participant had access to Copilot Edits.Source: §6.1 · p. 12
- The study did not counterbalance treatment order, leaving the No Copilot learning effect unmitigated.The researchers chose to observe initial programming without GenAI first.Source: §6.1 · p. 12
- Behavioral coding inferred attention shifts from cursor movements between windows, although eye tracking would have provided a more reliable indication.A moving cursor does not necessarily mean participants’ attention shifted.Source: §6.1 · p. 12
6.2 External Threats to Validity
- The sample comprised 10 students from one university, requiring extreme caution when generalizing beyond that population.The researchers cited time-consuming video coding as the reason for limiting the sample.Source: §6.2 · p. 12
- Findings may not generalize beyond web programming, GitHub Copilot, and the limited development tasks studied.Participants had high academic performance in web development coursework, so findings may not represent students with other achievement or aptitude levels.Source: §6.2 · p. 12
7 Summary and Future Work
The study’s takeaways combine productivity gains with changed programming processes and concerns about understanding generated code. The authors call for research into longer-term effects, broader samples, improved behavioral methods, and comparisons across GenAI tools.
7 Summary and Future Work
- In brownfield tasks, GenAI use was associated with significant speed and productivity gains and a shift from read–understand–implement to prompt–view response–implement.The workflow shift describes a change in students’ programming processes.Source: §7 · p. 13
- Higher-performing students tended to use GenAI more selectively, while some students reported using generated code without understanding how it worked or fit the legacy code base.The more selective students spent higher percentages of implementation time integrating autocomplete suggestions and making manual edits.Source: §7 · p. 13
- Future research should examine prolonged Copilot use, recruit larger and more diverse samples, improve behavioral methods, and compare GenAI tools.Suggested methods include eye tracking and think-aloud protocols probing students’ decisions to adopt or reject AI suggestions.Source: §7 · p. 13