Source-linked AI summary
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Sida Peng, Eirini Kalliamvakou, Peter Cihon, Mert Demirer
TL;DR
Limited evidence exists on how AI-powered tools affect software-development productivity. This paper uses a controlled GitHub Copilot experiment and finds that treated developers completed an HTTP-server task 55.8% faster than controls.
Problem
Despite growing interest in AI-powered tools, there has been little research on their productivity impacts in software development.
Method
The paper conducts a controlled trial of GitHub Copilot, asking software developers to implement an HTTP server in JavaScript as quickly as possible.
Results
55.8% faster: developers with access to GitHub Copilot completed the task faster than the control group.
Takeaways & Limitations
The results suggest that Copilot can improve productivity in professional software development.
Takeaways & Limitations
All outliers were in the control group, although the results remained robust when those outliers were excluded.
Abstract
from arXiv · showhide
Generative AI tools hold promise to increase human productivity. This paper presents results from a controlled experiment with GitHub Copilot, an AI pair programmer. Recruited software developers were asked to implement an HTTP server in JavaScript as quickly as possible. The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group. Observed heterogenous effects show promise for AI pair programmers to help people transition into software development careers.
Introduction
The paper examines how AI tools affect software-development productivity, addressing limited evidence on professional productivity impacts. It reports a controlled GitHub Copilot trial in which treated developers completed an HTTP-server task faster, with benefits varying across developers.
- Professional-context evidence on the productivity impacts of AI-powered tools remains limited.
- The paper studies AI-tool productivity effects in software development and presents a controlled trial of GitHub Copilot.
- Developers implemented an HTTP server in JavaScript as quickly as possible, with the treatment group using Copilot and controls otherwise unconstrained.
- Less-experienced, older, and heavier-daily-use programmers benefited more from Copilot.
- These heterogeneous effects suggest promise for AI pair programmers in expanding access to software-development careers.
Study Design
The study randomly assigned recruited professional developers to complete a standardized JavaScript HTTP-server task with or without GitHub Copilot. Performance was measured using task success and completion time, supported by automated repository and test-suite records.
- 95 professional programmers were recruited through Upwork, and participants were randomly split into control and treatment groups.
- Five treated participants began without Copilot because they did not finish signup before the experiment.
- Participants implemented an HTTP server in JavaScript; the treatment group used GitHub Copilot while the control group did not.
- Both groups could use ordinary information sources such as internet search and Stack Overflow, while only treatment participants had Copilot access.
- Performance was measured by task success and task completion time, using a standardized coding task.
- Completion time ran from repository creation to the first commit that successfully passed all 12 tests.
Results
Copilot users completed the standardized task substantially faster than controls, while success-rate differences were not statistically significant. Benefits were heterogeneous, and participants’ perceived productivity gains understated the revealed performance improvement.
- Four completion-time outliers above 300 minutes were all in the control group, but results remained robust after dropping them.
- The treated group’s success rate was 7 percentage points higher than the control group, but the estimate was not statistically significant.The 95% confidence interval was [-0.11, 0.25].
- Less-experienced developers, developers coding more hours daily, and developers aged 25–44 benefited more.
- Both groups estimated a 35% productivity increase, below the 55.8% revealed productivity increase.
- Treated participants reported an average monthly willingness-to-pay price of $27.25 versus $16.91 for controls, a statistically significant difference.
Discussion
The paper provides controlled-experiment evidence that GitHub Copilot can raise productivity on a standardized software-development task, while emphasizing limits on generalization and unresolved effects on code quality and labor markets.
- Results: 55.8% faster task completion was observed for participants with GitHub Copilot than for the control group.The authors characterize the productivity impact as statistically and practically significant.
- Method: The study uses a standardized programming task to obtain a precise productivity measure rather than observing collaboration on large professional projects.The authors note that this design differs from proprietary or open-source development settings.
- Limitations: The productivity benefits may vary across specific tasks and programming languages, so further research is needed to assess generalization.The paper explicitly limits the scope of its findings beyond the studied task and language.
- Limitations: The study does not examine AI's effects on code quality, which may carry performance and security consequences.The authors note that AI assistance could either increase or reduce quality depending on how programmers attend to the code.
- Implications: Less experienced programmers may benefit more from Copilot, with potential implications for initiatives supporting transitions into software development if the result persists.The authors identify heterogeneous effects as warranting close attention and call for further research on the economic impacts of these models.
- Implications: If extrapolated to the population level, the 55.8% productivity increase could imply substantial cost savings and notable GDP effects, although distributional consequences remain unclear.The paper cautions that it is not yet clear how gains would be distributed or how job tasks would change.