Source-linked AI summary
Productivity Assessment of Neural Code Completion
Albert Ziegler, Eirini Kalliamvakou, Shawn Simister, Ganesh Sittampalam, Alice Li, Andrew Rice, Devon Rifkin, Edward Aftandilian
TL;DR
Commercial neural code-completion tools seek to improve productivity, but their productivity impact is difficult to measure directly. The paper links GitHub Copilot IDE usage data with developer survey responses and finds that acceptance rate correlates better with perceived productivity than persistence measures.
Problem
Productivity benefits of neural code-completion tools are difficult to measure directly, while offline and activity-based measures incompletely capture developer experience.
Method
The study matches GitHub Copilot usage measurements with 2,631 developers’ survey responses and compares acceptance and persistence metrics against perceived productivity.
Results
Acceptance rate was better correlated with reported productivity than measures of completion persistence.
Takeaways & Limitations
Acceptance rate can support coarse-grained monitoring, but fine-grained investigation still requires approaches that account for human factors.
Takeaways & Limitations
The analysis establishes correlation rather than causation, and its best measurement, acceptance rate, has a Pearson coefficient of 0.24 with considerable unexplained variance.
Abstract
from arXiv · showhide
Neural code synthesis has reached a point where snippet generation is accurate enough to be considered for integration into human software development workflows. Commercial products aim to increase programmers' productivity, without being able to measure it directly. In this case study, we asked users of GitHub Copilot about its impact on their productivity, and sought to find a reflection of their perception in directly measurable user data. We find that the rate with which shown suggestions are accepted, rather than more specific metrics regarding the persistence of completions in the code over time, drives developers' perception of productivity.
1 Introduction
The paper examines whether GitHub Copilot usage measurements reflect developers’ perceived productivity. It compares simple acceptance measures with more detailed persistence measures.
- Offline evaluation is difficult because longer code completions can have many acceptable alternatives without straightforward automatic labels.
- Perceived productivity is emphasized because standardized completion times capture only some benefits of developer-assistance tools.
- The study matches 2,631 GitHub Copilot survey responses with IDE usage measurements, including acceptance and subsequent persistence of contributed code.
- Acceptance rate of shown suggestions predicts perceived productivity better than measures tracking whether accepted completions persist unchanged.
- Acceptance rate varies significantly across developers and over time, motivating further analysis of these variations.
- Acceptance rate supports coarse-grained monitoring, but fine-grained investigation still requires other approaches because human factors are numerous.
2 Background
Background work highlights limitations of offline and activity-based productivity evaluation. It motivates acceptance rate while noting that its interpretation depends on context and may not capture the full value of code suggestions.
- Offline benchmarks can differ significantly from real-world code-completion usage, even when completion correctness is measurable.
- Researchers also use software testing and online evaluation because multi-line completions admit diverse possible solutions.
- Acceptance rate is defined as shown completions subsequently accepted into the source file; GitHub Copilot records 27% in this study.
- Acceptance-rate interpretation is difficult because developer objectives, programming language, and time of day affect the data.
- Developers may tolerate or value erroneous suggestions, indicating that correctness alone may not capture suggestion value.
- The SPACE framework motivates measuring perceived productivity through self-reported data across multiple dimensions alongside automatically measured data.
- The authors identify this study as the first, to their knowledge, to link code-suggestion usage measurements with developer productivity or happiness.
3 Data and Methodology
The study combines GitHub Copilot IDE telemetry with an online survey of everyday users. It operationalizes usage through completion events and productivity through multiple self-reported dimensions plus an aggregate score.
- 3.1 Usage Measurements: GitHub Copilot requests completions from a cloud-hosted Codex model using prompts drawn from code currently in the IDE.
- 3.1 Usage Measurements: Usage measurements count developer completion events collected for GitHub Copilot users.
- 3.1 Usage Measurements: Persistence measures extend acceptance by tracking whether accepted completions are later deleted, corrected, or remain mostly unchanged.
- 3.1 Usage Measurements: Completion-related events form a funnel summarized quantitatively in Table 1.
- 3.1 Usage Measurements: Normalized metrics use X_per_Y notation, such as accepted_per_hour for accepted events divided by active coding hours.
- 3.2 Productivity Survey: The survey targeted 17,420 unpaid technical-preview users and matched 2,047 responses to usage measurements from the preceding four weeks.
- 3.2 Productivity Survey: Survey questions covered demographics and randomized Likert-style productivity statements, with respondents including a significant proportion of professional programmers.
- 3.2 Productivity Survey: Productivity was measured across four SPACE dimensions—Satisfaction, Performance, Communication and Collaboration, and Efficiency and Flow—using 12 statements.
4 What Drives Perceived Productivity?
Across correlation, regression, and latent-structure analyses, acceptance rate most positively predicts perceived productivity. Persistence-based measures contribute less, although combining acceptance rate with additional metrics can provide a fuller picture.
- Acceptance rate most positively predicts users’ perceived productivity across the individual-correlation, PLS, and incremental analyses.The authors identify acceptance rate as the most immediate indicator while noting unexplained variance.
- 0.24: acceptance rate correlates best with aggregate productivity (𝜌= 0.24, 𝑃< 0.0001).It is also the best-performing measure for at least one survey question in each SPACE dimension.
- Persistence measures are less well correlated with perceived productivity than acceptance rate.Shorter persistence windows generally perform better because they remain closer to acceptance rate.
- 43.2% of variance is explained by the first latent component, while the acceptance-rate/change-rate dichotomy explains a further 13.1%.The latent-structure model captures common variation among the usage metrics connected to aggregate productivity.
- Additional frequency, rate, and accepted-contribution metrics can add statistically significant information beyond an acceptance-rate model.No other additions were statistically significant in further iterations.
5 What Drives Acceptance Rate?
Acceptance rate varies with programming language and time. It is higher for some languages and outside typical working hours, with users’ usual activity regime explaining more than the actual time bucket.
- 5.1 Language Use: JavaScript and Python have higher acceptance rates, while survey participants show slightly higher acceptance than the overall user base regardless of language.TypeScript, JavaScript, and Python account for 24.7%, 21.3%, and 14.1% of all shown completions, respectively.
- 5.1 Language Use: Language explains 17% of the variance in perceived productivity predicted from acceptance rate, while acceptance rates explain 38% of languages’ average perceived-productivity differences.
- 5.2 Circadian and Weekly Rhythms: 23.5%: average weekend acceptance rate; 23%: typical non-working hours; 21.2%: typical working hours.The three time regimes are statistically significantly distinct at p < 0.001%.
- 5.2 Circadian and Weekly Rhythms: The boundary between non-working and working-hour regimes is fuzzy, probably partly because of time-zone variation within the United States.
- 5.2 Circadian and Weekly Rhythms: Users’ usual activity regime matters more than the actual time bucket for predicting acceptance rate.Users generally accept fewer suggestions when coding outside their usual activity range.
6 Threats To Validity
The study’s conclusions are limited by its correlational design, substantial unexplained variance, the distinction between perceived and actual productivity, and focus on one completion system.
- The study investigates correlation rather than causation, so its metrics indicate an underlying quality that predicts productivity rather than directly increasing it.
- 0.24: the best-performing acceptance_per_shown measurement leaves considerable unexplained variance.The study sought a signal between perceived productivity and usage metrics, not accurate prediction of individual survey answers.
- Perceived productivity is not necessarily actual productivity, and maximizing acceptance_per_shown may not reduce task-completion time.A cited comparison found no measurable task-completion-time impact despite positive developer feedback.
- The study considers one completion system with one fixed neural engine, so effects may differ with quality, latency, length, or interface.
7 Conclusion
The paper connects Copilot usage measurements with developer-reported productivity and finds acceptance rate more informative than persistence measures. It interprets this result as evidence that useful progress, not merely retained code, captures Copilot’s value, while suggesting chatbot-oriented evaluation for future work.
- The study sought correlations between usage measurements and user-reported productivity to connect Copilot’s productivity benefit with developer activity.
- Acceptance rate correlated better with reported productivity than measures of how long accepted completions persisted unchanged.The approach compared acceptance measurements with persistence measures motivated by possible post-acceptance deletion or correction.
- Copilot’s value lies in helping users make progress toward goals, so a useful template may be as valuable as a perfectly correct keystroke-saving suggestion.
- The authors suggest viewing IDE code suggestions more like chatbot conversation, with specificity and repetition potentially affecting perceived quality.They identify programming parallels to factors previously linked with judgments of conversation quality.
- Future work will explore the chatbot analogy by borrowing ideas from chatbot and natural-language-generation evaluation.
8 Broader Impact
The paper identifies risks in evaluating neural code completion with acceptance rate and telemetry. Acceptance rate can support coarse comparisons, but it should not be treated as a singular quality criterion because it may favor popular use cases and be artificially optimized.
- A single acceptance-rate metric may bias tools toward popular languages, natural languages, IDEs, and locations, reducing quality for underrepresented groups.The authors recommend slicing data by these dimensions and avoiding changes that improve the aggregate metric while degrading other slices.
- Telemetry collection required to compute the metrics creates potential security and privacy concerns for users.The authors describe strict data-access controls and collaboration with organization and industry experts as mitigations.
- Blindly optimizing acceptance rate can encourage artificial changes that improve the proxy without substantially increasing user benefit.Splitting one suggestion into two could raise acceptance rate without increasing usefulness, or even while decreasing it.
- Acceptance rate is useful for applications such as comparing incremental model changes, but its validity is limited in other cases.The limitation is especially relevant when operational parameters change substantially.
A Summary of usage measurements collected
The appendix table summarizes the core usage metrics and places them alongside alternative normalization-based metrics for context.
- The table reports summary statistics for core metrics, including sample size, mean, standard deviation, minimum, median, and maximum.
- Other metrics derived from different normalization options are included for contextual comparison.
B Correlations between usage measurements and survey questions
The appendix examines correlations between survey-based productivity measures and usage metrics, including acceptance, persistence, and activity measures. It also maps survey statements to productivity dimensions and code metrics.
- The correlation table compares aggregate survey productivity scores with core usage metrics and their PLS scores.
- Correlations: mostly_unchanged_120_per_shown had coefficient 0.23 with P-value <0.0001 across N 1,780 observations.
- Correlations: unchanged_30_per_shown had coefficient 0.21 with P-value <0.0001 across N 1,780 observations.
- Correlations: mostly_unchanged_120_per_opportunity had coefficient 0.21 with P-value <0.0001 across N 1,789 observations.
- Correlations: unchanged_120_per_shown had coefficient 0.21 with P-value <0.0001 across N 1,780 observations.
- Survey mapping: The appendix relates survey statements to productivity aspects, SPACE dimensions, and corresponding code metric names.Statements cover learning, flow, task speed, repetitive work, search effort, perceived productivity, fulfillment, frustration, focus, unfamiliar languages, and code quality.
E Publicly posted comments
Public comments portray GitHub Copilot as an interactive partner that supports developers’ thinking, adapts to their style, and helps them work through code.
- Developers liken GitHub Copilot to pair programming with an eager, broadly knowledgeable junior developer.
- Suggestion cycling and manual editing are described as creating an enjoyable programming flow.
- One comment values the tool’s adaptation to the developer’s own code style.
- Comments frame Copilot as facilitating developers’ thinking rather than replacing it with finished code.
- Examples show developers steering suggestions toward desired implementations, including correcting an initial polynomial-evaluation approach.
- Copilot’s debugging assistance can be useful despite occasional errors, helping a developer reach a resolution.