GitHub Copilot AI pair programmer: Asset or Liability?

Arghavan Moradi Dakhel; Vahid Majdinasab; Amin Nikanjam; Foutse Khomh; Michel C. Desmarais; Zhen Ming; Jiang

Source-linked AI summary

GitHub Copilot AI pair programmer: Asset or Liability?

Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, Zhen Ming, Jiang

arXiv:2206.15331v2Jun 30, 2022cs.SEcs.LG

TL;DR

Developers need empirical evidence on whether Copilot can serve as an effective pair programmer beyond merely producing correct code. This study evaluates its algorithmic solutions and compares them with human solutions, finding that Copilot can help expert developers but requires them to detect and filter buggy or non-optimal suggestions.

Problem
Prior studies had not compared Copilot’s code quality with humans or examined its buggy solutions and suggestion diversity for effective pair programming.
Method
The study empirically evaluates Copilot on fundamental algorithmic problems and compares its generated code with human solutions using repair costs.
Results
Copilot generates correct and optimal solutions for some fundamental problems, but its quality depends on prompt quality and expert review remains necessary to identify buggy or non-optimal suggestions.
Takeaways & Limitations
Copilot can be an asset when used by expert developers as a pair programmer, but a liability for users unfamiliar with the problem context and coding methods.
Takeaways & Limitations
The paper discusses study boundaries and threats to validity, without specifying them in the supplied passages.

Abstract

from arXiv · show

Automatic program synthesis is a long-lasting dream in software engineering. Recently, a promising Deep Learning (DL) based solution, called Copilot, has been proposed by OpenAI and Microsoft as an industrial product. Although some studies evaluate the correctness of Copilot solutions and report its issues, more empirical evaluations are necessary to understand how developers can benefit from it effectively. In this paper, we study the capabilities of Copilot in two different programming tasks: (i) generating (and reproducing) correct and efficient solutions for fundamental algorithmic problems, and (ii) comparing Copilot's proposed solutions with those of human programmers on a set of programming tasks. For the former, we assess the performance and functionality of Copilot in solving selected fundamental problems in computer science, like sorting and implementing data structures. In the latter, a dataset of programming problems with human-provided solutions is used. The results show that Copilot is capable of providing solutions for almost all fundamental algorithmic problems, however, some solutions are buggy and non-reproducible. Moreover, Copilot has some difficulties in combining multiple methods to generate a solution. Comparing Copilot to humans, our results show that the correct ratio of humans' solutions is greater than Copilot's suggestions, while the buggy solutions generated by Copilot require less effort to be repaired.

1. Introduction · 2. Related Works

The paper positions Copilot within the broader progress of deep-learning-based program synthesis and addresses gaps in prior evaluations by studying solution quality, reproducibility, diversity, and human comparison. It reports that Copilot can solve most fundamental algorithmic problems but may produce buggy or non-reproducible code, has lower correctness and diversity than human solutions, and requires expert oversight.

1. Introduction: Deep Learning and Transformer advances have revived software engineering’s goal of automating code generation for programming assistance and algorithm discovery.Program synthesis can support teaching, programmer assistance, and discovering algorithmic solutions.
1. Introduction: Codex, the GPT-3-based model underlying Copilot, was pretrained on 159 GB of code from 54 million GitHub repositories and performed well on the HumanEval programming benchmark.Codex has up to 12 billion parameters and was evaluated on handwritten Python problems outside its training dataset.
1. Introduction: Prior studies mainly assessed Copilot’s correctness or isolated issues, leaving its code quality relative to humans, buggy suggestions, diversity, and effective developer use insufficiently examined.The paper therefore focuses on the quality of code Copilot adds to software projects rather than only the task types or difficulty levels it can handle.
1. Introduction: The study evaluates Copilot on fundamental algorithmic problems for correctness and reproducibility, then compares its programming-task solutions with human-provided solutions.The authors assume findings from these datasets can generalize to more professional programming tasks because programs concretely formulate abstract algorithms.
1. Introduction: Copilot provides efficient solutions for most fundamental problems, but some are buggy or non-reproducible, and it struggles to combine multiple methods into one solution.Compared with human programmers, Copilot’s task solutions have lower correct ratio and diversity, although its buggy code can be repaired easily.
1. Introduction: Copilot may be an asset when experts supervise it but a liability for novices who may accept buggy, complex, or difficult-to-understand suggestions as correct.The paper contributes an empirical study of fundamental algorithmic problems, a comparison with human Python solutions, and publicly available data and detailed results for replication.
2. Related Works: Related studies examined GP-based synthesis, task assistance, correctness, vulnerabilities, productivity, and non-determinism, but none compared Copilot with humans on programming tasks.This study instead combines fundamental algorithmic problems with human comparisons on real-world programming tasks.

3. Study Design

The study evaluates Copilot through two complementary methods: testing recommendations for fundamental algorithmic problems and comparing them with human-provided solutions. It uses problems drawn from an algorithm-design textbook, with summarized and cross-validated prompts for Copilot.

Research Questions: The evaluation addresses whether Copilot produces correct, efficient solutions for fundamental algorithmic problems and competitive solutions relative to humans.These aims correspond to RQ1 and RQ2.
RQ1: Fundamental Algorithms: The fundamental-problem benchmark covers sorting algorithms, binary search trees, elementary graph algorithms, and greedy activity selection.The sorting set includes eight algorithms, while graph tests include graph construction, BFS, DFS, DAGs, and reachable vertices.
RQ1: Fundamental Algorithms: Problems were selected from a widely used algorithm-design textbook and chosen across implementation complexities ranging from easy to hard.The textbook organizes principal algorithms into categories used to structure the Copilot tests.
Prompt Engineering: Because detailed textbook descriptions can be lengthy, the researchers summarized each problem while cross-checking descriptions against coding websites to reduce memorization concerns.The second author created the summaries, and the first two authors cross-validated them for correctness, understandability, and sufficient information.
RQ1: Fundamental Algorithms: For problems requiring prior code, Copilot first generated underlying subproblems and then used that generated code to implement succeeding solutions.This sequential procedure was applied because such reuse is common in functional and object-oriented programming.

2. Correct Ratio

This section defines the percentage of correct solutions suggested by Copilot in each trial and introduces related measures of optimality, reproducibility, and similarity.

Correct Ratio: The correct ratio is the percentage of correct solutions suggested by Copilot in each trial.It is reported as a percentage (%).
Correct Ratio: Code optimality records whether at least one correct solution suggested in each trial has optimal time complexity.The measure is binary: Yes or No.
Correct Ratio: Code reproducibility records whether correct solutions are repeated within a trial or across two trials.Within-trial reproducibility compares attempts in one trial, whereas across-trial reproducibility compares the first and second trials.
Correct Ratio: Code similarity measures the similarity of correct solutions within a trial and across trials on a scale from 0 to 1.Within-trial similarity compares all correct solutions in one trial; across-trial similarity compares correct solutions between two trials.

(1) Response Received · (2) Correct Ratio

The study measures whether Copilot returns code rather than irrelevant responses and evaluates correctness using both functional tests and adherence to the requested algorithm. A solution counts as correct only after passing both checks within a trial.

(1) Response Received: “Response Received” records whether Copilot generates code for the problem description instead of irrelevant output.Irrelevant responses include repeated prompts, import-only code, or natural-language responses.
(1) Response Received: The researchers collected up to 10 Copilot suggestions across six attempts and two trials using problem descriptions written as comments.An attempt is successful when at least one suggestion within it contains code.
(2) Correct Ratio: Correct ratio is the fraction of Copilot suggestions per problem that are functional and address the problem’s objective.A suggestion is functionally correct if it passes a set of unit tests.
(2) Correct Ratio: For algorithmic problems, unit-test success alone is insufficient because the solution must also use the algorithm specified in the description.This criterion checks whether Copilot understands the intended solution structure.
(2) Correct Ratio: Functionally correct alternatives, such as Bubble sort or built-in sorting instead of Selection sort, are not correct when they fail to implement the requested algorithm.The same issue applies to implementing requested BST or graph data structures.
(2) Correct Ratio: The evaluation checks functional correctness on unit tests and separately compares Copilot’s selected algorithm with reference algorithms.The two first authors independently checked the suggested solutions, using references from a book and coding websites.
(2) Correct Ratio: Solutions within a trial are labeled correct only if they pass both evaluation steps, after which the correct ratio is calculated from the fraction of correct solutions.Cohen’s Kappa measures agreement between the two authors during algorithm comparison.

(3) Code Optimality · (4) Code Reproducibility and Similarity

The study evaluates Copilot’s ability to select optimal algorithms among correct suggestions and to reproduce and structurally resemble correct solutions across attempts and trials. Reproducibility and similarity are assessed using repeated outputs and AST-based comparisons that abstract away variable names, comments, and statement order.

(3) Code Optimality: Copilot’s code optimality metric tests whether at least one correct suggestion uses the problem’s optimal algorithm.The metric accounts for problems admitting multiple algorithms with different computational complexities.
(3) Code Optimality: Because optimality cannot be checked automatically, the authors compare correct Copilot solutions with a reference optimal implementation.The first two authors independently inspect whether an optimal algorithm appears among the correct solutions for each trial.
(4) Code Reproducibility and Similarity: Code Reproducibility measures whether Copilot reproduces a correct solution within a trial or across two trials over a time window.Within a Trial compares attempts in one trial, whereas Across Trials compares solutions between two trials.
(4) Code Reproducibility and Similarity: AST similarity compares correct solutions after pruning variable and function-name leaves and ignoring comments or natural-language text.The measure ranges from 0 to 1, with 1 indicating structural equivalence regardless of semantic similarity.
(4) Code Reproducibility and Similarity: The similarity measure is unaffected by different statement orders or variable names but distinguishes semantically similar programs implemented differently.It returns 1 for structurally equivalent recorded programs and structurally equivalent renamed identical programs.
(4) Code Reproducibility and Similarity: Within a Trial reproducibility is positive when at least one correct solution appears across all 3 attempts with similarity equal to 1.Otherwise, the problem is marked negative for Within a Trial reproducibility.
(4) Code Reproducibility and Similarity: Code Similarity is reported as the average AST similarity between pairs of correct solutions within a trial or between two trials.The average comparison is reported separately for each fundamental algorithmic problem.

1. Correct Ratio (pass@Topk) … (4) Cyclomatic Complexity

The study defines evaluation procedures for Copilot’s correctness, repairability, solution diversity, and understandability, including comparisons with human submissions. It emphasizes that Copilot can produce duplicates and buggy solutions, while repair effort and code complexity provide additional quality measures.

(1) Correct Ratio (pass@Topk): Because Copilot returns only its Top 10 solutions per attempt, the study cannot accurately calculate the standard pass@k metric.Instead, it evaluates pass@Topk across multiple attempts by measuring the fraction of collected suggestions that pass all test units.
(1) Correct Ratio (pass@Topk): Correct Ratio is the fraction of correct Copilot or human solutions, calculated per problem from Copilot’s Topk suggestions across attempts or students’ submissions.The study also examines how increasing Copilot attempts affects the distribution and average of Correct Ratio.
(2) Repairing Costs: The study compares Copilot’s buggy solutions with students’ buggy submissions because some Copilot outputs can become correct after small changes.Repairing cost is treated as an indicator of code quality, with longer repair times potentially reflecting structural problems and larger required changes.
(2) Repairing Costs: An automated repair tool is used to reduce dependence on one human’s expertise and reflect common automated bug-fixing practices in software projects.The tool reports Repair Rate, Avg. Repair Time in seconds, and Relative Patch Size based on normalized AST Tree-Edit-Distance.
(3) Diversity: Solution diversity is measured after removing duplicates using AST similarity, while increasing attempts is examined for its effect on unique solutions and Copilot–student novelty.The comparison includes both correct and buggy solutions.
(3) Diversity: Although Copilot claims to remove duplicates within one attempt, observations show structurally identical Top 10 suggestions differing only in variable names or comments.The example contains three solutions with the same approach, including differences such as “item” versus “i” and an additional comment.
(4) Cyclomatic Complexity: Programming tasks can have multiple valid implementations, but differing syntax and built-ins may affect efficiency, readability, maintainability, and demonstrated language mastery.The paper motivates comparing alternative implementations as part of evaluating generated code quality.
(4) Cyclomatic Complexity: Cyclomatic Complexity measures independent paths or decisions and is used to estimate code understandability, readability, and effort required for modification.Lower C.C. indicates more readable and understandable code; the study compares Copilot and human solutions using RADON, with C.C. close to or above 10 interpreted as poor practice.

(5) Syntactic Mastery … (2) Correct Ratio

The study evaluates Copilot across syntactic diversity and algorithmic problem-solving outcomes, including response receipt and correctness. Copilot can produce diverse or advanced solutions, but its responses and correctness vary with problem difficulty and trial.

(5) Syntactic Mastery: Syntactic diversity reflects familiarity with programming keywords and built-ins, but greater diversity does not necessarily indicate the best practice.The study frames syntax diversity as a possible indicator of developer mastery while noting that less diverse solutions may be more advanced and less complex.
(5) Syntactic Mastery: The study measures syntax patterns by converting solutions to abstract syntax trees and collecting keyword and built-in-function nodes.Built-in calls are identified by matching AST call-node names against a Python built-ins list.
4. Empirical results: The empirical-results section presents findings for the research questions one by one and separates first- and second-trial results for solutions within a trial.The two trials were conducted 30 days apart, and evaluation criteria differed by problem category.
(5) Syntactic Mastery: Figure 4 contrasts a human solution with cyclomatic complexity 4.13 and a Copilot solution with cyclomatic complexity 1, with the latter described as more readable and less complex.The human code has more diverse syntax patterns and built-ins, whereas the Copilot code uses more advanced syntax and built-in methods.
(1) Response Received: For sorting algorithms, author agreement was 89% for metrics requiring manual inspection, including response received, correct ratio, and code optimality.The agreement was reported for manual validation of sorting-algorithm results.
(1) Response Received: On harder sorting tasks, Copilot’s response receipt varied across trials: it generated Heap and Radix sort code in one of three first-trial attempts but all three second-trial attempts.Merge sort showed the opposite trial pattern, according to the reported comparison.
(2) Correct Ratio: 100% of Copilot’s first-trial bubble-sort suggestions and 85.71% of bucket-sort suggestions were correct, while correctness varied across trials and problem difficulty.The paper reports that Copilot may also select an incorrect implementation algorithm as task difficulty changes.
(2) Correct Ratio: Heap sort had 0% correct suggestions in the first trial and 9.09% in the second, while Radix sort had no correct second-trial code despite responses in all three attempts.The paper attributes Heap sort’s difficulty to implementing a max heap before writing the sorting function.

(3) Code Optimality … (2) Correct Ratio

Copilot often produced optimal sorting solutions when correct, but correctness and exact reproducibility varied across trials. For BST tasks, it generally understood the problems, yet response generation and correctness were inconsistent.

(3) Code Optimality: 7 of 8 sorting algorithms received correct and optimal solutions in trial 1, compared with 4 in trial 2.Optimality was not applicable when no correct solution existed.
(3) Code Optimality: Copilot sometimes generated Pythonic code using list comprehensions and built-in functions instead of reimplementing functionality.For Quick sort, one suggestion used the built-in sort function after partitioning.
(4) Code Reproducibility and Similarity: Correct sorting solutions were not exactly reproduced for the majority of algorithms.Similarity remained high in some cases despite non-reproduction, including 0.99 for Quick sort’s second-trial correct solutions.
(4) Code Reproducibility and Similarity: 6 of 8 sorting problems produced responses in trial 1, compared with 7 of 8 in trial 2.Response Received measures whether Copilot generated code for the proposed prompt.
(2) Correct Ratio: 51.42% of solutions were correct on average in trial 1, versus 34.77% in trial 2.Trial 1 produced correct solutions for all sorting problems except Heap sort; trial 2 produced them for 4 of 8 problems.
(4) Code Reproducibility and Similarity: Correct solutions were not exactly reproduced for the 4th, 3rd, and 2nd sorting problems in trial 1, trial 2, and across trials, respectively.Some non-reproduced correct solutions had similarity above 0.6; several metrics were unavailable without comparable correct solutions.
(1) Response Received: In the second trial, Copilot struggled in 2 of 3 attempts to generate the BST data structure, although it understood the BST problems in both trials.The BST evaluation covered the data structure and operations including insertion, extrema, traversal, and successor finding.
(2) Correct Ratio: 94.12% of in-order tree-walk suggestions were correct in trial 1, falling to 16.67% in trial 2.Correct ratios for Min and Max Values in Tree and Finding The Successor Node were close across trials.

(3) Code Optimality … (3) Code Optimality

Copilot generally generated algorithmic code and often achieved optimal solutions, but its correctness varied across trials, with recurring bugs and limitations in reproducing or composing prior code. Graph results further showed that reused errors could propagate through dependent algorithms and reduce optimality.

(3) Code Optimality: Copilot produced at least one correct solution with optimal time complexity for every BST problem except “Finding successor nodes” in the first trial.It also generated iterative and recursive versions for several BST tasks, although duplicated functions sometimes over-complicated the code.
(4) Code Reproducibility and Similarity: Copilot’s BST correctness declined from 81.86% in the first trial to 54.76% in the second trial.The paper characterizes this as inconsistent behavior across trials.
(4) Code Reproducibility and Similarity: Within trials, Copilot struggled to exactly reproduce correct BST solutions, whose similarity did not exceed 0.51 when they differed.Across trials, it reproduced at least one first-trial correct solution for all BST problems except “Finding maximum and minimum values in the Tree”; those solutions had similarity 0.83.
(1) Response Received: Copilot generated code in all three attempts for every graph problem except “Simple Data Structure” and “Directed Acyclic Data Structure” in the first trial.In the second trial, it struggled in only one of three attempts on “Simple Data Structure”.
(2) Correct Ratio: Graph correctness varied sharply across trials: first-trial correct ratios were 50% for “Simple Data Structure”, 75% for “Depth First Search”, and 86.37% for “Directed Acyclic Data Structure”.In the second trial, Copilot did not generate correct solutions for those three problems.
(2) Correct Ratio: Copilot sometimes generated BFS and DFS code together despite being asked for only one algorithm.This indicates difficulty separating requested graph-algorithm outputs across attempts.
(2) Correct Ratio: A node-naming assumption caused a bug to snowball through successor methods, although the affected code was labeled correct under normal usage.The code failed when nodes used names other than expected numeric or conventional labels.
(3) Code Optimality: In the first graph trial, Copilot generated one optimal solution for each problem, whereas in the second trial only BFS included an optimal solution among two correctly addressed problems.For cyclic checking, failure to reuse BFS or DFS code produced repeated operations considered non-optimal; list comprehensions were accepted as optimal for graph construction.

(4) Code Reproducibility and Similarity … (3) Code Optimality

Copilot can understand and generate code for elementary graph and activity-selection problems, but its outputs are often buggy, inconsistent across trials, and difficult to reproduce exactly. Optimality could only be assessed for two simple activity-selection tasks because most generated solutions were incorrect.

(4) Code Reproducibility and Similarity: Copilot exactly reproduced at least one correct solution in the first trial for only BFS and Finding Reachable Vertices.For DFS and Directed Acyclic Data Structure, the corresponding similarities were 0.73 and 0.63.
(4) Code Reproducibility and Similarity: Across trials, Copilot exactly reproduced a correct solution only for BFS, while Finding Reachable Vertices had similarity 0.076.In the second trial, it reproduced the correct solutions for the two problems it addressed correctly.
(4) Code Reproducibility and Similarity: In the first trial, Copilot generated correct solutions for all graph problems with an average correct ratio of 74.27%.In the second trial, it generated correct solutions for only two problems, and 100% of its correct solutions were correct.
(3) Code Optimality: Copilot’s code was optimal for all graph problems it addressed correctly in both trials except Finding Reachable Vertices in the second trial.For activity selection, optimality was analyzed only for Comparing activities and Adding Activities to a Set because most solutions were incorrect; these tasks were simple and required boundary or Boolean checks.
4.1.4. Greedy Algorithms: The activity-selection evaluation asked Copilot to implement the activity class, compare activities, and check overlaps to assess whether its solutions were greedy.The problem requires non-overlapping activities, including start times after previous end times and no activity occurring during another.
(1) Response Received: Copilot was capable of understanding the activity-selection problem and generating code, including when given the entire problem definition in one prompt.The full definition included the activity class, activity comparison, and adding activities to a set.
(2) Correct Ratio: Copilot’s generated activity-selection code was either buggy or incorrect despite understanding the requested problem.One comparison method implemented the description but failed when inputs were not activity instances because it lacked type checking.

(4) Code Reproducibility and Similarity

Copilot’s generated solutions were often non-reproducible or duplicated, and it struggled with multi-function and correctness requirements. Increasing attempts or collecting more suggestions can raise correctness, while buggy Copilot solutions were generally easier and faster to repair than students’ solutions.

Reproducibility: Copilot reproduced solutions only for “Adding activities to a set,” while “Comparing Activities” produced identical solutions within one trial but no correct solution in the second trial.These results show both cross-trail non-reproducibility and trial-dependent correctness.
Task compliance: Copilot failed the activity-selection task because it mishandled type checking and variable-boundary checking despite explicit prompt requirements.For “q2: Unique Dates Months,” it also attempted one function instead of the three requested functions.
Correctness across suggestions: pass@Topk generally increases with k, but Copilot could not find a correct solution for “q2: Unique Dates Months” among Top1–Top10 suggestions.Each Copilot attempt contained only the Top10 suggestions, and the task required implementing three different functions.
Correctness across attempts: The average Correct Ratio increased from 0.32 in the first attempt to 0.44 in the fifth, although trends varied by task.For example, q1 decreased from 0.7 in atp4 to 0.4 in atp5, while q5 increased from 0.7 to 0.9.
Repair cost: Copilot’s buggy solutions took 4.94 seconds to repair on average versus 6.48 seconds for students, alongside lower average Relative Patch Size values of 0.33 versus 0.35.Correct-and-buggy pairs sometimes had BLEU scores of 0.75 or greater, indicating that small edits could produce a correct solution.

5. Discussion and Limitation

Copilot struggles with natural-language details, long or multi-part descriptions, and ambiguous input structures that humans readily interpret. Rewriting requirements with technical keywords, decomposing tasks, or adding sample tests substantially improves some correct ratios.

Limitations: Copilot misinterpreted explicit requirements, using sort and sorted despite instructions and returning ascending tuples in 92% of q4 suggestions.Most students avoided those built-ins and considered the requested sorting order, including in buggy submissions.
Limitations: Copilot may misunderstand problems entirely when descriptions contain multiple sentences or require combining multiple functions.The paper also reports difficulty putting explicit limits on variables and understanding long descriptions.
Prompt revisions: Changing q4’s “older people are at the front” wording to “descending order” improves CR from 14% to 79%.The result suggests technical keywords can make requirements more understandable to Copilot.
Prompt revisions: For q2, Copilot’s CR equals zero because all 50 generated solutions failed at least one unit test when asked to implement three functions together.Separating the descriptions into three functions increases CR for unique day and contains unique day, but unique month remains at zero.
Prompt revisions: Adding a sample unit test to the unique month description improves CR from 0% to 91%.Manual checking found that Copilot interpreted the second birthday-tuple item as the month, whereas tests used the first item.

6. Threats to Validity

The study’s validity is constrained by Copilot’s closed-source, evolving behavior and the use of classical tasks and student participants. Quantitative metrics reduce bias but cannot capture human-tool interaction or the full construct of an AI pair programmer.

Internal validity: Copilot’s closed-source model and training data prevent assessing its expected behavior, memorization, or solution originality.The researchers can only investigate Copilot’s functionality through its suggestions.
Internal validity: Copilot’s suggestions change over time and are inconsistent, so repeated experiments may not produce the same results.Variation may reflect both LLM inconsistency and continuous product improvement through new code samples or queries.
External validity: Because no industrial dataset pairs programming-task statements with corresponding code, the study relies on classical programming tasks to assess Copilot.This follows prior software-engineering research and reflects the stated advantages of such tasks.
External validity: The student participants are assumed to represent junior developers, but their unknown backgrounds mean they may not represent the broader population.The comparison with students was chosen to support a fair evaluation of Copilot.
Construct validity: Quantitative metrics reduce the chance of biased conclusions but cannot assess how humans interact with Copilot.The metrics were selected based on prior software-engineering studies.
Construct validity: Quantitative metrics cannot capture all capacities of a good AI pair programmer, making human experience necessary for a comprehensive assessment.Pair programming involves interaction between humans or between a human and a tool, so user opinions are required.

7. Conclusion

Copilot can generate correct and optimal solutions for some fundamental algorithmic problems, but output quality depends greatly on prompt conciseness and depth. It is an asset with expert oversight, but may become a liability when buggy or non-optimal suggestions are trusted, especially by novices.

Findings: Copilot generates correct and optimal solutions for some fundamental algorithm-design problems, but code quality depends greatly on prompt conciseness and depth.The study also indicates that Copilot still needs further development in fully understanding natural language.
Findings: Copilot can be an asset as an expert-supervised pair programmer, but an expert is required to detect and filter buggy or non-optimal solutions.Its solutions are described as more advanced than junior developers’ and comparable to humans’ in correctness, optimality, reproducibility, and repairing costs.
Limitations: Copilot solutions may be troublesome when novice developers or students fully trust them, although its suggestions may help improve their programming skills.Wider access is expected to enrich Copilot’s training dataset while exposing more shortcomings.
Future work: A filtering layer that removes buggy and non-optimal suggestions could reduce Copilot’s liability in software projects.Future studies should examine more diverse programming tasks with heterogeneous participants in human-centered comparisons with human programmers.