PaperlayerView exact revision on arXiv

Source-linked AI summary

GitHub Copilot AI pair programmer: Asset or Liability?

Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, Zhen Ming, Jiang

arXiv:2206.15331v2cs.SEcs.LG

TL;DR

Prior evaluations had not compared Copilot's code quality with human code or examined the diversity of its buggy suggestions. This study evaluates Copilot on fundamental algorithmic problems and against human solutions, finding correct and optimal solutions for some problems while concluding that experts are needed to filter buggy or non-optimal suggestions.

  • Problem

    Prior studies had not compared Copilot's code quality with human code or examined buggy suggestions and their diversity, leaving its effective use as a pair-programming tool unclear.

    Source: §1 · p. 2

  • Method

    The study empirically evaluates Copilot's suggestions on fundamental algorithmic problems and compares its generated code with human solutions, including through buggy-code repair costs.

    Source: §1 · p. 2, (2) Repairing Costs · p. 10

  • Results

    Copilot generated correct and optimal solutions for some fundamental problems, but code quality depended on prompt conciseness and depth, and experts remained necessary to filter buggy or non-optimal suggestions.

    Source: §7 · p. 25

  • Takeaways & Limitations

    Copilot can be an asset in software projects when expert developers use it as a pair programmer, but a liability for users unfamiliar with the problem context and correct coding methods.

    Source: §7 · p. 25

    The discussion identifies Copilot's boundaries as a limitation to address in making it more beneficial for real programming tasks.

    Source: §5 · p. 22

Abstract

from arXiv · show

Automatic program synthesis is a long-lasting dream in software engineering. Recently, a promising Deep Learning (DL) based solution, called Copilot, has been proposed by OpenAI and Microsoft as an industrial product. Although some studies evaluate the correctness of Copilot solutions and report its issues, more empirical evaluations are necessary to understand how developers can benefit from it effectively. In this paper, we study the capabilities of Copilot in two different programming tasks: (i) generating (and reproducing) correct and efficient solutions for fundamental algorithmic problems, and (ii) comparing Copilot's proposed solutions with those of human programmers on a set of programming tasks. For the former, we assess the performance and functionality of Copilot in solving selected fundamental problems in computer science, like sorting and implementing data structures. In the latter, a dataset of programming problems with human-provided solutions is used. The results show that Copilot is capable of providing solutions for almost all fundamental algorithmic problems, however, some solutions are buggy and non-reproducible. Moreover, Copilot has some difficulties in combining multiple methods to generate a solution. Comparing Copilot to humans, our results show that the correct ratio of humans' solutions is greater than Copilot's suggestions, while the buggy solutions generated by Copilot require less effort to be repaired.

1. Introduction · 2. Related Works

The paper evaluates Copilot on fundamental algorithmic problems and compares its solutions with human solutions on programming tasks. It reports limitations in correctness, reproducibility, and context understanding, while identifying circumstances in which Copilot may help or harm software projects.

1. Introduction

  • The study assesses algorithmic correctness and reproducibility and compares Copilot with human solutions on programming tasks containing up to 4,000 human solutions, assuming findings generalize to professional work.The authors note that these datasets may not represent typical professional programming tasks.Source: §1 · p. 2
  • Copilot efficiently solved most fundamental problems, but some solutions were buggy or non-reproducible; compared with humans, its solutions were less often correct and less diverse, though bugs were easier to repair.The study also observed difficulty combining multiple methods and understanding details of problem context that humans readily understood.Source: §1 · p. 2
  • Copilot may be an asset when experts use it, but a liability for novices who may accept buggy, difficult-to-understand suggestions as correct and harm project quality.The paper frames safe use as depending on familiarity with problem context and correct coding methods.Source: §1 · p. 2
  • The paper contributes empirical evaluations of Copilot on fundamental algorithms and Python problems against human solutions, and makes its dataset and detailed results publicly available.The released materials are intended to support replication and further research or practice.Source: §1 · p. 2

2. Related Works

  • Prior work found Genetic Programming slower and costly to train [47], while a 24-participant study found Copilot offered useful starting points without necessarily improving task time or success.The participant study involved three Python tasks, mostly centered on programming-language libraries, and did not compare suggestions with unaided solutions.Source: §2 · pp. 2–3
  • Codex studies reported better grades than most students and varied solutions, while Copilot’s LeetCode evaluation tested correctness and understandability but not whether solutions were optimal.The Codex work used programming-course questions and a rainfall problem; the LeetCode study covered 33 questions in four languages.Source: §2 · p. 3
  • Other studies examined vulnerability reproduction [40], [4], copyright and safety concerns, and developer productivity, finding solution acceptance the most relevant productivity indicator [56].The productivity survey included 2,631 developers and analyzed interaction metrics such as acceptance and persistence rates.Source: §2 · p. 3
  • Prior studies had not compared Copilot with humans on programming tasks, motivating this study’s focus on fundamental algorithms and real-world programming problems.Most earlier work emphasized correctness and issues such as vulnerabilities in generated solutions.Source: §2 · p. 3

3. Study Design

The study evaluates Copilot’s recommendations for fundamental algorithmic problems and compares its solutions with human-provided solutions. Its design specifies the problem sets, prompts, generation procedure, and evaluation criteria for these inquiries.

3. Study Design

  • The study asks whether Copilot can recommend adequate code for fundamental algorithmic problems and whether its solutions are competitive with human solutions.These are the study’s two research questions.Source: §3 · p. 3

3.1. RQ1: Copilot on Algorithm Design

  • RQ1 examines whether Copilot can produce correct, efficient solutions from clear problem descriptions, treating fundamental algorithmic ability as a prerequisite for more complex software tasks.The authors note these problems are not necessarily typical of professional projects, but professional tasks combine fundamental algorithmic and data-structure problems.Source: §3.1 · p. 4

3.1.1. Data set: Fundamental Algorithmic Problems

  • The RQ1 dataset draws on and covers sorting, binary search trees, elementary graph algorithms, and greedy activity selection.The selected sorts are bubble, bucket, heap, insertion, merge, quick, radix, and selection sort.Source: §3.1.1 · p. 4
  • The selected binary-search-tree tasks include finding minimum and maximum values, in-order traversal, and finding a node’s successor.The graph set also includes breadth-first search, depth-first search, directed acyclic graphs, and reachable-vertex problems.Source: §3.1.1 · p. 4

3.1.2. Prompt Engineering

  • To limit prompt length and memorization, the authors summarized ’s problems in their own words, checked descriptions against coding websites, and had authors cross-check them.The cross-checking authors reported Cohen’s Kappa of 0.95; they discussed disagreements and obtained agreement from a third author.Source: §3.1.2 · p. 5
  • For problems other than sorting, Copilot was asked to generate code for underlying subproblems first, then use that code for succeeding problems.This followed the book’s examples; the prompts and responses were recorded in the replication package.Source: §3.1.2 · p. 5

3.1.3. Solving Fundamental Algorithmic Problems with Copilot

  • Each prompt was tried six times, collecting up to ten suggestions per attempt across two trials of three attempts, with the second trial 30 days later.Unrelated script names were used to prevent filenames from influencing Copilot’s queries.Source: §3.1.3 · p. 6

3.1.4. Evaluation Criteria

  • The evaluation criteria are calculated per fundamental algorithmic problem; “Response Received” counts attempts per trial in which Copilot understood the problem and generated code.The metric ranges from 0 to 3.Source: §3.1.4 · p. 6

2. Correct Ratio (%).

This section defines measures of Copilot’s solution correctness, optimality, reproducibility, and similarity.

2. Correct Ratio (%).

  • Correct ratio is the percentage of Copilot-suggested solutions that are correct in each trial.Source: §2 · p. 6
  • Code optimality records whether at least one correct solution suggested in each trial has optimal time complexity.Source: §2 · p. 6
  • Code reproducibility records whether a correct solution is repeated across attempts within a trial or between the first and second trials.Within-trial repetition compares one attempt with two other attempts; across-trial repetition compares the first trial with the second.Source: §2 · p. 6
  • Code similarity measures the similarity of correct solutions within a trial and across two trials on a scale from 0 to 1.Source: §2 · p. 6

(1) Response Received · (2) Correct Ratio

The study measures whether Copilot returns relevant code and whether its suggestions are both functional and faithful to the described algorithm. Correctness is assessed per trial using unit tests and, for algorithmic problems, checks of algorithm choice.

(1) Response Received

  • The metric distinguishes code for the summarized program description from irrelevant replies such as repeated prompts, import-only code, or natural-language responses.Source: (1) Response Received · p. 6
  • Response Received counts an attempt when at least one of Copilot’s top 10 suggestions contains code, with three attempts measured per trial.The authors collected suggestions across six attempts and two separate trials.Source: (1) Response Received · p. 6

(2) Correct Ratio

  • Correct ratio is the fraction of Copilot’s suggestions per problem that are functional and address the objective, with code correctness checked using unit tests.Source: (2) Correct Ratio · p. 6
  • For algorithmic problems, passing unit tests is insufficient: suggestions must also use the algorithm specified, such as Selection sort rather than Bubble sort or a built-in sort.The same algorithm-adherence check applies to BST and graph implementations.Source: (2) Correct Ratio · p. 6
  • A suggestion is labeled correct only after passing both the unit-test and algorithm-adherence checks, and the correct ratio is calculated within each trial.Source: (2) Correct Ratio · pp. 6–7
  • Two authors independently compared Copilot’s selected algorithms with reference algorithms, using ground truth from,, and and Cohen’s Kappa to measure agreement.Source: (2) Correct Ratio · pp. 6–7

(3) Code Optimality · (4) Code Reproducibility and Similarity

The study assesses whether Copilot finds optimal algorithms among its correct suggestions and whether it reproduces correct code across attempts and trials, while measuring structural similarity between solutions.

(3) Code Optimality

  • Copilot’s optimality is assessed by checking whether at least one correct suggestion in a trial uses the reference optimal algorithm; two authors evaluate solutions independently and report Cohen’s Kappa.Automatic checking is ruled out because of the halting problem.Source: (3) Code Optimality · p. 7

(4) Code Reproducibility and Similarity

  • Reproducibility asks whether Copilot produces a correct solution again within a trial or across two trials, despite the tool’s closed-source behavior.Within-trial checks span three attempts; across-trial checks span a time window between two trials.Source: (4) Code Reproducibility and Similarity · p. 7
  • AST similarity compares code structure after removing variable and function names and ignoring comments or natural-language text; it is unaffected by renamed code or statement order.Scores range from 0 to 1, with 1 indicating structural equivalence, not necessarily semantic similarity; semantically similar implementations can score differently.Source: (4) Code Reproducibility and Similarity · p. 7

3.2. RQ2: Copilot vs. Human

  • For RQ2, the study compares Copilot-generated code with student code across quantitative metrics, using programming tasks, generated solutions, and selected student submissions.The methods section introduces the dataset, Copilot generation process, student-solution selection, and comparison criteria.Source: §3.2 · p. 8

3.2.1. Dataset: Python Programming Tasks

  • The human comparison uses five assignments from a Python course, with human-written task descriptions, student submissions, unit tests, and tools for checking correctness and repair cost.The dataset contains 2,442 correct and 1,783 buggy submissions; it was selected to compare Copilot with junior developers on programming-course tasks.Source: §3.2.1 · p. 8

3.2.2. Solving Programming Problems with Copilot

  • For each task, Copilot returns its top 10 solutions per attempt; the study repeats this five times and labels a solution correct only if it passes all related unit tests.This yields 50 Copilot solutions per problem; other solutions are classified as buggy.Source: §3.2.2 · pp. 8–9

3.2.3. Downsampling Student Solutions

  • Because Copilot provides fewer solutions than the student dataset, the study downsamples student submissions to 50 to match Copilot’s solution count.The authors report an average of 689.8 student submissions for these tasks and avoid increasing Copilot attempts because that would add duplicates.Source: §3.2.3 · p. 9

3.2.4. Evaluation Criteria

  • The study compares Copilot and student solutions using multiple evaluation criteria, which the paper defines as separate metrics.For code similarity, it averages AST similarity across pairs of correct solutions, comparing attempts within a trial and solutions between trials.Source: (4) Code Reproducibility and Similarity · pp. 7–8, §3.2.4 · p. 9

1. Correct Ratio (pass@Topk) … (4) Cyclomatic Complexity

The section defines measures for correctness, repair cost, solution diversity, and code understandability. It also describes how Copilot and student solutions are evaluated on these measures.

(1) Correct Ratio (pass@Topk)

  • Copilot’s main result is measured with pass@Topk across repeated attempts, since its Top 10 outputs do not support standard pass@k calculation and pass@K cannot be calculated for students.For pass@Top2, for example, the study pools the Top 2 solutions from five attempts and calculates the fraction passing all test units.Source: (1) Correct Ratio (pass@Topk) · pp. 9–10
  • Correct ratio is the fraction of correct solutions among Copilot’s suggestions or students’ submissions for each problem, tracked across Copilot attempts.The study also examines the distribution and average of Copilot’s correct ratio across independent attempts to assess how additional attempts affect it.Source: (1) Correct Ratio (pass@Topk) · p. 10

(2) Repairing Costs

  • Repair cost is assessed by fixing buggy solutions and comparing the required changes, since code churn and structural problems can affect repair time.The study uses an automated repair tool to reflect common software practice and avoid bias from a particular human’s expertise.Source: (2) Repairing Costs · p. 10
  • The automated repair tool reports Repair Rate, average repair time in seconds, and Relative Patch Size, the normalized AST edit distance between buggy and repaired code.Repair Rate is the fraction of buggy programs passing all tests after repair; Relative Patch Size is normalized by the buggy code’s AST size.Source: (2) Repairing Costs · p. 10

(3) Diversity

  • Copilot’s Top 10 can contain functionally duplicate solutions despite its duplicate-removal claim; Figure 3 shows variants differing only in variable names or comments.The three q3 suggestions share the same approach, but Copilot treats them as unique because it compares character sequences.Source: (3) Diversity · p. 10, Figure 3 · p. 11
  • The study investigates whether increasing attempts—and thus total solutions—increases unique solutions, and compares the diversity of Copilot’s correct and buggy solutions with students’ solutions on a programming task.Prior work found that increasing sampled solutions can increase correct solutions passing all test units [8] [31], but did not determine whether the increase reflects diversity or duplication.Source: (3) Diversity · p. 10

(4) Cyclomatic Complexity

  • The study compares Copilot’s and humans’ cyclomatic complexity using RADON 3; complexity close to or above 10 is interpreted as not best-practice code.Lower complexity indicates more readable and understandable code; the paper’s q4 example reports complexity 4.13 for nested loops and 1 for a sort-and-lambda solution [13].Source: (4) Cyclomatic Complexity · p. 11

(5) Syntactic Mastery … (2) Correct Ratio

Copilot’s sorting performance varied across trials: easier algorithms sometimes had high correct ratios in the first trial but none in the second, while harder algorithms also showed inconsistent or absent correctness. The study also compares Copilot’s and humans’ use of syntax and built-ins, noting that greater syntax diversity need not mean a better solution.

(5) Syntactic Mastery

  • The syntactic-mastery criterion compares the diversity of programming keywords and Python built-ins in Copilot’s and human solutions, since syntax familiarity can reflect developer experience but does not guarantee best practice.The authors convert solutions to ASTs and collect syntax nodes and built-in function calls; pair programming can transfer expert experience to novices.Source: (5) Syntactic Mastery · p. 11

4.1. RQ1: Copilot on Algorithm Design

  • The study reports results by research question and evaluates Copilot on algorithmic problems across two trials conducted 30 days apart, considering problem-category difficulty.For each marker, results within a trial are shown separately as first- and second-trial results.Source: §4 · p. 11, §4.1 · p. 12

4. Empirical results

  • Figure 4 contrasts a human solution with more diverse syntax and built-ins (cyclomatic complexity 4.13) against Copilot’s less complex solution (1), which uses more advanced syntax and methods.The caption describes Copilot’s solution as more readable and less complex to understand.Source: (5) Syntactic Mastery · p. 11, Figure 4 · p. 12

4.1.1. Sorting Algorithms

  • For sorting metrics requiring author judgment, the authors achieved 89% Kappa agreement.These metrics included response received, algorithm validation for correct ratio, and code optimality.Source: §4.1.1 · p. 12

(1) Response Received

  • For Heap and Radix sort, Copilot generated code in one of three first-trial attempts and all three second-trial attempts; Merge sort had three first-trial responses and two second-trial responses.The authors report that Copilot struggled to generate solutions as implementation details increased, with different behavior across trials.Source: (1) Response Received · p. 12

(2) Correct Ratio

  • Bubble and bucket sort suggestions were 100% and 85.71% correct, respectively, in the first trial, but neither had correct solutions in the second.The authors report that problem difficulty affected correctness, alongside variation between trials.Source: (2) Correct Ratio · p. 12
  • Heap sort had no correct first-trial solutions but 9.09% correct suggestions in the second; second-trial Radix sort had no correct solutions despite three generated codes.Heap sort requires implementing a max heap and then a sorting function.Source: (2) Correct Ratio · p. 12
  • Some second-trial Bucket sort suggestions called an unimplemented Quick sort function, while reviewers accepted selection-sort solutions that followed the algorithm without the prompt’s exact steps.For selection sort, reviewers ultimately agreed to count solutions as correct even if they did not create the specified sorted and unsorted lists.Source: (2) Correct Ratio · p. 12

(3) Code Optimality … (2) Correct Ratio

Copilot’s sorting results combine substantial correct-solution rates with optimality among its correct outputs, but reproducibility varies; BST results likewise show inconsistent correctness across trials for some tasks.

(3) Code Optimality

  • Copilot generated correct sorting solutions for 7 of 8 algorithms in the first trial and 4 in the second, with optimality inapplicable when no correct solution was produced.Some algorithms, such as bubble sort and heap sort, have no implementation other than the optimal one.Source: (3) Code Optimality · pp. 12–13
  • For Quick sort, Copilot sometimes used Python’s built-in sort on subarrays; it also used for-loops or list comprehensions instead of while loops, avoiding infinite-loop risk.List comprehensions can improve running time, and Figure 5 also shows a buggy Quick sort suggestion that sorts partitions with the built-in function.Source: (3) Code Optimality · p. 13, Figure 5 · p. 13

(4) Code Reproducibility and Similarity

  • Copilot’s correct sorting solutions were not exactly reproduced for most sorting algorithms, although some non-identical outputs were similar, including Quick sort at 0.99 and selection sort at 0.61 and 0.63.The selection-sort similarities refer respectively to the first trial and across trials; similarity and reproducibility are not applicable without comparable correct solutions.Source: (4) Code Reproducibility and Similarity · p. 13, Table 3 · p. 15
  • Copilot responded to 6 of 8 sorting problems in the first trial and 7 of 8 in the second.The paper reports that correct solutions were not exactly reproduced for the fourth sorting problem in the first trial, the third in the second, and the second across trials.Source: (4) Code Reproducibility and Similarity · p. 13

(2) Correct Ratio

(1) Response Received

  • Copilot understood the BST problems in both trials, but struggled to suggest code in 2 of 3 attempts for generating the BST data structure in the second trial.The evaluation asked Copilot to generate the BST structure and methods for insertion, deletion, search, minimum and maximum, in-order traversal, and successor lookup.Source: §4.1.2 · pp. 13, 15, (1) Response Received · p. 15

(2) Correct Ratio

  • 94.12% of In-order Tree Walk suggestions were correct in the first BST trial, versus 16.67% in the second; Finding The Successor Node was 100% correct in both.Correct ratios for Min and Max Values in Tree and Finding The Successor Node were also described as very close across trials.Source: Table 2 · p. 14, (2) Correct Ratio · p. 15

(3) Code Optimality … (3) Code Optimality

Copilot often produced correct and optimal code for BST and graph tasks, but its correctness and reproducibility varied by problem and trial. Graph solutions also showed limitations when reusing generated methods and handling non-integer node names.

(4) Code Reproducibility and Similarity

  • BST solution correctness fell from 81.86% in the first trial to 54.76% in the second, while BFS solutions were 100% correct in both trials.For graph problems, first-trial correctness was 50% for Simple Data Structure, 75% for Depth First Search, and 86.37% for Directed Acyclic Data Structure; Copilot generated no correct solutions for these problems in the second trial.Source: (4) Code Reproducibility and Similarity · p. 15, (2) Correct Ratio · p. 16

(3) Code Optimality

(4) Code Reproducibility and Similarity

  • Copilot exactly reproduced correct solutions inconsistently within trials, with similarity no greater than 0.51 for solutions not exactly reproduced; across trials, the exception was BST Finding maximum and minimum values, with similarity 0.83.In the second trial, the Data Structure solution similarity was 0.46; in the first trial, Data Structure and Finding successor nodes had similarities of 0.51 and 0.33, respectively.Source: (4) Code Reproducibility and Similarity · pp. 15–16

4.1.3. Elementary Graph Algorithms

  • As graph algorithms became more complex, the authors noted that Copilot needed to use previously generated code; the two authors’ Kappa agreement on metrics requiring manual inspection was 83%.Source: §4.1.3 · p. 16

(1) Response Received

  • Copilot generated graph code in all three first-trial attempts for every problem except Simple Data Structure and Directed Acyclic Data Structure, and struggled in only one second-trial attempt on Simple Data Structure.Source: (1) Response Received · p. 16

(2) Correct Ratio

  • Graph attempts for BFS and DFS produced code for both algorithms regardless of whether only one was requested, and a node-name assumption caused a syntax error when names were non-integers.Because successor methods reused earlier methods, the bug propagated through generated code; the authors still labeled the code correct when it worked under normal usage.Source: (2) Correct Ratio · p. 16

(3) Code Optimality

  • In the first trial, Copilot generated one optimal solution for each graph problem, but in the second trial only BFS had an optimal solution among the two problems addressed correctly.Graph-cycle checks were considered non-optimal when they repeated BFS or DFS operations rather than using previously generated code.Source: (3) Code Optimality · p. 16
  • For graph construction, list comprehensions were considered optimal over explicit loops for large inputs, and some generated code used operator overloading to rewrite native Python functions.Source: (3) Code Optimality · p. 16

(4) Code Reproducibility and Similarity … (3) Code Optimality

Copilot generated solutions for elementary graph algorithms but behaved inconsistently across trials and often failed to reproduce correct code. On activity selection, it could respond to full problem definitions, yet generated code was often buggy or incomplete.

(4) Code Reproducibility and Similarity

  • Copilot reproduced a correct solution in the first trial for only two graph problems, while first-trial solutions for “Depth First Search” and “Directed Acyclic Data Structure” had similarities of 0.73 and 0.63.Across trials, it exactly reproduced a correct solution only for BFS; “Finding Reachable Vertices” had similarity 0.076.Source: (4) Code Reproducibility and Similarity · p. 16
  • 74.27% was the average correct ratio for graph problems in the first trial; in the second, Copilot generated correct solutions for only two problems, and all correct solutions were correct.It generated optimal code for every problem it addressed correctly in both trials except “Finding Reachable Vertices” in the second trial.Source: (4) Code Reproducibility and Similarity · p. 17

4.1.4. Greedy Algorithms

  • The activity-selection task asked Copilot to define activities with start and end times and select the maximum number that do not overlap.The evaluation examined whether generated code implemented the activity class, comparisons, and overlap checks.Source: §4.1.4 · p. 17

(1) Response Received

  • Copilot could generate code for activity selection even when given the entire problem definition in one prompt.The task involved defining activities, comparing them, and adding non-overlapping activities to a set.Source: (1) Response Received · p. 17

(2) Correct Ratio

  • For activity selection, Copilot’s generated code was buggy or incorrect: it omitted input type and boundary checks, and could not generate the method for adding a non-overlapping activity to a set.It implemented the comparison description but did not check that inputs were activity instances.Source: (2) Correct Ratio · p. 17

(3) Code Optimality

  • Because most activity-selection solutions were incorrect, optimality analysis was limited to comparing activities and adding activities to a set, whose solutions were simple.The assessed operations involved checking class-attribute boundaries or whether a function returned true.Source: (3) Code Optimality · p. 17

(4) Code Reproducibility and Similarity

Copilot’s suggestions had lower correctness and diversity than students’ submissions, but its buggy solutions were cheaper to repair. Reproducibility and task-description requirements remained challenges.

(4) Code Reproducibility and Similarity

  • Copilot reproduced solutions across trials only for “Adding activities to a set”; for “Comparing Activities,” it repeated solutions within a trial but failed to produce a correct solution in the second trial.Copilot also struggled with type checking and variable-boundary checking, even when the prompt explicitly required them.Source: (4) Code Reproducibility and Similarity · p. 17

4.2.1. Correct ratio of Copilot’s suggestions and students’ submissions

  • Across all questions, students’ submission CR exceeded Copilot’s CR@Top10, which counts correct solutions among 50 suggestions per task; on some questions, Copilot’s CR@Top1 and CR@Top5 exceeded students’ CR.Average CR was 0.32 in attempt 1 and 0.44 in attempt 5, though this increase did not occur for every question.Source: §4.2.1 · p. 18

4.2.2. Repairing costs of Buggy solutions generated by Copilot and students

  • Copilot’s average repair rate was 0.95 versus students’ 0.89; average repair time was 4.94 seconds versus 6.48 seconds, with average RPS of 0.33 versus 0.35.For q4: Sorting Tuples and q5: Top-k Elements, all Copilot buggy solutions were fixed (100%), while students’ submissions had a repair rate of 85%.Source: §4.2.2 · p. 19

4.2.4. The Cyclomatic Complexity of Codes

  • Copilot’s correct solutions were generally less complex than students’, except for “q1”; it had no correct “q2” solutions and only an 8% correct ratio for “q4.”For “q5,” Copilot used the prohibited built-ins “Sort” and “Sorted.”Source: Table 6 · p. 21, §4.2.4 · p. 21

4.2.3. Diversity of Copilot’s suggestions and students’ submissions

  • Students’ correct and buggy submissions were more diverse; Copilot often repeated solutions, so additional attempts sometimes increased correct-solution counts without adding new correct solutions.For “q3” at attempt 5, Copilot had 28 correct but only 2 non-duplicate correct solutions, while students had 22 correct and 21 non-duplicate correct solutions.Source: §4.2.3 · p. 21, Figure 10 · p. 22

4.2.5. Syntactic Mastery

  • Students used more diverse syntax patterns and built-ins than Copilot, which did not follow the “q5” restriction against using “sort” and “sorted.”For “q3,” Copilot used only “append,” while students used several built-ins, including “count,” “remove,” “index,” “copy,” “reverse,” and “pop.”Source: §4.2.5 · p. 22

4.2. RQ2: Copilot vs. Human in Solving Programming Problems

  • Students’ submissions had higher correct ratios and diversity, while Copilot’s buggy solutions cost less to repair and its code was less complex.Copilot also struggled to understand some task requirements that students followed.Source: §4.2.5 · p. 22

5. Discussion and Limitation

The discussion finds that Copilot often misses requirements expressed in natural language, while technical wording, decomposing tasks, and providing sample inputs can improve its results. These prompt changes help in some cases but do not resolve every failure.

5.1. Description of Problems (Prompts)

  • Copilot used forbidden sort and sorted functions in q5 and returned q4 tuples in default ascending order, while students accounted for the stated requirements.The q5 prompt prohibited Python’s built-in sorting functions; in q4, students considered the requirement to put older people first, even in some buggy submissions.Source: §5.1 · p. 23
  • Copilot missed explicit variable limits and could misunderstand long or multi-sentence descriptions, a difficulty also observed by [31].The authors also report that Copilot may misunderstand a problem entirely when its description contains multiple sentences.Source: §5.1 · p. 23

5.2. Experimental Suggestions

  • Changing q4’s “older people are at the front” requirement to “descending order” improved CR from 14% to 79%.The authors suggest that replacing natural-language details with programming-specific keywords can help Copilot recommend relevant solutions.Source: §5.2 · p. 23
  • Separate descriptions for q2’s three functions yielded CRs of 88% for unique day, 0% for unique month, and 40% for contains unique day.Decomposition improved unique day and contains unique day, but unique month remained at zero.Source: §5.2 · p. 24
  • Adding a sample unit test raised Copilot’s unique-month CR from 0% to 91%, after its suggestions misread the birthday tuple’s month field.Copilot treated the second tuple item as the month, although the first item was the month used by the tests.Source: §5.2 · p. 24

6. Threats to Validity

The authors identify limits from Copilot’s opacity and changing suggestions, nonrepresentative tasks and participants, and quantitative evaluations that cannot capture human interaction with an AI pair programmer.

6.1. Internal Validity

  • Because Copilot is closed-source, the authors could assess only prompt-based functionality [8], not model behavior or whether outputs were memorized from training data.Suggestions also change over time, so other researchers cannot be assured of reproducing the same results.Source: §6.1 · p. 24

6.2. External Validity

  • Because no industrial-context task-and-code dataset was available, the authors used classical human-written tasks [47] [16] [49], which do not represent all programming in real projects.They assumed Python-course students represented junior developers, while noting they may not represent the whole population.Source: §6.2 · p. 24

6.3. Conclusion Validity

  • The authors used quantitative metrics from prior studies to compare Copilot and human code, but these metrics do not assess how humans interact with the tool.They say the metrics reduce the chance of biased conclusions, but do not enable qualitative assessment.Source: §6.3 · p. 25

6.4. Construct Validity

  • Quantitative metrics cannot capture all capacities of an AI pair programmer, so a comprehensive study also requires humans’ opinions about their tool-use experience.For example, a user may prefer voice commands and discussion to a tool that presents suggested solutions.Source: §6.4 · p. 25

7. Conclusion

Copilot can generate correct, optimal solutions for some fundamental problems, but its usefulness depends on prompt quality and expert oversight. The authors see it as a potential asset for expert-led work and propose ways to reduce risks and broaden evaluation.

7. Conclusion

  • Copilot generates correct, optimal solutions for some fundamental algorithm problems, but quality depends heavily on prompt conciseness and depth.It may fail to satisfy all prompt criteria, though its code can often be incorporated after small or moderate changes to the prompt or code.Source: §7 · p. 25
  • Copilot’s suggestions are more advanced than junior developers’ and comparable to humans’ on correctness, optimality, reproducibility, and repair costs, but require expert filtering.The authors characterize Copilot as an asset when expert developers use it as a pair programmer, but a liability for users unfamiliar with the problem context and correct coding methods.Source: §7 · p. 25
  • Novices who fully trust Copilot’s solutions may encounter trouble, although the authors hypothesize its suggestions could help improve their programming skills.Wider commercial access may enrich Copilot’s training data while also exposing more shortcomings.Source: §7 · p. 25
  • Future work proposes filtering buggy or non-optimal suggestions and testing more diverse tasks with heterogeneous participants in human-centered comparisons.The proposed filter or layer on top of Copilot is intended to reduce risks in software projects.Source: §7 · p. 25

Related papers

Loading 2206.15331v2…