Source-linked AI summary
Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming
Dominik Sobania, Martin Briesch, Franz Rothlauf
TL;DR
Automatic program synthesis may become a standard software-development tool, but GitHub Copilot had not yet been compared with genetic programming on common benchmarks. The paper evaluates Copilot on those benchmarks against results from the GP literature. It finds similar performance on the studied problems, while identifying practical disadvantages for GP-based approaches.
Problem
GitHub Copilot’s program-synthesis performance had not yet been compared with genetic programming on common benchmark problems.
Source: §5 · p. 7Method
Results
GitHub Copilot and GP perform similarly on the studied benchmarks, although GP solves more problems overall and its practical use is constrained by costly training cases and slow solution generation.
Source: §5 · p. 8Takeaways & Limitations
The authors recommend improving GP’s execution time, generated-code readability, training-data requirements, and integration into standard development infrastructure.
Source: §5 · p. 8Current GP-based frameworks may take days to synthesize a solution, limiting their readiness for real-world use.
Source: §4.2 · p. 7
Abstract
from arXiv · showhide
GitHub Copilot, an extension for the Visual Studio Code development environment powered by the large-scale language model Codex, makes automatic program synthesis available for software developers. This model has been extensively studied in the field of deep learning, however, a comparison to genetic programming, which is also known for its performance in automatic program synthesis, has not yet been carried out. In this paper, we evaluate GitHub Copilot on standard program synthesis benchmark problems and compare the achieved results with those from the genetic programming literature. In addition, we discuss the performance of both approaches. We find that the performance of the two approaches on the benchmark problems is quite similar, however, in comparison to GitHub Copilot, the program synthesis approaches based on genetic programming are not yet mature enough to support programmers in practical software development. Genetic programming usually needs a huge amount of expensive hand-labeled training cases and takes too much time to generate solutions. Furthermore, source code generated by genetic programming approaches is often bloated and difficult to understand. For future work on program synthesis with genetic programming, we suggest researchers to focus on improving the execution time, readability, and usability.
1 INTRODUCTION
Automatic program synthesis could become another standard software-development tool, but large language models had not yet been compared with genetic programming on common benchmarks. This work evaluates GitHub Copilot on those benchmarks and compares it with results reported in the GP literature.
1 INTRODUCTION
- Automatic program synthesis generates source code from a specification, such as a natural-language description or input/output examples, and could become a standard development tool.Source: §1 · p. 1
- Copilot suggests complete functions and test cases, while GP evolves programs from a random population using input/output examples to evaluate them.Source: §1 · p. 1
- Common benchmarks had enabled comparisons among GP approaches, but no comparison with large-scale language models had yet been carried out.Source: §1 · p. 1
- The paper compares GitHub Copilot’s benchmark performance with results reported for genetic-programming approaches.Source: §1 · p. 1
- The evaluation supplies each Copilot problem with a function signature and textual description, checks up to ten suggestions, and draws GP results from benchmark-specific literature sources.Source: §1 · p. 1
2 RELATED WORK
The related work surveys genetic-programming approaches to program synthesis and introduces large-scale language models, including the Transformer foundations relevant to Copilot.
2.1 GP-Based Program Synthesis
- PSB1 and PSB2 established common problems and training/test examples for comparison, with stack-based, grammar-guided, and linear GP among the resulting method categories.Source: §2.1 · p. 2
- PushGP uses typed stacks, and mutation and selection changes improved its synthesis success; grammar-guided GP can represent languages such as Python, while other work adds problem descriptions or mined code.Source: §2.1 · p. 2
- Linear GP operates on data registers, with tag-based memory proposed to make program representation more stable under variation.Source: §2.1 · p. 2
2.2 Large-Scale Language Models
- Language models can use natural-language specifications to sample programs; Codex was pretrained on code and fine-tuned on programming functions, and Copilot is its production version.Source: §2.2 · pp. 2–3
- Transformers model sequence dependencies with attention rather than recurrence, enabling more efficient training on longer sequences; their encoder-decoder blocks process inputs and preceding outputs.Source: §2.2 · pp. 2–3
- Attention compares a position’s query with other sequence elements, and Transformer layers use scaled dot-product attention with learned query, key, and value projections.Source: §2.2 · pp. 2–3
3 PERFORMANCE COMPARISON
The study evaluates GitHub Copilot on PSB1 and PSB2 problems using textual descriptions, then compares its suggestions with GP results from the literature. Across both suites, the reported performance is similar, with differences in which problems each approach solves and in the evaluation criteria.
3.1 Methodology
- Copilot is tested with each benchmark's textual description and function signature, and a problem counts as solved if its first suggestion or one of ten alternatives is correct.The GP comparisons use results from the literature rather than new experiments.Source: §3.1 · p. 3
3.2 Results
- 24 PSB1 problems were solved by Copilot versus up to 25 by stack-based and grammar-guided GP; linear GP reports 7, with lower literature coverage.The authors caution that fewer papers studied linear GP, so its lower reported count does not establish generally worse performance.Source: Table 1 · p. 4, §3.2 · p. 5
- The comparison distinguishes a correct first Copilot suggestion from a correct alternative; GP success can be reported from a successful run in the literature.For PSB1, the review cited by the authors found 54 relevant papers, with different numbers studying stack-based, grammar-guided, and linear GP.Source: §3.2 · pp. 3–4, Table 1 · p. 4
- For five of PSB1's 29 problems, Copilot found no solution; standard GP evaluations often use 100 runs and count a problem solved if any run succeeds.The authors note that this gives GP approaches a higher chance of finding a solution.Source: §3.2 · p. 5
- Some near-correct first suggestions return a result where PSB1 requires printing, while an alternative can print the Checksum result in the correct format.The authors describe replacing the return statement with print() as an easy adjustment for a programmer.Source: §3.2 · p. 5, Figure 3 · p. 5, Figure 4 · p. 5
- 15 PSB2 problems were solved by Copilot versus 17 by GP, with similar distributions; GP's six exclusive successes each occurred in fewer than 10% of runs.Copilot uniquely solved Luhn, Shopping List, Spin Words, and Vector Distance; GP uniquely solved Bouncing Balls, Camel Case, Dice Game, Indices of Substring, Snow Day, and Solve Boolean.Source: Table 2 · p. 5, §3.2 · p. 6
4 DISCUSSION
Copilot and GP perform similarly on benchmarks, but their different specification inputs, response times, and code-generation characteristics shape their practical trade-offs. The discussion examines Copilot’s suggestions and cautions that its outputs can depend on description specificity and may introduce security risks.
4.1 Analyzing Suggested Solutions
- Copilot and GP can both generalize poorly on some program-synthesis benchmarks, despite the need to handle previously unseen inputs.The discussion cites prior work on low generalization in language models and GP-based approaches.Source: §4.1 · p. 6
- Coin Sums suggestions may reflect prior exposure: a partial prompt elicited a continuation, while the complete PSB2 description produced a correct solution.The authors note that the benchmark problem may have appeared in Copilot’s training data in the same form.Source: §4.1 · p. 6, Figure 5 · p. 6, Figure 6 · p. 6
- Copilot often offered useful suggestions for varied tasks, including relevant libraries and function calls even when its code was not fully correct.For a changed Checksum description, it implemented the specified relationships in code and remained readable, though GP may produce bloated code.Source: §4.1 · pp. 6–7
- Copilot’s performance on PSB2 problems depends on adequate textual descriptions, and a more specific Solve Boolean description yielded a correct solution.The added description explained each operator and terminal; the original description did not produce a correct solution in the experiments.Source: §4.1 · p. 7
- Copilot-generated code can use unchecked eval(), posing potential security hazards in production; GP grammars can exclude eval() or exec().The discussion links the risk to publicly available training code that may contain insecure source code.Source: §4.1 · p. 7
4.2 Different Approaches, Different Strengths
- Copilot and GP achieve similar benchmark results, but GP can solve some problems Copilot cannot when descriptions are insufficient.The GP results aggregate findings from different papers, and a programmer choosing one approach may see lower benchmark performance.Source: §4.2 · p. 7
- Text descriptions are easier to provide than many hand-created input/output examples, while examples make edge cases easier to specify.When descriptions are difficult and many examples are available, the authors identify GP as the better alternative.Source: §4.2 · p. 7
- Copilot suggests solutions in seconds, whereas current GP frameworks may take days and repeat training for each new input/output sample.The authors say GP is not yet ready for real-world use, though hardware acceleration could reduce execution time.Source: §4.2 · p. 7
5 CONCLUSIONS
GitHub Copilot and genetic programming perform similarly on the studied benchmarks, though GP can solve more problems at practical cost. The authors recommend improving GP's speed, code readability, training-data requirements, and integration into standard development infrastructure.
5 CONCLUSIONS
- GitHub Copilot and GP perform similarly on the studied benchmarks, although GP can solve more problems overall.Source: §5 · p. 8
- GP usually requires many expensive hand-labeled training cases and takes too long to generate solutions for practical use.Source: §5 · p. 8
- Copilot suggestions are usually human-readable, whereas GP-generated code is often bloated and difficult to understand.Source: §5 · p. 8
- Copilot is a black box with unknown training data and code that could be malicious, biased, or insecure; GP allows control over training data and evolutionary choices.Source: §5 · p. 8
- The authors recommend improving GP's execution time, generated-code readability, and training-data requirements, and integrating it into standard development infrastructure.Source: §5 · p. 8