PaperlayerView exact revision on arXiv

Source-linked AI summary

From Copilot to Pilot: Towards AI Supported Software Development

Rohith Pudari, Neil A. Ernst

arXiv:2303.04142v1cs.SEcs.AIcs.LG

TL;DR

AI code completion has advanced, but software engineering also demands idioms, avoidance of code smells, and design decisions. This paper studies Copilot’s suggestions and proposes an abstraction taxonomy; it finds that syntax-level support is more advanced than support for idioms, smells, and design rules.

  • Problem

    Code-generation progress does not resolve the broader software-engineering challenge of following idioms, avoiding code smells, and making design decisions, especially where rules are harder to specify.

    Source: §I · p. 1

  • Method

    The paper explores Copilot’s code suggestions for idioms and code smells and introduces an abstraction taxonomy from basic programming functionality to architecture analysis and design.

    Source: §I · p. 2

  • Results

    The authors report that syntax and warnings are closer to useful AI support, while code smells, language idioms, and design rules remain far from solvable.

    Source: §IX · p. 11

  • Takeaways & Limitations

    The authors conclude that AI-supported software development for more complex engineering tasks is possible, while current support remains stronger at coding syntax and warnings than at abstract concerns.

    Source: §IX · p. 11

    Copilot’s single-file input prevents evaluating design suggestions that depend on multiple files, such as recognizing an MVC project structure.

    Source: §VI · p. 9

Abstract

from arXiv · show

AI-supported programming has arrived, as shown by the introduction and successes of large language models for code, such as Copilot/Codex (Github/OpenAI) and AlphaCode (DeepMind). Above human average performance on programming challenges is now possible. However, software engineering is much more than solving programming contests. Moving beyond code completion to AI-supported software engineering will require an AI system that can, among other things, understand how to avoid code smells, to follow language idioms, and eventually (maybe!) propose rational software designs. In this study, we explore the current limitations of AI-supported code completion tools like Copilot and offer a simple taxonomy for understanding the classification of AI-supported code completion tools in this space. We first perform an exploratory study on Copilot's code suggestions for language idioms and code smells. Copilot does not follow language idioms and avoid code smells in most of our test scenarios. We then conduct additional investigation to determine the current boundaries of AI-supported code completion tools like Copilot by introducing a taxonomy of software abstraction hierarchies where 'basic programming functionality' such as code compilation and syntax checking is at the least abstract level, software architecture analysis and design are at the most abstract level. We conclude by providing a discussion on challenges for future development of AI-supported code completion tools to reach the design level of abstraction in our taxonomy.

I. INTRODUCTION

AI code completion may improve programming productivity, but software development also requires idioms, best practices, and design decisions that are harder to specify than linting rules. The study explores Copilot’s handling of idioms and code smells and introduces an abstraction taxonomy to clarify its current boundaries.

I. INTRODUCTION

  • Programming and code completion tools are presented as potential ways to improve developer productivity and accessibility, while enabling more effective code modeling could change how developers interact with completion tools.The introduction describes code completion as predicting intended code and offering suggestions.Source: §I · p. 1
  • Software development involves best practices, code smells, idioms, and design choices; rules for identifying idioms and code smells can be harder to specify than rules used by smell detectors or linters.The paper frames this as a challenge beyond generating code and asks where users should remain vigilant about Copilot’s suggestions.Source: §I · p. 1
  • Choosing how to proceed depends on case-specific conditions, limits, and conventions, and learning idiomatic approaches takes time and experience.The paper argues that clearer boundaries help users judge where to rely on code-completion suggestions and where to be vigilant.Source: §I · pp. 1–2
  • The study investigates Copilot’s Pythonic idiom suggestions and JavaScript best-practice suggestions, then introduces a taxonomy spanning basic programming functionality to architecture analysis and design.The research questions ask how well Copilot manages idioms and suggests non-smelly code, and what boundaries apply to current tools.Source: §I · p. 2

II. BACKGROUND

The paper situates Copilot among code-generation tools and focuses on its comment-to-code feature. Its study samples Python idioms and JavaScript best practices, uses minimally constrained prompts, and compares suggestions with recommended approaches.

II. BACKGROUND

  • Copilot and AlphaCode exemplify code-completion systems that can solve programming-contest problems, while prior work examined how developers use Copilot and how code models might add architecture tactics.The cited prior exploratory study found that Copilot saved search time and helped users get started, though it did not make tasks faster.Source: §II · p. 2

III. STUDY DESIGN

  • The study evaluates Copilot’s suggestions against sampled Pythonic idioms and JavaScript best practices, after describing its scenario construction and evaluation approach.It focuses on converting comments to code; the triggering input combines a natural-language comment, function name, and parameters.Source: §III · p. 2, §II · p. 2

A. Sampling approach

  • The sample includes 25 popular Python idioms and 25 JavaScript practices drawn from the Airbnb style guide, emphasizing broadly accepted practices rather than project-specific styling.The JavaScript sample favors practices closer to design and performance or efficiency than conventions such as trailing commas; popular Python idioms were selected to give Copilot a strong chance of suggesting them.Source: A. Sampling approach · pp. 2–3

B. Input to Copilot

  • Scenario prompts pair a title comment with minimal code that leaves Copilot room to choose whether to follow the desired practice.Because Copilot cannot override or update the input, the researchers restricted prompts to avoid forcing a particular solution; the style also represents a novice who may not know the idiom or best practice.Source: B. Input to Copilot · p. 3

C. Evaluation

The evaluation counts a scenario as a pass when Copilot’s first suggestion matches the recommended practice, while also recording whether the practice appears among its ten viewable suggestions. In the Python idiom sample, first suggestions rarely matched, and some idioms were absent even from the top ten.

C. Evaluation

  • A scenario passed only when the first suggestion matched the recommended idiom or best practice; the researchers separately noted whether it appeared in any of ten viewable suggestions.They used the first suggestion for pass/fail because searching through alternatives would undermine the assumed productivity benefit of code completion.Source: C. Evaluation · p. 3

A. Pythonic Idioms

  • 2 of 25 tested idioms appeared as Copilot’s first suggestion; 8 more appeared within the top ten, while 15 were absent from the top ten.The paper reports that the first suggestion was not the idiomatic approach in 92% of tested idioms.Source: A. Pythonic Idioms · p. 3
  • In the list-comprehension example, the paper says Python’s idiom is more concise and readable than Copilot’s top suggestion, and about one second faster on a sample of 100 million.The figure is described as showing the user input, Copilot’s top suggestion, and the idiomatic version.Source: A. Pythonic Idioms · p. 4, Figure 1 · p. 4
  • Copilot sometimes included an idiom among its ten suggestions while ranking a non-idiomatic approach first; the authors suggest ranking may be influenced by training-data frequency.They report successful first suggestions for set comprehension and an if-condition check, described as frequent idioms in open-source code.Source: A. Pythonic Idioms · p. 4
  • Copilot is more likely to include an idiom in its top ten when a task has only a few common ways to perform it.The paper gives Boolean comparison as an example, with two common forms: “if boolean:” and “if boolean == True.”Source: A. Pythonic Idioms · p. 4
  • The authors argue that code-completion tools should recognize idiomatic approaches in public repositories and rank them above more frequent non-idiomatic alternatives.Source: A. Pythonic Idioms · p. 4

B. JavaScript Code Smells

Copilot’s JavaScript best-practice suggestions often missed the recommended approach, and its performance was worse on coding style than on language idioms. The study also notes that project-specific styles were not tested and that Copilot’s closed source limited investigation of the causes.

B. JavaScript Code Smells

  • Copilot’s top suggestion matched the AirBNB best practice in 3 of 25 scenarios; among the remaining 22, only 5 had it in the top 10, leaving 17 without it.The authors observed that top-10 matches were more likely for common beginner tasks, while Copilot sometimes produced fewer than ten suggestions.Source: B. JavaScript Code Smells · pp. 4–5
  • Copilot did not put the recommended JavaScript best practice first in 88% of the 25 practices tested.The authors could not determine the reason because Copilot is closed source.Source: B. JavaScript Code Smells · p. 4
  • The study did not test project-specific coding styles because Copilot could not customize suggestions to match project preferences.The authors also state that Copilot’s closed source prevented them from investigating possible reasons for its behavior or recommending fixes.Source: B. JavaScript Code Smells · pp. 4–5
  • For copying array contents, Copilot suggested a slower element-by-element loop instead of the guide’s spread approach, [..items].The spread approach avoids iterating through the elements to copy them.Source: B. JavaScript Code Smells · pp. 4–5
  • Copilot performed worse on coding style than on language idioms, despite the tested practices coming from a widely accepted style guide.The authors describe this as evidence that current tools are not yet capable of suggesting these best practices reliably.Source: B. JavaScript Code Smells · p. 5

S No. Best Practice Title Top 10

The paper introduces a six-level software-abstraction hierarchy, from basic programming functionality to architecture analysis and design. It places Copilot at the correctness level and describes higher levels as more challenging and requiring greater human oversight.

S No. Best Practice Title Top 10

  • The table lists the tested JavaScript best practices from the AirBNB coding style guide.Its title identifies columns for serial number, best-practice title, and top-10 suggestions.Source: Table II · p. 5

V. TAXONOMY

  • The six-level hierarchy runs from basic programming functionality to software architecture analysis and design, with upper levels depending on lower ones.The authors adapt the hierarchy idea from autonomous-vehicle safety levels.Source: §V · pp. 5–6
  • As abstraction rises, the hierarchy calls for more human oversight; the paper marks levels Copilot handles reasonably well green and challenging levels red.The authors also note that rules for detecting problems are easier to formulate at lower levels.Source: §V · p. 6, Figure 4 · p. 6
  • Copilot’s tested suggestions were syntactically correct and solved the coding tasks, placing it at the taxonomy’s correctness level.The hierarchy’s top levels concern architecture analysis and design, and are important to software quality attributes.Source: §V · pp. 5–6

A. Syntax level

The syntax level concerns code that has no syntax or compilation errors, not whether it correctly completes the programming task. The paper illustrates this distinction with a sorting example.

A. Syntax level

  • At the taxonomy’s lowest abstraction level, suggested code must be syntactically correct and compile without errors, but need not solve the task correctly.The sorting example requires syntactically valid code, not a correct sorting operation.Source: A. Syntax level · p. 6
  • The sorting scenario illustrates syntax-level suggestions, and Figure 5 shows code suggestions for that level.The example’s code is not required to perform the sorting operation correctly.Source: A. Syntax level · p. 6, Figure 5 · p. 7
  • The stated criteria are syntactic correctness and the absence of compilation errors.These criteria define the required capabilities for this abstraction level.Source: A. Syntax level · pp. 6–7

B. Correctness level

The taxonomy moves from correctness—producing a syntactically correct solution that need not be optimal—to paradigms and idioms, which adds common language practices.

B. Correctness level

  • At the correctness level, suggestions must solve the task with syntactically correct code, but need not be optimal or cover every edge case.The target is a decent solution, not necessarily the best one.Source: B. Correctness level · p. 7

C. Paradigms and Idioms level

  • At the paradigms-and-idioms level, suggestions must use common practices and language idioms while satisfying the lower abstraction levels.For the sorting example, Pythonic swapping contrasts with using a temporary variable.Source: C. Paradigms and Idioms level · pp. 7–8, Figure 7 · p. 7

B. Correctness level

  • Figure 6 illustrates the correctness-level sorting suggestion, which performs the task without requiring an optimal sorting method.The example distinguishes a workable solution from an idiomatic one.Source: Figure 6 · p. 7, B. Correctness level · p. 7

D. Code Smells level

The taxonomy’s code-smells level adds avoidance of bad practices and optimization; the design level extends support to file- and project-level decisions.

D. Code Smells level

  • At the code-smells level, suggestions must satisfy lower levels, avoid common bad practices and unresolved edge cases, and provide the most optimized available suggestion.The taxonomy also expects code not to need immediate refactoring.Source: D. Code Smells level · p. 8
  • The code-smells level expects the first suggestion to be the most optimized version, while avoiding smells found in public code.Figure 8 presents a sorting example at this abstraction level.Source: D. Code Smells level · p. 8, Figure 8 · p. 8

E. Design level

  • Design is the highest abstraction level: module-level support involves more user input for file-level choices, while system-level support makes project-level choices with minimal input.System-level suggestions may include design patterns and architectural tactics.Source: E. Design level · p. 8

1) Module level design:

Module-level design addresses file-level choices and implementation safeguards; system-level design extends to project-level decisions with minimal user input.

1) Module level design:

  • Module-level design calls for applicable algorithms, user-specified style, vulnerability checks, functional coverage, tests, and continuous integration when applicable.The goal is to support file-level design choices while meeting task requirements.Source: 1) Module level design · pp. 8–9

2) System level design:

  • System-level design is the taxonomy’s highest abstraction level and aims to support the software development process with minimal user input.This level is described as the most complex to define rules for.Source: 2) System level design · p. 9
  • System-level design calls for project-level patterns and architectural tactics, coverage of nonfunctional requirements, and decisions informed by requirements and explained to the user.The taxonomy also expects tools to identify and adapt to the project’s coding style.Source: 2) System level design · p. 9

VI. DISCUSSION

Automating software design requires collecting and generalizing design knowledge, supporting work across multiple files, and keeping training data current as patterns and practices evolve. These challenges leave current tools far from the design level, although future models may support higher-abstraction work and programming accessibility.

VI. DISCUSSION

  • Software design shapes system performance, maintainability, robustness, and security, while development tasks can require context-specific solutions for multiple stakeholders.The paper characterizes software development as complex and involving tasks that may need inventive modification.Source: §VI · p. 9
  • Design knowledge is scattered across developer communication channels, while Stack Overflow is proposed as a source and semi-automatic collection as a possible approach.Gorton et al. demonstrated semi-automatic design-knowledge collection for a particular big-data domain.Source: §VI · p. 9
  • Copilot cannot infer an MVC project’s structure from a single code block or file, so design-pattern suggestions must span multiple program units.The paper gives direct Model-to-View communication as an MVC suggestion to avoid.Source: §VI · pp. 9–10
  • Design patterns and coding practices evolve, so training data needs regular updates to reflect changes such as Redux and Flux or the shift from callbacks to promises.The paper notes that models trained before new design patterns emerge will lack data about them.Source: §VI · p. 10
  • Current code-completion tools remain far from the design level and require relevant training data, multi-file input, and regular updates.The paper identifies all three as necessary to reflect changes in software development.Source: §VI · p. 10

VII. IMPLICATIONS

  • Future code-generation models may support work at higher abstraction levels, improve programming accessibility, or help train new programmers.The paper also suggests models may recommend more effective or idiomatic implementation methods.Source: §VII · p. 10

A. Implications for practitioners

The paper proposes improving AI-supported development through higher-quality pretraining data, quality checks and user-context adaptation, and suggestions that operate beyond tokens. Researchers also face the challenge of accounting for design reasoning and potential societal drawbacks such as automation bias.

1) Pre-training the LLM:

  • Pretraining should use higher-quality data, potentially curated from verified sources or repositories ranked by code quality and filtered for known flaws and bad practices.The paper notes limited clarity on current curation and evaluation, particularly for non-experts.Source: 1) Pre-training the LLM · p. 10

2) Code completion time:

  • Suggestions could be checked with code-smell or review tools, while active learning could adapt them to a user’s context, such as company coding style.The paper cites CodeWhisperer’s security scan as an example of a project- or file-level check.Source: 2) Code completion time · p. 10

B. Implications for researchers

1) Moving Beyond Tokens:

  • Moving beyond tokens to code blocks or files could support suggestions across program units, including architectural concerns such as keeping MVC’s Model from communicating directly with View.The paper points to chain-of-thought reasoning [32], framework sketches, and scenario-based architectural reasoning as relevant approaches, while noting that human design strategies remain imperfectly understood.Source: 1) Moving Beyond Tokens · pp. 10–11

VIII. THREATS TO VALIDITY

The paper concludes that code syntax and warning-level tasks are approaching useful AI support, while code smells, language idioms, and design rules remain far from solvable. Its findings are bounded by Copilot’s input sensitivity and opacity, and by the authors’ chosen abstraction taxonomy and programming languages.

VIII. THREATS TO VALIDITY

  • The study used Copilot extension version 1.30.6165, but its closed-source Codex model and API-based results complicate versioning and clarity.Source: §VIII · p. 11

A. Internal Validity

  • Copilot’s sensitivity to prompts, variable outputs, and closed-source black-box operation limit replicability and leave open whether different inputs might yield the preferred idiom.The authors archived their examples in a replication package.Source: A. Internal Validity · p. 11

B. Construct Validity

  • The abstraction taxonomy reflects the authors’ view, and results from Python and JavaScript or Copilot may differ for other languages or tools.Source: B. Construct Validity · p. 11
  • The evaluation deliberately used inputs resembling those of less experienced programmers, who the authors argue are likely to use Copilot and lack idiom knowledge.The authors also argue that suggestion instability is itself important to understand.Source: B. Construct Validity · p. 11

IX. CONCLUSION

  • Syntax and warning-level tasks are nearing useful AI support, but code smells, language idioms, and design rules remain far from solvable.The paper concludes that AI support for more complex software engineering tasks is possible.Source: §IX · p. 11
Loading 2303.04142v1…