PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models and Simple, Stupid Bugs

Kevin Jesse, Toufique Ahmed, Premkumar T. Devanbu, Emily Morgan

arXiv:2303.11455v1cs.SEcs.CLcs.LG

TL;DR

The paper asks whether code-completion models reproduce single-statement bugs, a category not previously evaluated in Codex studies. It evaluates Codex and related models on known bugs and fixes, and tests comment-based prompting. The models avoid some SStuBs but also reproduce known bugs at a high rate, while comments appear to reduce their production.

  • Problem

    Codex had been evaluated for other tasks but not for SStuBs, leaving its propensity to produce this bug category unexamined.

    Source: §I · p. 1

  • Method

    The study evaluates Codex, PolyCoder, and CodeGen on ManySStuBs4J and tests whether adding automatically generated comments to prompts changes their completions.

    Source: §I · p. 2, A. Experimental Setup · p. 4

  • Results

    Codex and PolyCoder produce nearly twice as many known SStuBs as fixes, while Codex Davinci patches about 30% (5,149) of SStuBs in the best case.

    Source: §VII · p. 11

  • Takeaways & Limitations

    Commenting practices, including automatically generated comments, appear to help reduce SStuBs and improve the bug-to-patch ratio in this study.

    Source: §I · p. 2, §VII · p. 11

Abstract

from arXiv · show

With the advent of powerful neural language models, AI-based systems to assist developers in coding tasks are becoming widely available; Copilot is one such system. Copilot uses Codex, a large language model (LLM), to complete code conditioned on a preceding "prompt". Codex, however, is trained on public GitHub repositories, viz., on code that may include bugs and vulnerabilities. Previous studies [1], [2] show Codex reproduces vulnerabilities seen in training. In this study, we examine how prone Codex is to generate an interesting bug category, single statement bugs, commonly referred to as simple, stupid bugs or SStuBs in the MSR community. We find that Codex and similar LLMs do help avoid some SStuBs, but do produce known, verbatim SStuBs as much as 2x as likely than known, verbatim correct code. We explore the consequences of the Codex generated SStuBs and propose avoidance strategies that suggest the possibility of reducing the production of known, verbatim SStubs, and increase the possibility of producing known, verbatim fixes.

I. INTRODUCTION

The paper examines whether Codex and related code-completion models generate single-statement bugs, a category not covered by prior Codex evaluations, and tests whether comments can reduce them. It reports that models avoid some SStuBs but also reproduce known bugs, motivating attention to both their risks and possible prompting strategies.

I. INTRODUCTION

  • SStuBs can be difficult to locate and persist in code, while developers may trust generated code and struggle to understand or repair its bugs.The paper notes that generated code can lead developers into time-consuming debugging.Source: §I · p. 1
  • The study uses the ManySStuBs4J dataset and asks about bug production, historical repair time, model confidence, and the effects of comments and other prompt improvements.The research questions also examine how bug-derived comments affect completions.Source: §I · pp. 1–2
  • Codex and related models had not been evaluated against SStuBs, so the study tests their production, consequences, and whether generated comments can help avoid them.The authors also aim to produce commented code.Source: §I · pp. 1–2
  • LLMs avoid some SStuBs, yet Codex and other models produce known SStuBs at nearly twice the rate of fixes, and Codex-generated bugs are associated with longer resolution times.The paper also reports that buggy completions appear as natural as correct statements; automatically generated comments appear to reduce known SStuBs for most models.Source: §I · p. 2

II. RELATED WORK

The related-work discussion situates SStuBs as prevalent, often difficult to locate, and potentially costly to repair, then reviews work on their detection, repair, and relationship to code changes. The study focuses on established Java SStuB patterns.

A. Simple, Stupid Bugs

  • SStuBs are single-statement bugs with small template-matching fixes; they account for 33% of single-statement bugs found in 1,000 Maven projects.Prior work reports that static analysis did not detect them and estimates one SStuB per 1,600 lines of code.Source: A. Simple, Stupid Bugs · p. 2
  • Only 40% of SStuBs were fixed by their original author, while fixes by another author took a median 136 days to find and complete.Other work likewise found that bugs from other developers take longer to fix, and Codex-generated bugs may resemble this case.Source: A. Simple, Stupid Bugs · p. 2
  • Prior work links SStuBs to change clones and reports that 29% of change clones introduced bugs matching 16 SStuB patterns.Because Codex tends to repeat code it has seen, the authors suggest it could generate SStuBs in multiple locations, increasing repair effort.Source: A. Simple, Stupid Bugs · p. 2
  • Prior studies examine SStuB detection and repair, including CodeBERT fixing 19% of the deduplicated ManySStuBs4J dataset without requiring bug-location assumptions.The study focuses on established Java patterns; the authors plan to expand to other languages and patterns when resources permit.Source: A. Simple, Stupid Bugs · pp. 2–3

B. Examining Codex Completions

Prior Codex and Copilot studies describe developer over-reliance and difficulty assessing or repairing generated code, while evaluations report that correctness varies with language and additional input. This paper explores comments as a prior-free way to guide completions toward fewer SStuBs.

B. Examining Codex Completions

  • In a study of Codex use, participants often failed to assess generated code, underestimated repair effort, and used varied prompts that sometimes produced undesired completions.Most participants preferred Codex to IntelliSense and used it as a starting point instead of searching online.Source: B. Examining Codex Completions · p. 3
  • Copilot evaluations report that correctness can improve with developer input and varies by programming language, while prior Codex work spans repair, security, synthesis, and other tasks.Prior studies also used prompt engineering to elicit desired responses from Codex.Source: B. Examining Codex Completions · p. 3
  • This study tests traditional code comments as prior-free prompts and finds that good commenting practices can guide Codex toward completions with fewer SStuBs.Figure 1 identifies the prompt, candidate completion, and code following the SStuB.Source: B. Examining Codex Completions · p. 3, Figure 1 · p. 3

III. METHODOLOGY

The study evaluates Codex, PolyCoder, and CodeGen on the ManySStuBs4J dataset, matching model completions to known bugs or fixes and manually checking samples that match neither. The setup uses identical pre-bug code as prompts and tests completions with and without automatically generated comments.

A. Experimental Setup

  • The evaluation uses 63,923 SStuBs mined from 1,000 repositories, then removes cases with other changes and deduplicates examples sharing the same prefix, bug, and fix.The filtered set contained 34,595 bugs before deduplication.Source: A. Experimental Setup · pp. 3–4
  • Figure 3 reports Codex Davinci match rates and shows that completions matching neither a patch nor an SStuB are validated by hand.The study also excludes costly models and models it could not run locally, following the stated resource constraints.Source: Figure 3 · p. 4, A. Experimental Setup · p. 4
  • Codex, PolyCoder, and CodeGen provide model-family comparisons, but the authors caution that behavior in other similarly trained LLMs remains uncertain.PolyCoder and CodeGen are included to assess whether findings generalize beyond Codex.Source: A. Experimental Setup · pp. 4–5
  • Completions are matched to known bugs or fixes using substring matching that ignores whitespace, then a random sample of 401 is manually checked for semantic matches.The survey checks completions that automatic evaluation cannot capture, including equivalent constants and literals.Source: Figure 2 · p. 4, A. Experimental Setup · p. 5

Prompting LLMs with Comments

Prompting with comments is explored as a way to guide code generation without requiring prior knowledge of a bug. Comments generated from buggy or fixed code improve Codex completions, with the first added comment producing the greatest improvement.

Prompting LLMs with Comments

  • The first added comment produces the greatest improvement in generated code.The figure adds neural-generated comments step by step in the prompt before the SStuB.Source: Figure 4 · p. 5
  • Hard prompting uses a fixed-language prompt and requires prior knowledge about the input, such as knowing that it is buggy.The study contrasts this requirement with traditional commenting practices.Source: Prompting LLMs with Comments · p. 5
  • The study incrementally adds automatically generated comments around the SStuB, using CodeTrans comments conditioned on two statements plus the buggy or fixed line.The comments are placed to resemble developer-written comments.Source: Prompting LLMs with Comments · p. 5
  • Comments generated from fixed code describe correct logical steps, while those from buggy code describe mostly correct steps with a minor single-statement mistake.Both comment types are tested to compare their influence on SStuB avoidance or generation.Source: Prompting LLMs with Comments · pp. 5–6

Prompt input, length, and SStuB completion

The prompt uses code preceding the SStuB as model input, with comments and completion length constrained by model token windows. Generated completions are checked against the bug or fix using exact matching after whitespace is ignored.

Prompt input, length, and SStuB completion

  • Code preceding the SStuB is the conditional input, and the same prompt is used for all 16,899 SStuBs to avoid bias toward either the bug or fix.Generated comments are placed above their source lines in a natural tabulated style.Source: Prompt input, length, and SStuB… · p. 6
  • The prompt includes code before the bug up to each model’s input limit; comments reduce the remaining context available under fixed token windows.Davinci has an 8,000-token window, while Cushman, PolyCoder, and CodeGen have 2,048-token windows.Source: Prompt input, length, and SStuB… · p. 6
  • The completion length further reduces available prompt context because generation is autoregressive.The models are given code before the SStuB as their prompt when available.Source: Prompt input, length, and SStuB… · p. 6
  • Models generate 64 tokens rather than stopping at a newline, which could leave unmatched completions because models often add arbitrary newlines and whitespace.SStuBs average 29 tokens and have a median length of 25 tokens.Source: Prompt input, length, and SStuB… · p. 6
  • Generated sequences are compared with the SStuB after ignoring whitespace, and must match it completely to count as a bug or fix.Source: Prompt input, length, and SStuB… · p. 6
  • The results cover how often models produce SStuBs, the commits needed to fix generated SStuBs, and the effect of comment annotations.Source: Prompt input, length, and SStuB… · p. 6

A. SStuB production in LLMs (RQ1)

Codex and the other studied models produce nearly twice as many known SStuBs as patches, although Codex avoids some SStuBs. Codex-generated bugs are associated with longer times to fix, measured in commits.

A. SStuB production in LLMs (RQ1)

  • Codex, PolyCoder, and CodeGen 350M produce nearly 2x as many bugs as patches, while Codex avoids 13.41% of dataset SStuBs.The bug/patch ratio is used to compare models because total successful completions vary; Table I covers SStuB production rates on off-the-shelf LLMs.Source: Table I · p. 6, A. SStuB production in LLMs (RQ1) · p. 6

B. Number of commits to fix LLM produced SStuBs (RQ2)

C. SStuB regularity (RQ3)

Codex rates SStuB bugs and patches as similarly natural, making detection difficult. Adding comments improves bug/patch ratios across models, with larger gains for Codex than simply increasing model size.

C. SStuB regularity (RQ3)

  • Codex rates SStuB bugs and patches similarly natural, while unmatched completions have higher negative log-likelihood, making SStuB detection difficult.All pairwise differences shown in Figure 6 are statistically significant at p ≤ 0.0001.Source: C. SStuB regularity (RQ3) · p. 7, Figure 6 · p. 7

D. Avoiding SStuBs (RQ4)

  • Commenting improves the bug/patch ratio across models; Codex Cushman and Davinci generate 20% more bugs but 127% more patches, cutting the ratio almost in half.The study reports about 3,000 more patches with comments (5,149 vs. 2,267).Source: D. Avoiding SStuBs (RQ4) · pp. 7–8
  • With comments, all models perform better on match rate and bug/patch ratio, and Codex performs much better than the next-best model, CodeGen 16B.Without comments, Codex Cushman and Davinci were not competitive with other off-the-shelf models.Source: D. Avoiding SStuBs (RQ4) · p. 8, Figure 7 · p. 8
  • A 160M-parameter PolyCoder with a comment improves the bug/patch ratio by 14% over Codex Cushman 12B and Davinci 175B without comments.The study reports that adding a comment helps more than increasing parameter counts.Source: D. Avoiding SStuBs (RQ4) · p. 8, Figure 8 · p. 8
  • The largest bug/patch-ratio improvements with sufficient samples occur for LESS_SPECIFIC_IF, MORE_SPECIFIC_IF, OVERLOAD_METHOD_MORE_ARGS, and DIFFERENT_METHOD_SAME_ARGS.CHANGE_IDENTIFIER and DIFFERENT_METHOD_SAME_ARGS gain the most patches; SWAP_ARGUMENTS worsens, but has only four examples.Source: D. Avoiding SStuBs (RQ4) · p. 8

E. Commenting vs Traditional Prompting (RQ4 cont.)

Traditional hint and bug/fix prompts improve Codex's bug/patch ratio and match rate, but not as much as natural-language comments. Buggy comments still improve Codex's ratio, though lower-capacity models can be misled.

E. Commenting vs Traditional Prompting (RQ4 cont.)

  • Traditional prompting uses hints or SStuBs with fixes, both of which require prior knowledge about the snippet to be generated.The hint identifies a SStuB location; the bug/fix prompt supplies the SStuB and how to fix it.Source: E. Commenting vs Traditional Prompting… · p. 8
  • Traditional hint and bug/fix prompts improve bug/patch ratio and match rate, but perform worse than natural-language comments and greatly increase bugs.These prompts require prior knowledge of the code location or the bug and its fix.Source: E. Commenting vs Traditional Prompting… · p. 8, Table V · p. 9
  • The comparison evaluates Codex Davinci across prompting techniques.Table V is identified as the performance comparison across techniques; the supplied passage does not give its cell values.Source: Table V · p. 9

F. What if we insert a ‘buggy’ comment? (RQ5)

V. DISCUSSION

Codex and other LLMs can avoid some SStuBs, but direct use can produce more known bugs than patches. The discussion points to comments as a practical aid while noting that their placement and density need further study.

Implications of Codex Producing SStuBs

  • In the dataset, Codex avoids 2,267 bugs, or 13.41%, yet directly used models can produce up to twice as many SStuBs as patches.The paper also reports that fixing Codex-produced SStuBs takes more than twice as long as fixing avoided SStuBs.Source: Implications of Codex Producing SStuBs · pp. 9–10
  • The authors attribute Codex's reproduction of SStuBs in part to bugs persisting in training data, since SStuBs can last long and evade CI and static analysis.They note that removing SStuBs from training data drawn from 54 million repositories would be challenging.Source: Implications of Codex Producing SStuBs · p. 10

Avoiding SStuBs with LLM

  • Automatically generated high-level comments may help avoid SStuBs by guiding Codex toward a smaller search space of non-buggy solutions.The comments studied describe code at a high level rather than giving deep technical commentary on computed values or algorithmic mechanisms.Source: Avoiding SStuBs with LLM · p. 10
  • Comment placement was uniform in the experiments, and the best placement and adequate, non-excessive comment density remain open questions.The authors identify automatic placement methods as available for future investigation.Source: Avoiding SStuBs with LLM · p. 10

Maintaining AI Generated Code

The discussion connects comments with both SStuB avoidance and code maintainability, while identifying evaluation and training-data uncertainties that qualify the findings.

Maintaining AI Generated Code

  • Comments can help avoid SStuBs in addition to improving readability; readable, understandable code is described as more maintainable.The authors note that SStuBs are usually fixed by someone other than the inserting developer, and an AI assistant is always that “other developer.”Source: Maintaining AI Generated Code · p. 10
  • The authors propose further study of comment-generation models for assistant-generated comments, consistency checks, training-data quality, and applications such as program repair and defect prediction.These are presented as future research directions following preliminary results.Source: Maintaining AI Generated Code · p. 10

A. Internal Validity

  • The study assumes dataset samples are mostly actual bugs, though some non-bug-fixing commits may remain after refactoring changes are removed.The supplied passage states this as an assumption about the dataset.Source: A. Internal Validity · p. 10

Manual Inspection

  • Automated matching may miss semantically equivalent but syntactically different bugs or patches, potentially hiding their true counts; sampled unmatched completions were mostly inappropriate code.Independent raters found over 80% inappropriate completions in the no-comment sample and about 70% in the non-buggy-comment sample; they also found comments substantially increased patches.Source: Manual Inspection · pp. 10–11

Data Leakage from LLM Training

  • The authors cannot verify whether ManySStuBs4J was excluded from model training, whose data are unpublished.All studied SStuBs were fixed by February 2019, before the earliest stated training-data collection dates, but the authors still report models producing twice as many SStuBs.Source: Data Leakage from LLM Training · p. 11

Reproductions of Generations

Codex and other large language models can avoid some human-produced SStuBs, but they also generate SStuBs that may take longer to resolve. High-level functionality comments improved the bug/patch ratio, while the findings are bounded by dataset scope and model scale.

Reproductions of Generations

  • The generation study sampled the top-1 completion for each ManySStuBs4J sample across Codex, PolyCoder, and CodeGen, with and without comments.The authors note that generation is nondeterministic depending on hyper-parameters.Source: Reproductions of Generations · p. 11

B. External Validity

  • The findings may not generalize beyond Java single-statement bugs or to less simple bugs, and the dataset size reflected compute and Codex API constraints.The authors also cite a 20-requests-per-minute API rate ceiling and the time required to run CodeGen 16B locally.Source: B. External Validity · p. 11

Models at Scale

  • Results may vary with the next generation of language models as models continue to grow.Source: Models at Scale · p. 11

VII. CONCLUSION

  • Codex and other large language models help avoid human-produced SStuBs, yet Codex and PolyCoder produce nearly twice as many SStuBs as fixes, and generated SStuBs take longer to resolve.The authors describe the overall findings as somewhat promising, while noting that LLMs may help avoid at least some SStuBs.Source: §VII · p. 11
  • As many as 2,300 SStuBs could have been avoided if developers had used code completion instead of writing them.Source: §VII · p. 11
  • Comments describing high-level functionality improved Codex’s bug/patch ratio, and the authors found that comments may not need to be strictly correct to help avoid SStuBs.They suggest that good commenting practices, including automatic ones, can help other developers and Codex.Source: §VII · p. 11
Loading 2303.11455v1…