PaperlayerView exact revision on arXiv

Source-linked AI summary

Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions

Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan

arXiv:2302.07248v3cs.HCcs.AI

TL;DR

AI code completions can introduce bugs or security vulnerabilities, and empirical evidence on uncertainty highlighting and which uncertainty measure best serves programmers has been lacking. This paper compares completion alone with generation-probability and edit-likelihood highlights in a study of programmers. Edit-model highlights were associated with faster completion, more targeted edits, and higher reported utility than generation-probability highlights.

  • Problem

    There had been no empirical studies of uncertainty highlighting for generative code completion, including which notion of uncertainty best fits programmers’ needs.

    Source: §1 · p. 1, §8 · p. 26

  • Method

    The study compared code completion alone with highlights based on generation probability or a model’s predicted likelihood that a programmer would edit a token.

    Source: §1 · p. 2, §8 · p. 26

  • Results

    Edit-model highlighting was associated with faster task completion, more targeted edits, and higher reported utility than generation-probability highlighting.

    Source: §7 · p. 24

  • Takeaways & Limitations

    The findings favor edit models as a way to highlight code regions more aligned with programmers’ likely changes than generation probability.

    Source: §7 · p. 24

    Errors that programmers do not edit may remain unmarked by an edit model.

    Source: §7.1 · p. 25

Abstract

from arXiv · show

Large-scale generative models enabled the development of AI-powered code completion tools to assist programmers in writing code. However, much like other AI-powered tools, AI-powered code completions are not always accurate, potentially introducing bugs or even security vulnerabilities into code if not properly detected and corrected by a human programmer. One technique that has been proposed and implemented to help programmers identify potential errors is to highlight uncertain tokens. However, there have been no empirical studies exploring the effectiveness of this technique -- nor investigating the different and not-yet-agreed-upon notions of uncertainty in the context of generative models. We explore the question of whether conveying information about uncertainty enables programmers to more quickly and accurately produce code when collaborating with an AI-powered code completion tool, and if so, what measure of uncertainty best fits programmers' needs. Through a mixed-methods study with 30 programmers, we compare three conditions: providing the AI system's code completion alone, highlighting tokens with the lowest likelihood of being generated by the underlying generative model, and highlighting tokens with the highest predicted likelihood of being edited by a programmer. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits, and is subjectively preferred by study participants. In contrast, highlighting tokens according to their probability of being generated does not provide any benefit over the baseline with no highlighting. We further explore the design space of how to convey uncertainty in AI-powered code completion tools, and find that programmers prefer highlights that are granular, informative, interpretable, and not overwhelming.

1 INTRODUCTION

AI code completions can introduce bugs or security vulnerabilities, and it is unclear which form of uncertainty highlighting best helps programmers catch errors in generated code. In a study with 30 programmers, edit-likelihood highlights supported faster, more targeted work, while generation-probability highlights provided no benefit over no highlights.

1 INTRODUCTION

  • AI-generated code can contain errors that introduce bugs or security vulnerabilities, so programmers need to detect and correct them.Even coherent, plausible outputs may be incorrect.Source: §1 · p. 1
  • There had been no empirical study of uncertainty highlighting for code, and it was unclear which uncertainty measure best helps programmers identify code needing oversight.Generation probability measures how likely a token is to be generated in context; the alternative estimates whether a programmer would modify or delete it to solve the task.Source: §1 · p. 2
  • The study compares code completion alone with highlights for low generation probability or high predicted likelihood of a programmer edit.The preregistered mixed-methods study involved 30 programmers and used a simple edit model trained on a closed-world set of programming tasks.Source: §1 · p. 2
  • Edit-model highlighting led to faster task completion and more targeted edits, while generation-probability highlighting offered no benefit over no highlights.Participants also reported higher subjective utility for edit-model highlights than for generation-probability highlights.Source: §1 · p. 2
  • Participants preferred highlights that were granular and informative without being overwhelming, and favored shading over precise probabilities.They also wanted explanations for the uncertainty and control over settings such as the highlighting threshold.Source: §1 · p. 2

2 RELATED WORK

Prior work examines uncertainty communication, error highlighting, and programmer performance, but how to compute and present uncertainty for generative code completion remains an open question. This paper builds on those strands by quantitatively studying uncertainty highlighting in AI code completion.

2.1 Computing and Communicating Uncertainty

  • Uncertainty communication, explanations, and user training have helped users detect AI errors, but how to compute and present uncertainty for generative tasks remains open.Prior work spans uncertainty communication, explanations, and onboarding and training.Source: §2 · p. 3
  • Generative tasks pose distinct uncertainty-communication challenges because outputs are unstructured tokens of varying length rather than simple atomic predictions.The passage notes that methods developed for classification-like tasks may not generalize directly to code generation.Source: §2.1 · p. 3

2.2 Highlighting Errors and Regions of Uncertainty

  • Prior programmer studies identified desires for uncertain-code highlights and explanations, while this work quantitatively measures highlighting’s effects in AI code completion.Sun et al. studied a wavy-line highlight; Vaithilingam et al. reported difficulty reading and debugging long Copilot suggestions.Source: §2.2 · p. 4

2.3 Generative Models and Programmer Performance

  • Programmer performance is multidimensional, so this study uses unit tests passed and task time as informative, but incomplete, measures.The cited work also raises concerns about insecure code with AI assistance [51] and “simple, stupid bugs” in generated code [23].Source: §2.3 · pp. 4–5

3 TWO NOTIONS OF UNCERTAINTY

The paper compares generation probability with an edit-oriented notion of uncertainty: whether a programmer would modify or delete a generated token to meet the task. It uses a closed-world edit model as a proof of concept and probe of whether communicating that signal helps programmers.

3 TWO NOTIONS OF UNCERTAINTY

  • The study considers two notions of uncertainty and illustrates their calculations and highlights across three coding tasks before presenting its hypotheses.The section introduces generation-model uncertainty and the alternative edit-oriented measure.Source: §3 · p. 5

3.1 Uncertainty in Code Generation Models

  • Generation probability is the conditional probability of a token in context and a localized measure of model uncertainty; many valid variable names can lower a choice’s probability, which programmers may mistake for an error.Source: §3.1 · p. 5
  • The proposed alternative is the likelihood that a programmer will modify or delete a generated token to meet their needs, approximated with a separate edit model.The study’s closed-world model serves as a proof of concept and a probe of whether revealing this signal helps programmers; a general-purpose open-world model is discussed as future feasibility work.Source: §3.1 · pp. 5–6

3.2 Building the Closed-World Edit Model

The closed-world edit model predicts which completion tokens programmers would edit for the study’s specific coding tasks. It estimates each token’s edit probability from participants’ observed changes to Codex completions.

3.2 Building the Closed-World Edit Model

  • The model predicts which completion tokens programmers are most likely to edit for the study’s three task completions.The study also piloted two additional tasks.Source: §3.2 · p. 6
  • Researchers ran task instructions through Codex to obtain completions and generation probabilities, then asked participants to revise the code to complete the task properly.Participants could use unit tests to check their code.Source: §3.2 · p. 6
  • Nine participants generated edit-model data; they were U.S.-based employees of a large technology company with Python experience.The passages state that they were recruited through a mix of methods.Source: §3.2 · p. 6
  • A token counted as edited if at least one character was changed, deleted, or commented out, and its edit probability was the fraction of participants who edited it.The model was built by programmatically tracking participant edits to each completion.Source: §3.2 · p. 6

3.3 Coding Tasks and OpenAI’s Completions

The authors screened LeetCode tasks for short completion time and imperfect Codex outputs, then selected three tasks spanning different error types. The completions included syntax and conceptual errors, from many syntax errors to small fixes.

3.3 Coding Tasks and OpenAI’s Completions

  • Fifteen easy LeetCode tasks were narrowed to three after piloting, aiming for roughly ten-minute tasks, imperfect completions, and diverse error types.Five tasks were used in the edit-model training phase before the final three were selected.Source: §3.3 · p. 6
  • Completions came from Code Davinci 002 using LeetCode prompts and mostly default parameters; the model’s training data was undisclosed, so problem overlap could not be ruled out.The model was retrieved in June 2022; the authors state that selected generations were imperfect and contained errors.Source: §3.3 · p. 6
  • For Ugly Number, Codex produced many syntax errors and the wrong function name despite correct overall logic that offered conceptual help.Examples include using “end” as a keyword and “is_ugly” instead of “isUgly.”Source: §3.3 · pp. 6–7
  • The study used one nondeterministic completion per task to reduce between-participant variance, ensure errors, and support a closed-world edit model.The authors note that Code Davinci 002’s training data was undisclosed and could include these LeetCode problems.Source: §3.3 · p. 6
  • For Base 7, Codex’s completion had no syntax errors but appended an extra “0” that had to be removed to pass unit tests.The task asks for an integer’s base-7 representation.Source: §3.3 · p. 7
  • For Most Common Word, fixing the unimported “List” type and dictionary update mistake was sufficient to reach correct code.The task asks for the most frequent non-banned word, returned in lowercase.Source: §3.3 · p. 7

3.4 Setting Thresholds

The authors set token thresholds for generation probability and the edit model so the two highlighting conditions marked approximately the same amount of text. They chose a participant-edit threshold for the edit model and a probability cutoff for generation probabilities.

3.4 Setting Thresholds

  • Thresholds were chosen so the two methods highlighted similar total character counts across the three coding tasks.Generation-probability tokens below a cutoff were highlighted, while edit-model tokens above a cutoff were highlighted.Source: §3.4 · p. 7

3.4.1 Edit Model Threshold.

  • Highlighting tokens edited by at least 4 of 6 participants yielded 203 highlighted characters across the tasks.Lowering the threshold to 3 participants produced identical highlights on two tasks and changed only one token on the third.Source: §3.4.1 · p. 7

3.4.2 Generation Probabilities Threshold.

  • A generation-probability cutoff below 0.694 yielded 205 highlighted characters across the three tasks.Source: §3.4.2 · p. 7

3.5 Differences and Imperfections in Generation Probabilities and the Edit Model

Across the three tasks, edit-model highlights tended to align with changes made to correct or improve code, while generation-probability highlights often marked important sequences or correct code. The task examples show these differences in what the two methods highlighted.

3.5 Differences and Imperfections in Generation Probabilities and the Edit Model

  • Generation-probability highlights also appeared on comments and correct tokens; edit-model highlights concentrated in the Ugly Number completion, which participants changed most.A generation-probability implementation bug missed two tokens and erroneously highlighted three; the authors did not believe this meaningfully affected results.Source: §3.5 · p. 7
  • Edit-model highlights tended to correspond to code errors, while generation-probability highlights often marked the starts and ends of important sequences.Figure 2 shows the highlights for both conditions across the three tasks.Source: §3.5 · p. 8, Figure 2 · p. 8
  • Highlighted tokens also included comparator and Boolean operators and, in several cases, newly introduced variable names.These are described as recurring patterns in the highlighted code.Source: §3.5 · p. 8
  • For Ugly Number, generation-probability highlights missed several obvious errors and marked correct code, whereas edit-model highlights marked those errors and unnecessary code.The missed errors included “end” as a keyword, incorrect capitalization of “True” and “False,” and the wrong function name.Source: §3.5 · p. 9
  • For Base 7, generation-probability highlights marked correct code, while the edit model highlighted the extra “0” whose removal passed unit tests.The corrected code would still fail one edge case not covered by the unit tests.Source: §3.5 · p. 9
  • For Most Common Word, generation-probability highlights missed the “List” error and marked correct code, while edit-model highlights marked both errors needed for the minimal correction.The errors were “List” and using “fromkeys” as a function that modifies variables in place.Source: §3.5 · p. 9

3.6 Pre-registered Hypotheses

The authors preregistered nine hypotheses about how highlighting conditions might affect task performance, editing, and participant judgments. They note that generation probabilities and the edit model have different distributions and characteristics, and may affect programmers differently.

3.6 Pre-registered Hypotheses

  • The authors expected highlighting condition to affect task completion time and unit-test accuracy, with highlights potentially helping direct attention to errors or distracting participants.The hypotheses were part of a preregistered set of nine.Source: §3.6 · p. 9
  • They also hypothesized effects on code additions, overall token survival, and token-level survival depending on whether a token was highlighted.Source: §3.6 · p. 9
  • The remaining hypotheses concerned cognitive load, perceived utility of the completion and highlights, and participants’ rankings of the tools.Source: §3.6 · pp. 9–10

4 STUDY DESIGN

The study used a within-subjects comparison of three code-completion highlighting conditions with 30 programmers, followed by interviews about their experiences and uncertainty-design preferences. Participants completed coding tasks, and the researchers measured task performance, edits, subjective perceptions, workload, and condition rankings.

4 STUDY DESIGN

  • The study involved 30 employees of a large U.S. technology company with Python experience; the sample size was limited by interviewer time and participant-payment constraints.Participants reported varied levels and frequency of Python experience.Source: §4 · p. 10

4.1 Study Procedure

  • In a randomized within-subjects procedure, participants completed three coding tasks with the same completions across conditions: no highlights, low-generation-probability highlights, or edit-model highlights.Each task was capped at 10 minutes; participants could run code and unit tests, then rated the tool and workload. Interviews also explored highlight granularity, specificity, and interactivity.Source: §4.1 · p. 11

4.2 Measured Variables

  • Measures covered task time, unit-test accuracy, added characters, completion-token survival, cognitive load, perceived completion and highlight utility, and participants’ final rankings.Token-level survival was also measured; survival meant a token was not edited, removed, or commented out.Source: §4.2 · p. 13

5 QUANTITATIVE RESULTS

The quantitative results examine how highlight condition relates to task performance, editing behavior, perceived usefulness, cognitive load, and participants’ rankings.

5.1 Impact of Highlight Condition on Time and Accuracy

  • Edit Model tasks averaged 8.59 minutes; completion time differed significantly from Generation Probability (p=0.003), with Edit Model participants taking 89.9% of the time required in that condition.Source: §5.1 · p. 15
  • Accuracy did not differ significantly by highlight condition (p=0.145): averages were 38.3% for No Highlights, 30.0% for Generation Probability, and 50.0% for Edit Model.Across coding tasks, Edit Model had the highest and Generation Probability the lowest accuracy, but this was a nonsignificant trend.Source: §5.1 · p. 16, Figure 5 · p. 16

5.2 Impact of Highlight Condition on Edits Made

  • Highlight condition had no significant effect on overall token survival, but Edit Model-highlighted tokens were already more likely to be edited and highlighting further increased the chance they were edited or removed.Source: §5.2 · pp. 16–17
  • Participants retained 148.9 added characters with No Highlights, 122.4 with Generation Probability, and 92.2 with Edit Model; only No Highlights versus Edit Model differed significantly (p=0.010).The study treats retained characters as only one component of effort, since minimizing edits alone could reward accepting incorrect completions.Source: §5.2 · p. 17
  • 79.3%, 79.8%, and 75.3%: overall token survival in No Highlights, Generation Probability, and Edit Model showed no significant condition effect (p = 0.164).Source: §5.2 · p. 17
  • 35.3% versus 87.1%: highlighted tokens in the Edit Model condition were less likely to survive than unhighlighted tokens; Generation Probability highlights also marked less-surviving tokens.In Generation Probability, highlighted-token survival was 73.9% versus 80.9% for unhighlighted tokens; the interaction between condition and highlight status was significant (p < 0.001).Source: §5.2 · p. 17, Figure 6 · p. 18
  • 35.3% versus 53.5%: Edit Model-highlighted tokens survived less often with highlights than in No Highlights, suggesting both token susceptibility and a highlighting effect.Source: §5.2 · p. 18
  • 73.9% versus 76.8%: Generation Probability-highlighted tokens had similar survival with highlights and in No Highlights, suggesting highlights alone did not meaningfully change behavior.The authors suggest highlights need to indicate plausible changes programmers would consider; generation probabilities did not appear to capture those changes.Source: §5.2 · p. 18

5.3 Impact of Highlight Condition on Cognitive Load and Utility

  • Highlight condition did not significantly affect cognitive load (p = 0.228) or code-completion utility (p = 0.679); tool rankings averaged 2.03, 2.10, and 1.87, respectively.The ranking means correspond to No Highlights, Generation Probability, and Edit Model; lower ranks indicate better rankings.Source: §5.3 · pp. 19–20
  • 3.92 versus 2.93: highlight utility was higher for Edit Model than Generation Probability (p < 0.001), though highlights alone gave insufficient guidance on what code to change.The scores were on a 1–7 scale; Edit Model utility was roughly neutral, while Generation Probability utility was roughly slightly disagreeing that highlights were useful.Source: §5.3 · p. 20
  • No pairwise ranking differences were significant; the Generation Probability–Edit Model difference was marginally significant (p = 0.063).The analysis used a cumulative link mixed model for rankings.Source: §5.3 · p. 20
  • 13 participants ranked Edit Model highest, compared with 9 for No Highlights and 8 for Generation Probability.Source: §5.3 · p. 20
  • The authors conjecture that participants ranked tools more by completion quality than highlight value, since completions varied across tasks assigned to highlighting conditions.Source: §5.3 · p. 20
  • Highlight utility and tool ranking did not always align: four participants favored Generation Probability despite reporting higher Edit Model highlight utility, while two showed the reverse.Source: §5.3 · p. 21
  • Participants’ comments reflected this tension, praising Generation Probability’s working code despite poor highlights while describing Edit Model as annoying or unhelpful.Source: §5.3 · p. 21

6 FURTHER EXPLORATION OF THE UNCERTAINTY HIGHLIGHTING DESIGN SPACE

Participants preferred uncertainty displays that help locate potential errors without overwhelming them. They reacted negatively to completion-level scores, exact probability values, and disruptive automated updates, while also warning that highlighting cannot cover errors caused by missing code.

Granularity of Uncertainty Highlighting

  • Participants found completion-level uncertainty too coarse to help locate errors, and described a number without context as abstract and hard to interpret.Some also worried a low score could prompt them to discard the whole completion.Source: §6 · pp. 21–22
  • Token-level highlights can pinpoint potential errors, but too many highlights may distract; some participants proposed digestible line- or block-level alternatives.Participants said the threshold determining how much to highlight is critical to avoiding a “sea of color.”Source: §6 · p. 22

Specificity of Uncertainty Highlighting

  • Participants preferred categorical uncertainty cues over exact numerical values, which they found difficult to interpret without context and not worth processing.Some participants liked multi-level opacity as a way to prioritize attention in long code.Source: §6 · pp. 22–23

Interactivity of Uncertainty Highlighting

  • Participants pushed back on both suggested alternative tokens and automatic highlight updates, preferring not to have the tool take disruptive initiative.Some considered token-level fixes quicker to make themselves and suggested line- or block-level alternatives instead.Source: §6 · p. 23
  • Participants requested control over highlight sensitivity and explanations of how uncertainty was computed, reinforcing the importance of transparency and user control.Suggestions included a sensitivity slider, buttons, or explanations of why the system considered code uncertain.Source: §6 · p. 23

Limitations of Highlighting

  • Highlighting cannot flag errors caused by missing code, so this limitation should be communicated when introducing such systems.The authors identify this as a limitation of all highlighting techniques.Source: §6 · p. 24

7 DISCUSSION

The discussion favors edit-model highlighting over generation-probability highlighting, while emphasizing limits to the model, study setting, and what highlighting can establish. It also cautions that the observed benefits have not been shown to reduce automation bias.

7 DISCUSSION

  • With the edit model, participants completed tasks faster, made more targeted edits, and reported higher utility than in the Generation Probability condition, which was often indistinguishable from No Highlights.The discussion points toward edit models for highlighting potential errors.Source: §7 · p. 24

7.1 Feasibility of Building and Limitations of an Edit Model

  • The edit model was closed-world and trained on the study’s own completions; it may miss errors people fail to catch and edit, and an open-world model remains unproven.Its effectiveness for identifying potential errors depends on edit data reflecting the changes people would want to make.Source: §7.1 · pp. 24–25

7.2 Representativeness of Tasks, Scenarios, and Participants

  • The LeetCode tasks were timed standalone challenges, and focusing on unit tests and algorithmic correctness may miss maintainability and security.The authors note these tasks may not represent programmers’ day-to-day coding or debugging scenarios.Source: §7.2 · p. 25
  • Participants were professionals from one US-based technology company, limiting generalizability; Python experience also corresponded with different task accuracy and struggles.Average unit-test accuracy ranged from 8.3% for participants with less than one year of Python experience to 51.5% for those with more than five years.Source: §7.2 · p. 25

7.3 Impact on Automation Bias

  • Highlights cannot flag missing code, and the observed gains in time, targeted edits, and preference have not yet been shown to increase oversight or reduce automation bias.Participants sometimes treated an absence of highlights as a signal that code was correct.Source: §7.3 · pp. 25–26

8 CONCLUSION

In a study with 30 programmers, edit-likelihood highlighting was associated with faster task completion and more targeted edits, while generation-probability highlighting offered no benefit over no highlights. Participants preferred highlights that were granular, informative, interpretable, and not overwhelming.

8 CONCLUSION

  • The study compared highlighting tokens least likely to be generated with highlighting tokens most likely to be edited by a programmer.These represent uncertainty about two distributions: token generation and programmer edits or deletions.Source: §8 · p. 26
  • Edit-likelihood highlighting led to faster task completion and more targeted edits, while generation-probability highlighting provided no benefit over no highlights.Participants also subjectively preferred edit-likelihood highlighting.Source: §8 · p. 26
  • In post-task interviews, programmers resisted highlights that covered too much or conveyed overly fine-grained uncertainty, preferring highlights that were granular, informative, interpretable, and not overwhelming.Source: §8 · p. 26
  • Token-highlighting strategies cannot communicate errors of omission.Source: §8 · p. 26

Related papers

Loading 2302.07248v3…