PaperlayerView exact revision on arXiv

Source-linked AI summary

A Deep Dive into Large Language Models for Automated Bug Localization and Repair

Soneya Binta Hossain, Nan Jiang, Qiang Zhou, Xiaopeng Li, Wen-Hao Chiang, Yingjun Lyu, Hoan Nguyen, Omer Tripp

arXiv:2404.11595v3cs.SE

TL;DR

Line-level bug localization leaves repair models to identify the faulty part of a line. Toggle localizes and fixes bugs at token granularity and explores prompt designs; it establishes a new benchmark on CodeXGLUE code refinement and reports notable performance across several datasets, including Defects4J.

  • Problem

    Most existing APR techniques locate bugs at line level, leaving the fixing model to identify which part of the line is faulty.

    Source: §1.2 · p. 3

  • Method

    Toggle localizes and fixes bugs at token granularity and uses four diverse prompts to optimize LLMs as bug-fixing models.

    Source: §6 · p. 20

  • Results

    Toggle establishes a new benchmark on CodeXGLUE code refinement and shows notable performance on several datasets, including Defects4J.

    Source: §6 · p. 21

  • Takeaways & Limitations

    Token-granulated localization reduces redundant shared prefixes or suffixes in generated fixes, and prompt design can improve bug-fixing accuracy.

    Source: §6 · p. 20

    The authors note that results on CodeXGLUE and CodeReviewer may not generalize to other datasets, prompting evaluation on Defects4J.

    Source: §5 · p. 20

Abstract

from arXiv · show

Large language models (LLMs) have shown impressive effectiveness in various software engineering tasks, including automated program repair (APR). In this study, we take a deep dive into automated bug fixing utilizing LLMs. In contrast to many deep learning-based APR methods that assume known bug locations, rely on line-level localization tools, or address bug prediction and fixing in one step, our approach uniquely employs LLMs to predict bug location at the token level and subsequently utilizes them for bug fixing. This methodological separation of bug localization and fixing using different LLMs enables effective integration of diverse contextual information and improved incorporation of inductive biases. We introduce Toggle: Token-Granulated Bug Localization and Repair, a comprehensive program repair framework that integrates a bug localization model, an adjustment unit, and a bug-fixing model. Toggle takes a buggy function as input and generates a complete corrected function. We investigate various styles of prompting to the bug fixing model to identify the most effective prompts that better utilize the inductive bias and significantly outperform others. Toggle achieves the new state-of-the-art (SOTA) performance on the CodeXGLUE code refinement benchmark, and exhibits better and comparable performance on several other widely-used APR datasets, including Defects4J.

1 INTRODUCTION

Existing APR approaches often depend on known buggy lines, line-level tools, or combined localization and fixing, while LLM bug-fixing prompts and token-level localization remain underexplored. Toggle separates token-level bug localization from fixing and investigates prompts, context, and tokenizer adjustment.

1.1 Prompting LLMs for Bug Fixing

  • Existing prompts can make LLMs regenerate shared code, although a fix may require changing only a small buggy fragment.The example changes “getProperty” to “get”; generating shared prefixes and suffixes is unnecessary and may introduce bugs.Source: §1.1 · pp. 2–3

1.2 Bug Localization

  • Line-level localization identifies a buggy line but leaves the fixer to determine which specific part of that line needs changing.Token-granular prediction targets the beginning and end of the buggy segment and can use contextual information such as code comments.Source: §1.2 · p. 3

1.3 Connection Between Bug Localization and Bug Fixing

  • Different tokenizers can map the localized “Property” span and the fixer’s “getProperty” token to incompatible edit boundaries.Toggle’s optional adjustment module considers both the localization prediction and the fixing model’s performance when seeking a more suitable location.Source: §1.3 · p. 4

1.4 Contributions

  • Toggle introduces token-level localization and four bug-fixing prompts; avoiding redundant shared-code generation yields the highest bug-fixing accuracy.The framework also identifies tokenizer inconsistency and introduces an adjustment module to mitigate it.Source: §1.4 · pp. 4–5
  • Fine-tuned LLM size is associated with bug-fixing accuracy, and CodeParrot-110M generalizes to unseen Defects4J bugs, outperforming existing methods on several metrics.The paper also advocates token-level rather than line- or method-level bug-location prediction.Source: §1.4 · p. 5
  • Additional context improves buggy-token prediction, while predicting both start and end positions is harder than predicting only the start.The evaluated context is buggy line numbers for Tufano datasets and code-review comments for CodeReviewer.Source: §1.4 · p. 5
  • The adjustment module improves bug-fixing accuracy across four 110–400M-parameter LLMs, though its training is relatively costly.The contribution recommends using the module when working with different tokenizers, particularly with smaller LLMs.Source: §1.4 · p. 5
  • Prompt 4 works better when predicted start and end locations are highly accurate; otherwise, the better choice depends on dataset, context, and backbone.The paper recommends testing both prompts 3 and 4 when accounting for localization error.Source: §1.4 · p. 5

2 IMPLEMENTATION

Toggle combines token-level bug localization, prompt designs that use the predicted location, and an optional adjustment model. Its implementation uses encoder models for localization and decoder models for fixing across several datasets.

2.1 Large Language Models for Code

  • Localization uses pretrained encoder models, while fixing uses autoregressive decoder models; the study uses CodeT5-large’s 347M-parameter encoder for localization.The paper selected models based on popularity, availability, and compatibility with its GPU resources.Source: §2.1 · p. 6

2.2 Dataset

  • Evaluation and training use CodeXGLUE’s Tufano Small and Medium datasets, CodeReviewer, Defects4J, and a curated GitHub dataset.Defects4J is used to investigate generalizability; the GitHub dataset and Tufano are used for fine-tuning and Defects4J evaluation.Source: §2.2 · pp. 6–7

2.3 Overview of Toggle

  • Toggle passes a buggy function through localization, fixing, and optional adjustment models to produce corrected code.The adjustment model tunes predicted locations when needed to improve patch generation.Source: §2.3 · p. 7
  • Toggle separates localization from fixing to simplify reuse of pretrained LLMs, integrate contextual information, and support flexible fixing prompts.The adjustment model connects the two stages when predicted locations need tuning.Source: §2.3 · pp. 7–8

2.3.1 Bug Localization Model: Integration of contextual information.

  • The localization model predicts the start and end of the buggy token span, using CodeT5’s encoder and optionally contextual information such as reviewer comments.A context mask can restrict attention to buggy-line tokens; reviewer comments can be appended to the code input.Source: §2.1 · p. 6, §2.3.1 · pp. 8–9

2.3.2 Bug Fixing Model: Incorporation of inductive bias.

  • Four fixing prompts vary how they use the localized span, from regenerating the full function to truncating shared prefix and suffix tokens.Prompt 4 also truncates the shared suffix, reducing generated tokens but requiring both bug boundaries to be predicted accurately.Source: §2.3.2 · pp. 9–10, Figure 5 · p. 10

2.3.3 Adjust Model: Connection between bug localization and bug fixing.

  • The optional adjustment model predicts location shifts that help the fixing model produce correct patches despite imperfect localization.Its training examples shift predicted locations from -3 to +3 and retain shifts for which the fixing model generates correct patches.Source: §2.3.3 · p. 11, Figure 6 · p. 11

3 EXPERIMENTAL STUDY

The experimental study is organized around six research questions covering Toggle’s comparisons, generalization, prompts, context, location adjustment, and the choice between two leading prompts.

3 EXPERIMENTAL STUDY

  • Six research questions examine comparisons with other methods, unseen-data generalization, prompt choices, additional context, location adjustment, and prompts 3 versus 4 with predicted locations.Figure 3 maps the framework components used in each study.Source: §3 · p. 12

3.1 RQ1: Toggle’s bug fixing effectiveness

Toggle’s bug-fixing effectiveness is evaluated on four datasets using exact-match accuracy. The reported results vary by dataset and model, with larger backbones generally performing better and prompt performance depending on dataset and context.

3.1.1 Experimental Setup.

  • The evaluation fine-tuned CodeT5-large for token-location prediction, then fine-tuned bug-fixing models with prompt 3 and measured test-set exact match across four datasets.The datasets were split roughly 80/10/10; an exact match required the generated solution to match the ground truth.Source: §3.1.1 · p. 12

3.1.3 Results.

  • On Tufano Small, PolyCoder-2.7B reached 25.07% versus NSEdit’s 23.86%; on Tufano Medium, it reached 16.19% versus CoText’s 15.36%.These are the highest reported accuracies on the respective datasets.Source: Table 1 · p. 12, §3.1.3 · p. 13
  • On CodeReviewer without comments, CodeGen-2B reached 12.31% versus NSEdit’s 11.97%; with comments, it reached 25.59% versus CodeT5’s 24.41%.CodeGPT underperformed the other models, which the authors attribute primarily to its Java-only pretraining on a multilingual dataset.Source: Table 1 · p. 12, §3.1.3 · p. 13

3.1.2 Baseline.

  • Baselines were selected from top-performing leaderboard models for Tufano and top CodeReviewer models, including CodeT5 and NSEdit.The CodeReviewer baselines were reported at 24.42% with comments for CodeT5 and 11.48% without comments for the replicated CodeT5 method.Source: §3.1.2 · pp. 12–13

3.1.3 Results.

  • Larger backbones generally performed better, while some models performed better with prompt 4 depending on the dataset and additional context.Within the same backbone family, the larger model outperformed its smaller counterpart; the study examines prompt conditions further in RQ4 and RQ6.Source: §3.1.3 · p. 13

3.2 RQ2: Toggle’s generalizability

Toggle was tested on Defects4J bugs excluded from fine-tuning, using generated patches validated against test cases. The authors report strong generalization and better patch-generation results than existing APR methods across several candidate counts.

3.2.1 Experimental Setup.

  • The evaluation covered 240 single-hunk bugs from 17 Defects4J Java projects that were not included in fine-tuning.A patch counted as correct only if it passed all test cases, which the authors also manually executed.Source: §3.2.1 · pp. 13–14, §3.2 · p. 13
  • The setup used fine-tuned CodeT5 localization models and CodeParrot-110M for fixing, generating 210 patches per bug across three localization/fixing combinations.The location model was fine-tuned on Tufano Small, Tufano Medium, or GitHub rather than Defects4J.Source: §3.2.1 · pp. 13–14

3.2.2 Results.

  • The localization models predicted the exact starting token for 139 bugs and a location before the ground truth for 209, allowing fixing models to generate patches from imperfect locations.The three combinations provided different strengths; the third identified the most correct locations and ranked second in total and unique patches.Source: §3.2.2 · p. 14, Table 2 · p. 14
  • Toggle with CodeParrot-110M and prompt 3 fixed more Defects4J bugs than the compared state-of-the-art techniques at 10, 30, 50, and 100 patches per bug.The paper’s finding reports outperformance across Top-10, 30, 50, 70, and 100 metrics.Source: §3.2.2 · p. 14, Table 3 · p. 15

3.3 RQ3: Impact of prompts on bug fixing effectiveness

This study tests how prompt format affects bug-fixing accuracy, using ground-truth bug-token locations to isolate prompt effects. Across the studied models, accuracy generally improves from prompt 1 to prompt 4, while line-granular locations can reduce accuracy.

3.3 RQ3: Impact of prompts on bug fixing effectiveness

  • With ground-truth buggy-token locations, six bug-fixing LLMs were fine-tuned on Tufano Small to compare four prompt formats without localization errors confounding accuracy.The resulting test accuracies are presented in Table 4.Source: Table 4 · p. 15, §3.3.1 · p. 15

3.3.1 Experimental Setup.

  • CodeParrot achieved 24.5% with prompt 1 and 63.83% with prompt 4, the highest accuracy across all models for prompt 4.Prompts 2 and 3 produced similar accuracy for CodeParrot.Source: Table 4 · p. 15, §3.3.2 · p. 15
  • CodeGPT’s accuracy rose from 16.07% with prompt 1 to 56.98% with prompt 4.The paper reports consistent and significant improvement across prompts for CodeGPT.Source: §3.3.2 · p. 15
  • CodeGen-350M improved from 22% to 60.22%, while PolyCoder improved from 23.66% to 62.29%.The reported increases are from prompt 1 to prompt 4.Source: §3.3.2 · p. 15

3.3.2 Results.

  • With line-granular locations, CodeGPT-110M reached only 28.5% exact-match accuracy on Tufano Small, below prompt 2 accuracy.The authors identify token-granular localization as important to bug-fixing effectiveness.Source: §3.3.2 · p. 15
  • Across the LLMs studied, bug-fixing accuracy noticeably improved from prompt 1 to prompt 4.The authors suggest later prompts might be more effective or relevant for assisting bug fixing.Source: §3.3.2 · p. 16

3.4 RQ4: Impact of contextual information on location prediction accuracy

The study evaluates whether additional context improves token-location prediction and compares predicting only the start of a buggy span with predicting both its start and end. Context improves prediction accuracy, while predicting both boundaries is harder.

3.4.1 Experimental Setup.

  • The evaluation fine-tuned CodeT5 on buggy code with or without additional context, predicting either the starting token or both span boundaries across Tufano and CodeReviewer datasets.Tufano context was the buggy line number; CodeReviewer context was code review comments.Source: §3.4.1 · p. 16
  • Adding buggy line numbers raised Tufano Small starting-token accuracy by 21% and both-token accuracy by 26%; for Tufano Medium, the gains were 30% and 28%.The table compares accuracy with and without additional context for starting-token prediction and joint start-and-end prediction.Source: Table 5 · p. 16, §3.4.2 · p. 16
  • Adding CodeReviewer comments increased starting-token prediction accuracy by 20% and both-token accuracy by 16%.The study used code review comments as the additional context for CodeReviewer.Source: Table 5 · p. 16, §3.4.2 · p. 16

3.4.2 Results.

  • Across all datasets, predicting both the starting and ending buggy tokens had lower accuracy than predicting only the starting token.Both token locations are required for prompt 4, whereas prompts 2 and 3 require only the starting token.Source: §3.4.2 · pp. 16–17

3.5 RQ5: Impact of adjustment module on bug fixing accuracy

The adjustment module improves bug-fixing accuracy across the studied datasets and models. It can shift a predicted location to avoid a newline-related premature method ending.

3.5 RQ5: Impact of adjustment module on bug fixing accuracy

  • The adjustment module targets tokenizer discrepancies that can make a localization model’s predicted position unsuitable for the fixing model.The experiment compares bug-fixing accuracy with the module enabled and disabled across four datasets and four models.Source: §3.5 · p. 17, §3.5.1 · p. 17

3.5.1 Experimental Setup.

  • The evaluation covers Tufano Small, Tufano Medium, and CodeReviewer datasets with and without comments.The study reports 16 dataset-and-model experiment sets.Source: §3.5.1 · p. 17

3.5.2 Results.

  • Accuracy improves with location adjustment across all studied datasets and four LLMs with 110–400M parameters.The reported results show consistent improvement across scenarios.Source: Table 6 · p. 17, §3.5.2 · pp. 17–18
  • Shifting the predicted location left by one character, excluding the newline, can prompt a newline before the correct fix.In the illustrated example, the correctly predicted location still led the fixing model to prematurely close the method; adjustment changed the outcome.Source: Figure 7 · p. 17, §3.5.2 · p. 18

3.6 RQ6: Effectiveness of prompt 3 vs 4 with predicted locations

With predicted locations, the better choice between prompts 3 and 4 depends on the dataset, context, and backbone model. Dual-location prediction is less accurate than single-token prediction, but prompt 4 can perform better when predicted locations are highly accurate.

3.6 RQ6: Effectiveness of prompt 3 vs 4 with predicted locations

  • Prompt 3 requires only the starting buggy-token location, whereas prompt 4 requires both the starting and ending locations.Prompt 4 outperforms the alternatives when supplied locations are highly accurate, but predicting both tokens is less accurate than predicting only the start.Source: §3.6 · p. 18

3.6.1 Experimental Setup.

  • The study evaluates four location-accuracy categories and compares prompts 3 and 4 using CodeGPT, CodeParrot, CodeGen-350M, and PolyCoder-400M.Each of four datasets is evaluated.Source: §3.6.1 · p. 18
  • Correctly predicting both buggy-token boundaries is less accurate than predicting either boundary alone; partial predictions may still cover the buggy portion.Partial accuracy indicates cases where the predicted start precedes and the predicted end follows the ground-truth buggy segment.Source: Table 7 · p. 18, §3.6.2 · p. 18

3.6.2 Results.

  • For Tufano Small and Medium, prompt 3 consistently outperforms prompt 4; for CodeReviewer datasets, prompt 4 is generally more effective in most instances.The comparison is reported across the models and datasets in Table 8.Source: §3.6.2 · pp. 18–19, Table 8 · p. 19
  • Higher partial-location accuracy and code-review comments are proposed as reasons prompt 4 tends to work better on CodeReviewer datasets.The comments provide additional context for the bug-fixing models.Source: §3.6.2 · p. 19
  • Prompt 3 or 4 can perform better depending on the dataset, additional context, and backbone; prompt 4 generally works better when both predicted locations are highly accurate.Otherwise, testing both prompts is recommended for optimal performance.Source: §3.6.2 · p. 19

4 RELATED WORK

Prior APR approaches use translation, edits, or LLMs, with different constraints and unresolved questions around localization and prompt design. The paper also situates LLM use in software engineering beyond APR.

4.1 Deep Learning-Based Automated Program Repair

  • NMT-based APR treats buggy-to-fixed code as a sequence-to-sequence translation task [5] [19] [44].The approach is straightforward because buggy and fixed code are both sequences, but large overlap may encourage copying that overfits and misses the goal of editing [5] [19] [44].Source: §4.1 · p. 19
  • Edit-based methods target buggy code directly, but may require multiple editing stages or graph representations that assume parseable code [13] [18] [35] [45] [46].The graph assumption may fail when bugs contain syntactical errors [43].Source: §4.1 · p. 19
  • Earlier LLM-based APR studies report fixing capability but do not address bug localization integration or comprehensive prompt design for fine-tuning [17].Source: §4.1 · p. 20

4.2 Large Language Models For Software Engineering

  • LLMs have also been used for code generation [3] [4] [21] [25], vulnerability detection and fixing [7] [34] [38], fuzzing [9], and software specification generation [41].Such adoption typically uses fine-tuning when sufficient data are available or prompt design for prompt-engineering approaches.Source: §4.2 · p. 20
  • The paper focuses on APR and suggests its prompt-design insight may benefit other LLM-based software-engineering tasks.Source: §4.2 · p. 20

5 THREATS TO VALIDITY

The authors identify possible limits to generalization and experimental validity, and describe their accuracy measures for the studied datasets.

5 THREATS TO VALIDITY

  • Results from CodeXGLUE and CodeReviewer may not generalize to other datasets; the authors also evaluate on unseen Defects4J.Defects4J is described as a widely used real-world bug dataset.Source: §5 · p. 20
  • Custom tools and scripts could contain bugs, so the authors used widely used libraries, validity checks, and repeated experiments to check consistency.Source: §5 · p. 20
  • Bug-fixing accuracy uses exact match; for Defects4J, a patch is correct only if it passes all test cases.Source: §5 · p. 20

6 CONCLUSION

Toggle combines token-level bug localization, bug-fixing prompts, and an adjustment module; the authors report a new CodeXGLUE code-refinement benchmark and notable performance on several datasets, including Defects4J.

6 CONCLUSION

  • Toggle localizes and fixes bugs at token granularity, uses four prompting strategies, and adds an adjustment module to reconcile localization and fixing model discrepancies.The token-level approach reduces redundant generation of shared prefixes and suffixes.Source: §6 · p. 20
  • The study evaluates four prompts across six LLMs and five benchmarks, offering insights into LLM bug-fixing capabilities.The conclusion describes the experiments as extensive.Source: §6 · p. 20
  • Toggle established a new benchmark in CodeXGLUE code refinement and showed notable performance on several datasets, including Defects4J.The conclusion describes this as the success of the proposed framework and strategy.Source: §6 · p. 21
Loading 2404.11595v3…