PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models for Test-Free Fault Localization

Aidan Z. H. Yang, Ruben Martins, Claire Le Goues, Vincent J. Hellendoorn

arXiv:2310.01726v1cs.SEcs.LG

TL;DR

Fault localization traditionally depends on tests, program analysis, or preprocessing, motivating a test-free approach for line-level debugging. LLMAO fine-tunes lightweight bidirectional adapters over frozen left-to-right LLM representations and reports stronger localization as model size increases, including security vulnerability detection.

  • Problem

    Existing fault-localization methods commonly require test information or extensive analysis and preprocessing, while left-to-right LLM representations are not immediately suited to discriminative line-level localization.

  • Method

    LLMAO trains lightweight bidirectional adapters on frozen left-to-right LLM representations to predict suspiciousness probabilities for each code line.

  • Results

    LLMAO’s fault-localization performance increases with LLM size, and the 16B model outperforms compared techniques, including 2.3% higher Top-1 and 14.4% higher Top-5 than TRANSFER-FL.

  • Takeaways & Limitations

    LLMAO localizes general defects and security vulnerabilities without test cases or extensive program-analysis artifacts.

  • Takeaways & Limitations

    Fault-line labels derived from bug-fixing diffs are a noisy proxy because annotators may disagree about the true defect cause.

Abstract

from arXiv · show

Fault Localization (FL) aims to automatically localize buggy lines of code, a key first step in many manual and automatic debugging tasks. Previous FL techniques assume the provision of input tests, and often require extensive program analysis, program instrumentation, or data preprocessing. Prior work on deep learning for APR struggles to learn from small datasets and produces limited results on real-world programs. Inspired by the ability of large language models (LLMs) of code to adapt to new tasks based on very few examples, we investigate the applicability of LLMs to line level fault localization. Specifically, we propose to overcome the left-to-right nature of LLMs by fine-tuning a small set of bidirectional adapter layers on top of the representations learned by LLMs to produce LLMAO, the first language model based fault localization approach that locates buggy lines of code without any test coverage information. We fine-tune LLMs with 350 million, 6 billion, and 16 billion parameters on small, manually curated corpora of buggy programs such as the Defects4J corpus. We observe that our technique achieves substantially more confidence in fault localization when built on the larger models, with bug localization performance scaling consistently with the LLM size. Our empirical evaluation shows that LLMAO improves the Top-1 results over the state-of-the-art machine learning fault localization (MLFL) baselines by 2.3%-54.4%, and Top-5 results by 14.4%-35.6%. LLMAO is also the first FL technique trained using a language model architecture that can detect security vulnerabilities down to the code line level.

1 INTRODUCTION

Fault localization traditionally uses tests, program analysis, or preprocessing, while prior learning approaches often use modest models or struggle with limited data. LLMAO instead adapts pretrained large language models with lightweight bidirectional layers to localize faults without tests.

  • Motivation: Traditional fault localization combines static or dynamic analysis to score program entities, while SBFL specifically relies on passing and failing test coverage.MLFL methods additionally use code, test, execution, suspiciousness, or fault-proneness features.
  • Motivation: Large language models capture increasingly useful code representations as model and training-data scale grow, but prior software-engineering work often uses smaller models.Program-synthesis performance increases linearly with the logarithm of model parameter count in the cited evidence.
  • Motivation: Left-to-right language models are poorly suited to line-level fault localization because each token representation is conditioned only on preceding context.This causal representation omits suffix information needed for discriminative prediction.
  • Contribution: LLMAO trains lightweight bidirectional adapters on frozen left-to-right language-model representations using only a few hundred real-bug training samples.The approach avoids test cases, test execution, test-case analysis, and substantial preprocessing overhead.
  • Contributions: LLMAO is presented as a language-model fault-localization technique spanning three languages and two application domains, including line-level security vulnerability detection.The paper also reports empirical evaluation against recent state-of-the-art fault-localization models.

2 MOTIVATION

Test-based fault localization can mis-rank or fail to expose defects when tests provide weak coverage or cannot exercise runtime concurrency behavior. LLMAO uses source-code representations to localize general defects and security vulnerabilities without test cases.

  • General Logic Defects: SBFL can fail to distinguish lines in a straight-line block and can be misled when only a few tests trigger the defect.In the Lang-47 example, test coverage assigns a higher score to a called line than to lines in the buggy methods.
  • General Logic Defects: DeepFL uses features such as SBFL suspiciousness and textual similarity, but its reported confidence is method-level, with limited statement-level results.This motivates evaluating a line-level alternative.
  • General Logic Defects: LLMAO ranks Lang-47’s actual faulty line fourth with score 0.33, while assigning the misleading called line score 0.09 outside the top 20.It also ranks another relevant line seventh with score 0.27.
  • Vulnerability Detection: Qemu’s CWE-362 race condition is not covered by tests for crashes or input behavior because the defect occurs during runtime concurrency.Traditional testing is described as ill-suited to discovering such concurrency bugs.
  • Vulnerability Detection: LLMAO detects security vulnerabilities without test cases and identifies three of five actual vulnerable lines in the Qemu example.Its predictions include lines 3, 4, 10, 12, and 23, while actual vulnerable lines include 10, 12, and 23–25.

3 APPROACH

LLMAO converts frozen left-to-right language-model representations into bidirectionally informed line-level bug probabilities. It trains only a compact adapter and output layer, enabling fault localization from buggy source code without modifying the base model.

  • Architecture: LLMAO tokenizes a buggy program, extracts one pretrained language-model representation per line, and predicts each line’s probability of being defective.Values close to 1 indicate lines likely to be buggy.
  • Architecture: The pretrained left-to-right language model remains frozen, while only the final bidirectional stage is trained.This makes the base model replaceable and reduces the trainable parameter burden.
  • Pretrained Backbone: CodeGen provides compatible left-to-right backbones ranging from 350M to 16B parameters, with the approach expected to work best for larger models trained on more code.The released CodeGen checkpoints are trained through natural-language, multi-language code, and BIGPYTHON stages.
  • Bidirectional Adapter: A bidirectional adapter removes causal masking so representations of later and earlier lines can exchange information.The added Transformer layers therefore encode context from the entire file rather than only its prefix.
  • Representation Processing: The pipeline extracts newline-token states as line representations, reduces their dimension from the pretrained size D to a smaller adapter size d, and experiments with d ∈ {256, 512, 1024}.The smaller adapter dimension is chosen because training data are limited.
  • Prediction: A sigmoid projection converts each adapted newline representation into a 0–1 bug probability trained with binary cross-entropy.The loss is backpropagated through the trainable layers but not the underlying language model.

4 EVALUATION

The evaluation examines LLMAO against prior fault-localization techniques, its generalization to unseen projects, and its applicability across languages and domains.

  • The evaluation addresses four questions covering comparison with prior FL techniques, generalization to new projects, component impact, and transfer across languages and domains.
  • LLMAO is evaluated on previously unseen code to assess generalizability beyond its training data.
  • The study evaluates LLMAO across different languages and application domains.

4.1 Setup

The setup evaluates source-only, statement-level fault localization using multiple buggy-program datasets, established baselines, cross-validation, and ablations of pretraining, model size, and bidirectionality.

  • Datasets: The study uses four datasets containing source code and labeled fault lines to evaluate real-world fault detection.
  • Datasets: Defects4J V1.2.0 contains 395 Java bugs from six projects, while BugsInPy contains 493 Python bugs from 17 projects and Devign contains 5,260 C vulnerabilities from two projects.
  • Labels: Faulty statements are identified from fixing-commit diffs, with changed statement line numbers used as training labels.
  • Task and baselines: LLMAO outputs a probability score for each code line using source code alone, enabling statement-level localization without tests or test coverage information.
  • Task and baselines: The comparison includes DeepFL, DeepRL4FL, TRANSFER-FL, and Ochiai, with the first two using test coverage and TRANSFER-FL using pretrained information from open-source Java programs.
  • Validation and analysis: Evaluation uses 10-fold cross-validation, early stopping by held-out precision and recall, Top-N metrics, ROC-AUC, and ablations of pretraining, CodeGen size, and bidirectional layers.The tested CodeGen sizes are 350 million, 6 billion, and 16 billion parameters.

4.2 Results

LLMAO outperforms prior fault-localization techniques, with performance improving as the underlying CodeGen model grows and generalizing to unseen projects, languages, and defect domains.

  • The CodeGen-16B version outperforms all compared deep-learning fault-localization techniques, while the 350M and 6B versions perform similarly to top prior methods.
  • 2.3% improvement over TRANSFER-FL at Top-1 and 14.4% improvement at Top-5 are achieved without test cases.
  • LLMAO generalizes to unseen Defects4J projects without retraining and localizes faults in BugsInPy and Devign, including C security vulnerabilities.
  • Fault detection improves with model scale: CodeGen-6B finds 3 more Top-1 faults than CodeGen-350M, and CodeGen-16B finds 3 more than CodeGen-6B.
  • Bidirectional adapters are crucial: removing them yields lower performance than every LLMAO model, despite CodeGen-16B alone detecting 85 Top-5 faults.
  • Top-1 effectiveness reaches 10.3% on BugsInPy and 28.1% on Devign, while Devign reaches 60.3% at Top-5.
  • The ROC analysis compares random prediction with Transformer and three pretrained CodeGen sizes, with higher AUC indicating stronger predictive power.

5 RELATED WORK

Prior fault-localization methods rely heavily on tests, program analysis, mutation, or structural features, whereas LLMAO applies bidirectional adapters directly to pretrained language-model representations of source code.

  • SBFL computes line suspiciousness from passed and failed test coverage, while MBFL uses mutants to measure code elements’ effects on test outcomes.
  • SBFL and MBFL depend on test-suite coverage and have limitations when statements share scores or suitable mutants cannot be instantiated.
  • MLFL methods learn from code, test, execution, suspiciousness, complexity, or coverage features, extending traditional fault-localization signals with learned models.
  • LLMAO requires neither test code nor an AST parser, instead using attention-based bidirectional adapters on pretrained left-to-right LLMs applied directly to source code.
  • LLM-based fault localization is motivated by pretrained code models’ learned representations, whose predictive power is compared across model sizes using ROC curves and AUC.

6 DISCUSSION AND THREATS

The paper argues that pretrained LLMs encode semantic and defect-related information useful for fault localization, while discussing threats from noisy labels, dataset overfitting, and possible training-data overlap.

  • Why does it work?: LLMs’ code-semantic knowledge may transfer to reasoning about defects and vulnerabilities because bug-related patterns can inform next-token prediction.The paper connects this hypothesis to pretrained representations that capture semantic information from code.
  • Threats to validity: Faulty-line labels derived from bug-fixing diffs are a noisy proxy because annotators may disagree about a defect’s true cause.The authors mitigate this concern with curated datasets, bug-relevant commit changes, and precedent for using developer fixes as ground truth.
  • Threats to validity: Training on one project can overfit to project-specific characteristics, so the study also evaluates training across datasets and generalization to unseen projects.The paper uses 10-fold validation and separately tests models on projects not seen during training.
  • Threats to validity: CodeGen may have seen some evaluation repositories during pretraining, so dataset inclusion bias cannot be fully eliminated.The authors note that CodeGen was trained on roughly 65 GiB of GitHub code through 2021, without manually annotated bug labels.

7 CONCLUSIONS

The paper presents LLMAO for localizing general defects and security vulnerabilities without extensive program-analysis artifacts or test cases. Across three real-bug and vulnerability benchmarks, it reports stronger localization than prior deep-learning approaches, with performance improving for larger LLMs and bidirectional adapters.

  • 7 CONCLUSIONS: LLMAO localizes general program defects and security vulnerabilities using an LLM-based approach.The empirical study covers 395 Defects4J bugs, 493 BugsInPy bugs, and 5,260 Devign vulnerabilities.
  • 7 CONCLUSIONS: 48/395 more faults are localized within Top-5 than by DeepFL, while 23/155 more bugs are localized within Top-5 than by TRANSFER-FL.DeepFL relies on SBFL and MBFL artifacts requiring extensive manual labor, whereas LLMAO does not use test cases or extensive program-analysis insights.
  • 7 CONCLUSIONS: Larger LLMs significantly improve AUC, and bidirectional adapter layers are essential for achieving state-of-the-art localization scores.The conclusion attributes the comparison to evaluations across different LLMAO model versions.
Loading 2310.01726v1…