PaperlayerView exact revision on arXiv

Source-linked AI summary

SkipAnalyzer: A Tool for Static Code Analysis with Large Language Models

Mohammad Mahdi Mohajer, Reem Aleithan, Nima Shiri Harzevili, Moshi Wei, Alvine Boaye Belle, Hung Viet Pham, Song Wang

arXiv:2310.18532v2cs.SE

TL;DR

Static bug detectors can generate false-positive warnings that require manual review, and detected bugs still need manual fixes. SkipAnalyzer uses ChatGPT for bug detection, warning filtering, and patch generation; experiments report improvements across these tasks, including higher detection precision than Infer.

  • Problem

    Static bug detectors can produce numerous false-positive warnings that require manual review, and detected bugs still need manual fixes.

    Source: §I · p. 1

  • Method

    SkipAnalyzer combines a ChatGPT-based bug detector, false-positive filter, and patch generator, evaluated on two bug types collected with Infer.

    Source: §I · pp. 1–2

  • Results

    Across detection, false-positive removal, and repair, SkipAnalyzer reports improvements over prior baselines; its detection precision is 12.86% higher for Null Dereference and 43.13% higher for Resource Leak bugs than Infer.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    The authors release the dataset and experiment source code for future use.

    Source: §I · p. 2

    The study focuses solely on Java projects because Infer, used to generate warnings and the dataset, cannot handle multiple programming languages in this study.

    Source: §VIII · p. 9

Abstract

from arXiv · show

We introduce SkipAnalyzer, a large language model (LLM)-powered tool for static code analysis. SkipAnalyzer has three components: 1) an LLM-based static bug detector that scans source code and reports specific types of bugs, 2) an LLM-based false-positive filter that can identify false-positive bugs in the results of static bug detectors (e.g., the result of step 1) to improve detection accuracy, and 3) an LLM-based patch generator that can generate patches for the detected bugs above. As a proof-of-concept, SkipAnalyzer is built on ChatGPT, which has exhibited outstanding performance in various software engineering tasks. To evaluate SkipAnalyzer, we focus on two types of typical and critical bugs that are targeted by static bug detection, i.e., Null Dereference and Resource Leak as subjects. We employ Infer to aid the gathering of these two bug types from 10 open-source projects. Consequently, our experiment dataset contains 222 instances of Null Dereference bugs and 46 instances of Resource Leak bugs. Our study demonstrates that SkipAnalyzer achieves remarkable performance in the mentioned static analysis tasks, including bug detection, false-positive warning removal, and bug repair. In static bug detection, SkipAnalyzer achieves accuracy values of up to 68.37% for detecting Null Dereference bugs and 76.95% for detecting Resource Leak bugs, improving the precision of the current leading bug detector, Infer, by 12.86% and 43.13%, respectively. For removing false-positive warnings, SkipAnalyzer can reach a precision of up to 93.88% for Null Dereference bugs and 63.33% for Resource Leak bugs. Additionally, SkipAnalyzer surpasses state-of-the-art false-positive warning removal tools. Furthermore, in bug repair, SkipAnalyzer can generate syntactically correct patches to fix its detected bugs with a success rate of up to 97.30%.

I. INTRODUCTION

SkipAnalyzer addresses the false-positive review and manual repair burdens of static analysis with an LLM-based detector, warning filter, and patch generator. The study evaluates the tool on two bug types and reports gains across its three tasks.

I. INTRODUCTION

  • Static bug detectors produce false-positive warnings requiring manual review, while detected bugs also need manual fixes.The paper describes these processes as time-consuming, labor-intensive, and demanding of developer expertise.Source: §I · p. 1
  • SkipAnalyzer combines an LLM-based detector, false-positive filter, and patch generator, and is built on ChatGPT as a proof of concept.The tool is intended to detect bugs, filter false-positive warnings, and patch detected bugs without human intervention.Source: §I · pp. 1–2
  • The evaluation uses Infer-collected Null Dereference and Resource Leak bugs from 10 open-source projects, with ChatGPT-3.5 Turbo and ChatGPT-4.The dataset contains 222 Null Dereference bugs and 46 Resource Leak bugs; prompting strategies include zero-shot, one-shot, and few-shot.Source: §I · p. 1
  • SkipAnalyzer’s detector improves precision over Infer by 12.86% for Null Dereference and 43.13% for Resource Leak bugs.Its false-positive filter improves Infer’s precision by 28.68% and 9.53% for those bug types, respectively; patch correctness rates are 97.30% and 91.77%.Source: §I · p. 2
  • SkipAnalyzer’s false-positive filter removes warnings from Infer and its detector, improving precision and surpassing existing baseline methods.The paper also reports that its patch generator repairs detected bugs with accurate patches compared against a prior-research baseline.Source: §I · p. 2
  • The authors release the experiment dataset and source code for future use.They also describe SkipAnalyzer as improving Infer’s base results.Source: §I · p. 2

II. BACKGROUND

The background describes static bug detection as rule-based code inspection and presents false-positive warnings as a substantial practical challenge. It also situates Infer among static-analysis tools and prior warning-removal approaches.

A. Static Bug Detection

  • Static bug detection inspects source code, object code, or binaries without executing the program, issuing warnings when code violates bug-pattern rules.Infer uses predefined rules and inter-procedural analysis; SpotBugs and ErrorProne are also described as static bug detectors.Source: A. Static Bug Detection · p. 2
  • Infer is the study’s baseline detector and warning-collection tool; SpotBugs targets Java bytecode, while ErrorProne targets Java programs.Infer supports several languages, including Java, C, C++, Objective-C, and C#.Source: A. Static Bug Detection · p. 2

B. False-Positive Warning Removal

  • False-positive warnings can reach 91% and require time-consuming manual verification, motivating methods to remove unactionable warnings.Prior approaches include syntactic model checking, feature-based prediction, machine learning, and neural models; the study selects prior tools as baselines.Source: B. False-Positive Warning Removal · p. 2

C. Automated Program Repair

The paper frames automated repair and LLM prompting, then describes collecting and manually labeling Infer warnings to create its evaluation data. It compares its repair approach with VulRepair, which also avoids test-case execution for validation.

C. Automated Program Repair

  • Automated program repair generates program fixes; tools may use test suites for dynamic validation or static methods for bug localization and validation.The paper selects VulRepair as its repair baseline because it, like SkipAnalyzer, does not require test-case execution for validation.Source: C. Automated Program Repair · p. 3

D. Large Language Models

  • Prompt engineering tailors input queries to LLMs and includes zero-shot, one-shot, and few-shot strategies.The paper uses ChatGPT models in its LLM-based approach.Source: D. Large Language Models · p. 3

III. DATA COLLECTION

  • Infer generates warnings for Null Dereference and Resource Leak bugs in seven prominent GitHub projects and three projects from prior research.The authors chose these bug types as typical and critical static-analysis targets and selected Infer partly for its adoption and precision.Source: §III · p. 3
  • Three experienced authors independently label Infer warnings as true or false positives, resolve disagreements, and collaboratively patch each true bug.The resulting dataset pairs warnings with ground-truth labels.Source: §III · p. 3

IV. APPROACH

SkipAnalyzer is an LLM-based pipeline for detecting specified bugs, filtering false-positive warnings, and generating patches. Its components analyze code and warnings using specialized instructions.

IV. APPROACH

A. LLM-based Static Bug Detector

  • The detector analyzes buggy code using bug-specific prompts, structured outputs, and optional examples to produce warnings and explanations.The prompts include common patterns for Null Dereference and Resource Leak bugs.Source: A. LLM-based Static Bug Detector · p. 4

B. LLM-based False-Positive Warning Filter

  • The false-positive filter evaluates a code snippet together with its warning, which may come from Infer or SkipAnalyzer.Its prompts can include examples for one-shot or few-shot strategies.Source: B. LLM-based False-Positive Warning… · p. 4

C. LLM-based Static Bug Repair

  • The repair component uses the buggy code and its warning to generate a fixed version, but its prompt does not support example-based strategies.The authors attribute this constraint to examples’ varying method lengths and the LLMs’ input-token limit.Source: C. LLM-based Static Bug Repair · p. 4

V. EXPERIMENT SETTINGS

The experiments evaluate bug detection, false-positive warning removal, and bug repair using ChatGPT models. For the tested models, code inputs are restricted to method scope.

A. Research Questions

  • The experiments ask whether SkipAnalyzer detects bugs, filters false-positive warnings, and repairs buggy code.The warning filter is evaluated on outputs from Infer and SkipAnalyzer; repair is compared with VulRepair.Source: A. Research Questions · p. 5
  • The repair comparison uses VulRepair because the identified bugs lack test cases required by most generate-and-validate repair techniques.VulRepair is described as a CodeT5-based static repair model that does not require test cases as ground truth.Source: A. Research Questions · p. 5

B. ChatGPT Versions

  • SkipAnalyzer uses ChatGPT-4 and ChatGPT-3.5 Turbo with default maximum token supports of 8,192 and 4,097, respectively.Because of these input limits, the code supplied to the LLMs is restricted to methods.Source: B. ChatGPT Versions · p. 5

C. Prompting Strategies

The study varies prompting strategies for detection and warning filtering, uses five-fold sampling, and evaluates detection, filtering, and repair with task-specific measures.

C. Prompting Strategies

  • Zero-shot prompts omit examples, one-shot prompts include one prior input-output example, and few-shot prompts include multiple examples.The study uses three examples for its few-shot strategy (K = 3).Source: C. Prompting Strategies · p. 5
  • Detection and false-positive filtering use zero-shot, one-shot, and few-shot strategies, while repair uses zero-shot only.The repair prompt’s example constraint is attributed to method-length variation and model token limits.Source: C. Prompting Strategies · p. 5
  • The prompts ask the LLM to explain its decision process, an approach the paper calls zero-shot Chain-of-Thought reasoning.The authors cite prior literature stating that this approach can enhance output robustness and validity.Source: C. Prompting Strategies · p. 5

D. Data Sampling

  • Five-fold cross-validation uses one-fifth of the data for validation and the remaining four-fifths to select prompting examples.Examples are sampled uniformly at random.Source: D. Data Sampling · p. 5
  • One-shot and few-shot examples include one true-positive and one false-positive record to avoid bias toward either group.This example composition is used when applying those prompting strategies.Source: D. Data Sampling · p. 5
  • Table II summarizes static bug detection results across model-strategy combinations on two datasets.Its highlighted rows identify the most effective combination separately for Null Dereference and Resource Leak bugs.Source: Table II · p. 6

E. Evaluation Metrics

  • Detection and warning filtering are assessed with accuracy, precision, and recall; F1-score selects the best model-strategy combination.The measures for repair are Logic Rate and Syntax Rate.Source: E. Evaluation Metrics · p. 6
  • Logic Rate compares repaired code with manually crafted ground-truth patches, while Syntax Rate uses a Java parser.Logic Rate measures correct logic, and Syntax Rate measures correct syntax.Source: E. Evaluation Metrics · p. 6

VI. RESULT ANALYSIS

SkipAnalyzer’s detection results are compared with Infer across two bug types and datasets. The section also describes evaluation of its warning filter against prior baselines.

A. RQ1: Performance of SkipAnalyzer on Static Bug Detection

  • 12.86% and 43.13% higher precision than Infer are reported for SkipAnalyzer on Null Dereference and Resource Leak bugs, respectively.For both bug types in the collected dataset, ChatGPT-4 with zero-shot prompting is the most effective model-strategy combination.Source: A. RQ1: Performance of SkipAnalyzer on… · p. 6
  • 68.37% accuracy, 63.76% precision, and 88.93% recall are reported for Null Dereference detection.For Resource Leak detection, the reported accuracy, precision, and recall are 76.95%, 82.73%, and 55.11%, respectively.Source: A. RQ1: Performance of SkipAnalyzer on… · p. 6
  • On the dataset from Kharkar et al.’s projects, ChatGPT-4 with zero-shot prompting yields precision boosts of 16.6% for Null Dereference and 46.2% for Resource Leak versus Infer.The passage identifies this as the most effective model-strategy combination for that dataset.Source: A. RQ1: Performance of SkipAnalyzer on… · p. 7

B. RQ2: Performance of SkipAnalyzer on False-Positive Warning Removal

  • The warning-filter evaluation supplies a code snippet and its warning, using outputs from Infer or SkipAnalyzer across ChatGPT models and prompting strategies.The comparison baselines are GPT-C, feature-based logistic regression, and DeepInferEnhance; the authors gathered a similar dataset because the prior study released neither its data nor tool code.Source: B. RQ2: Performance of SkipAnalyzer on… · p. 7

1) Option 1 – Infer:

SkipAnalyzer’s false-positive filter improves precision for warnings from both Infer and SkipAnalyzer, and outperforms earlier removal baselines. Its gains vary by bug type and source detector.

1) Option 1 – Infer:

  • 28.68% for Null Dereference and 9.53% for Resource Leak bugs are the respective improvements to Infer’s precision; SkipAnalyzer also surpasses existing removal baselines.For Resource Leak bugs, Kharkar et al. did not report results for two baselines.Source: 1) Option 1 – Infer · p. 7

2) Option 2 – SkipAnalyzer:

  • 16.31% is the precision improvement when the filter processes SkipAnalyzer’s Null Dereference warnings using few-shot ChatGPT-4.The authors selected the zero-shot ChatGPT-4 combination to generate those warnings; no Resource Leak false positives were associated with that combination.Source: 2) Option 2 – SkipAnalyzer · p. 7
  • SkipAnalyzer’s false-positive filter outperforms the previous state-of-the-art warning-removal baselines.The comparison with existing baselines is reported for warnings generated by Infer and SkipAnalyzer.Source: 2) Option 2 – SkipAnalyzer · p. 7, 1) Option 1 – Infer · p. 7

C. RQ3: Performance of SkipAnalyzer on Bug Repair

SkipAnalyzer generates patches for detected bugs and is compared with VulRepair; the discussion also examines patterns in missed detections. Patch correctness is assessed manually.

C. RQ3: Performance of SkipAnalyzer on Bug Repair

  • 78.91% for Null Dereference bugs and 78.87% for Resource Leak bugs are the respective logic-rate increases over VulRepair reported for SkipAnalyzer.SkipAnalyzer does not require training or fine-tuning, unlike VulRepair, which the authors adapted and trained on their dataset.Source: C. RQ3: Performance of SkipAnalyzer on… · pp. 7–8
  • SkipAnalyzer receives true-positive buggy snippets and their warnings, generates patches in zero-shot mode, and has patches manually assessed for syntax and logic.The evaluation uses ChatGPT-3.5 Turbo and ChatGPT-4; one-shot and few-shot prompting are not used for repair.Source: C. RQ3: Performance of SkipAnalyzer on… · p. 7

A. Reasons for Missing Detecting Bugs

1) Null Dereference:

The missed Null Dereference cases cluster around related objects in null checks, assumptions about entities outside the method’s scope, and chained method calls.

1) Null Dereference:

  • SkipAnalyzer may miss a dereference inside a null check for another object, especially when the objects are related through a method-call argument.Figure 2 is given as an example of this pattern.Source: 1) Null Dereference · p. 8, Figure 2 · p. 9
  • Limited access to implementations outside the current method can lead SkipAnalyzer to assume incorrectly that an entity’s return value is non-null.Figure 3 illustrates an example attributed to this issue.Source: 1) Null Dereference · pp. 8–9, Figure 3 · p. 9
  • SkipAnalyzer may detect the first object in a method-call chain yet miss later calls that could also return null.Figure 4 illustrates a missed Null Dereference in a method-call chain.Source: 1) Null Dereference · p. 9, Figure 4 · p. 9

2) Resource Leak:

Missed Resource Leak cases commonly involve partially released resources. The study’s validity is bounded by its Java-only, Infer-generated dataset and its use of two OpenAI models.

2) Resource Leak:

  • Resource Leak misses commonly occur when some resources are released but others remain unreleased.Figure 5 illustrates a case where tempDir is handled but asarZip, which lacks try-with-resources syntax, is missed.Source: 2) Resource Leak · p. 9, Figure 5 · p. 10

VIII. THREATS TO VALIDITY

  • The evaluation covers Java projects only because Infer was used to generate warnings and the dataset.The authors note that a similar analysis could be conducted in other Infer-compatible languages, including C and C#.Source: §VIII · p. 9
  • The dataset uses warnings from Infer, not alternative detectors such as SpotBugs or Error-Prone, to maintain bug-classification consistency.The authors also state that Infer has better precision and outperforms those tools in accuracy.Source: §VIII · pp. 9–10
  • Only ChatGPT-4 and ChatGPT-3.5 Turbo were evaluated; other LLMs may have different performance characteristics.Source: §VIII · p. 10
  • SkipAnalyzer’s performance may vary across project sets, although the study includes projects from varied repositories and backgrounds.Source: §VIII · p. 10

IX. RELATED WORK

Prior LLM work in software engineering spans repair, testing and fuzzing, oracle generation, and mobile applications. SkipAnalyzer applies LLMs to static analysis, with future work targeting broader models, bug types, and coding practices.

A. LLM Applications in Software Engineering

  • Prior LLM software-engineering research covers automated repair, testing and fuzzing, oracle generation, and mobile applications.Examples include LLM-based fuzzers, test generation from bug reports, assertion and test-statement generation, and extracting steps to reproduce Android bugs.Source: A. LLM Applications in Software… · p. 10

X. CONCLUSION

  • SkipAnalyzer’s components outperform baseline counterparts while offering advantages in cost and complexity.The conclusion presents it as an LLM-based tool for static bug detection, false-positive warning removal, and bug repair, using Infer warnings from open-source Java and prior-research projects and different ChatGPT models and prompting strategies.Source: §X · p. 10
  • Future work would broaden the evaluated LLMs and bug types, compare fine-tuning with task-specialized prompting, and add analysis of coding practices and styles.The authors name Llama, Bard, and PaLM2 as potential additional models.Source: §X · pp. 10–11

Related papers

Loading 2310.18532v2…