PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Ahead

Xin Zhou, Sicong Cao, Xiaobing Sun, David Lo

arXiv:2404.02525v3cs.SE

TL;DR

Research on LLMs for vulnerability detection and repair is growing, but no comprehensive survey had summarized the field’s approaches, limitations, and future directions. This paper systematically reviews 58 primary studies, categorizes the LLMs and adaptation techniques used for both tasks, and identifies larger-granularity and robustness limitations alongside a future roadmap.

  • Problem

    No comprehensive literature review had summarized LLM-based vulnerability detection and repair approaches, their limitations, and future research directions.

  • Method

    The paper conducts a systematic review of 58 primary studies, summarizing employed LLMs and categorizing adaptation techniques for vulnerability detection and repair.

  • Results

    Encoder-only LLMs dominate vulnerability detection, while commercial and decoder-only LLMs are prominent in vulnerability repair; detection adaptation uses fine-tuning, prompt engineering, and retrieval augmentation, whereas repair uses fine-tuning and prompt engineering.

  • Takeaways & Limitations

    The review identifies limitations in current LLM-based vulnerability detection and repair research and proposes a roadmap for future research.

  • Takeaways & Limitations

    Research on larger input granularities remains limited, with no studies focused on class-level detection or repair and none investigating repository-level repair.

Abstract

from arXiv · show

The significant advancements in Large Language Models (LLMs) have resulted in their widespread adoption across various tasks within Software Engineering (SE), including vulnerability detection and repair. Numerous studies have investigated the application of LLMs to enhance vulnerability detection and repair tasks. Despite the increasing research interest, there is currently no existing survey that focuses on the utilization of LLMs for vulnerability detection and repair. In this paper, we aim to bridge this gap by offering a systematic literature review of approaches aimed at improving vulnerability detection and repair through the utilization of LLMs. The review encompasses research work from leading SE, AI, and Security conferences and journals, encompassing 43 papers published across 25 distinct venues, along with 15 high-quality preprint papers, bringing the total to 58 papers. By answering three key research questions, we aim to (1) summarize the LLMs employed in the relevant literature, (2) categorize various LLM adaptation techniques in vulnerability detection, and (3) classify various LLM adaptation techniques in vulnerability repair. Based on our findings, we have identified a series of limitations of existing studies. Additionally, we have outlined a roadmap highlighting potential opportunities that we believe are pertinent and crucial for future research endeavors.

1 INTRODUCTION

Software vulnerabilities are increasing, while traditional detection and repair techniques face false positives and limited coverage. This survey addresses the lack of a comprehensive review by systematically analyzing LLM-based vulnerability detection and repair research.

  • Traditional vulnerability detectors can produce high false-positive rates, while program-analysis repair tools struggle with diverse vulnerability types.
  • LLM-based approaches have shown promising outcomes for automated vulnerability detection and repair.
  • LLMs may learn features from known vulnerabilities, find or fix unseen vulnerabilities, and use knowledge from large-scale pre-training.
  • The review analyzes 58 primary studies, summarizes employed LLMs, categorizes adaptation techniques, discusses limitations, and proposes future research directions.

2 BACKGROUND AND PRELIMINARIES

The paper frames vulnerability detection and repair as source-code tasks and introduces LLM architectures relevant to their study. Its background also situates the review within a structured research process.

  • Vulnerability detection is framed as binary classification, mapping an input source-code function X_i to vulnerable or non-vulnerable output Y_i.
  • The background includes the survey’s study-identification and selection process as part of its review structure.
  • The paper categorizes mainstream LLMs into encoder-only, encoder-decoder, and decoder-only architectures.

3 REVIEW METHODOLOGY

The review uses four research questions, searches major SE, AI, and Security venues and databases, applies eligibility and quality criteria, and supplements selection with snowballing. The resulting corpus shows sharply increasing recent interest.

  • Research questions: The review asks which LLMs are used, how they are adapted for detection and repair, and what datasets and deployment strategies characterize the studies.
  • Search strategy: The search covers publications from 2018 onward across 13 conferences, 4 journals, and 7 academic databases.
  • Eligibility and quality assessment: Studies had to be English full-text research papers using LLM techniques for source-code vulnerability detection or repair, while surveys and several non-research formats were excluded.
  • Eligibility and quality assessment: Quality assessment scored five criteria from 0 to 3 and retained papers reaching 12 points, or 80% of the maximum.
  • Study selection: 42 papers were obtained before snowballing, after which additional studies were gathered and the selection process repeated.
  • Study distribution: 46.6% of studied papers appeared in 2024, and ICSE contributed 20.7% of the studies.

4 RQ1: WHAT LLMS HAVE BEEN UTILIZED?

The review identifies 37 distinct LLMs across vulnerability detection and repair studies, with different model architectures and sizes predominating in each task. Encoder-only models dominate detection, whereas commercial and decoder-only models are prominent in repair.

  • 37 distinct LLMs were utilized across the 58 included studies.
  • LLMs Used for Vulnerability Detection: CodeBERT was predominant for vulnerability detection, representing 26.1% (24/92) of LLM usage, followed by GPT-3.5 at 9.8% (9/92).
  • LLMs Used for Vulnerability Detection: Encoder-only LLMs comprised 47.8% (44/92) of detection usage, compared with 23.9% (22/92) for decoder-only models and 9.8% (9/92) for encoder-decoder models.
  • LLMs Used for Vulnerability Repair: CodeT5 was predominant for vulnerability repair at 15.6% (7/45), followed by GPT-3.5 at 11.1% (5/45).
  • LLMs Used for Vulnerability Repair: Commercial LLMs with undisclosed architectures accounted for 35.6% (16/45) of repair usage, while decoder LLMs comprised 31.1% (14/45).
  • Detection studies predominantly used lightweight encoder-only models, whereas repair studies more often used larger models, including several exceeding 1 billion parameters.

5 RQ2: HOW ARE LLMS ADAPTED FOR VULNERABILITY DETECTION?

The review groups vulnerability-detection adaptation into fine-tuning, prompt engineering, and retrieval augmentation. Fine-tuning dominates the literature, with advanced methods targeting data, program analysis, model design, domain-specific pre-training, and causal learning.

  • Three adaptation techniques are identified for vulnerability detection: fine-tuning, prompt engineering, and retrieval augmentation.
  • 5.1 Fine-tuning: Fine-tuning updates LLM parameters using labeled vulnerable and non-vulnerable code samples, whereas prompt engineering and retrieval augmentation do not change parameters.
  • 5.1 Fine-tuning: Fine-tuning adaptations are classified into data-centric innovations, program-analysis combinations, additional deep-learning modules, domain-specific pre-training, and causal learning.
  • 5.1 Fine-tuning: Data-centric methods address imbalanced labels, noisy or incorrect labels, and scarce labeled data in vulnerability-detection datasets.
  • 5.1 Fine-tuning: Domain-specific pre-training adapts LLMs using vulnerability-related data before task-specific fine-tuning, while causal learning targets spurious features and robustness under perturbation or out-of-distribution data.
  • 73% of vulnerability-detection studies used fine-tuning, compared with approximately 17% using prompt engineering and 10% using retrieval augmentation.

6 RQ3: HOW ARE LLMS ADAPTED FOR VULNERABILITY REPAIR?

The review identifies fine-tuning and prompt engineering as the two main ways LLMs are adapted for vulnerability repair. Fine-tuning spans data, model, pre-training, and reinforcement-learning innovations, while prompting supplies vulnerability-specific guidance and examples.

  • Fine-tuning: Advanced fine-tuning techniques are categorized into data-centric, model-centric, domain-specific pre-training, and reinforcement-learning innovations.
  • Fine-tuning: Data-centric methods augment vulnerable code with inputs such as abstract syntax trees, vulnerability descriptions, examples, and vulnerability-inducing or fixing commits.
  • Fine-tuning: Model-centric methods modify Transformer-based architectures, including vulnerability queries, vulnerability masks, and simultaneous fix generation with developer-friendly explanations.
  • Fine-tuning: Domain-specific pre-training uses bug-fixing corpora before vulnerability-fix fine-tuning, while SecureCode combines CodeBLEU and BERTScore rewards with PPO.
  • Prompt engineering: Prompt engineering uses zero-shot and few-shot strategies incorporating vulnerability descriptions, locations, auxiliary information, program analysis, or repair examples.
  • Fine-tuning accounts for approximately 63% of vulnerability-repair studies, while prompt engineering accounts for approximately 37%.

7 RQ4: WHAT ARE THE CHARACTERISTICS OF DATASETS AND DEPLOYMENT?

The reviewed studies primarily use function- or line-level data, with limited larger-granularity coverage, heuristic detection labels, and repair datasets lacking tests. Deployment remains largely offline, with little developer interaction or workflow integration.

  • Input Granularity: Function-level inputs dominate vulnerability detection, only 7 studies address line-level detection, and all reviewed vulnerability-repair studies target function-level repairs.
  • Input Granularity: There are no studies on class-level vulnerability detection or repair, and none investigate repository-level vulnerability repair.
  • Data Quality: Vulnerability repair datasets frequently lack test cases, making repair correctness difficult to evaluate when multiple fixes are possible.
  • Data Quality: Vulnerability detection datasets mainly use heuristic-based labels, which can introduce noisy or incorrect classifications.
  • Deployment Strategies: Limited studies support developer communication, and all reviewed approaches are evaluated offline rather than integrated into developers’ workflows and tools.
  • Deployment Strategies: The review therefore identifies broader input granularity, higher-quality labeled and tested datasets, developer interaction, and workflow integration as key deployment and dataset gaps.

8 THE ROAD AHEAD

The review identifies limitations in current LLM-based vulnerability detection and repair, including narrow inputs, imperfect data and performance, limited model scale, and absent deployment integration. It proposes future work spanning higher-quality benchmarks, repository-level analysis, vulnerability-specialized models, advanced adaptation, and deployment-ready tools.

  • Limitations: Function- and line-level inputs can miss vulnerabilities spanning multiple functions or classes and constrain repository-wide repairs.The review calls for approaches that handle broader program units, including classes and whole repositories.
  • Limitations: Existing datasets contain noisy labels, synthetic examples, missing test cases, and possible overlap with LLM pre-training corpora.The authors call for high-quality real-world vulnerability-fixing datasets with test cases and benchmarks separated from pre-training data.
  • Limitations: Complexity reduces performance for inter-procedural, infrequent CWE, and multi-code-unit vulnerabilities.The review notes that future solutions should account for vulnerabilities spanning multiple functions and other complex structures.
  • Limitations: Most studies use lightweight LLMs, leaving large-model exploration comparatively limited.Reported adaptation strategies include data- and model-centric methods, program analysis, domain-specific pre-training, causal learning, and reinforcement learning.
  • Limitations: No reviewed study had integrated an LLM-based solution into developers’ workflows, and few incorporated developer interaction.The review recommends collaboration, trust-building, and seamless workflow integration.
  • Limitations: 67.6% and 20% accuracy scores were reported for vulnerability detection and repair, respectively, while LLMs also showed weakness against data perturbations.The review identifies accuracy and robustness as unresolved requirements.
  • Opportunities: Future research should curate high-quality detection test sets and pursue repository-level detection and repair using models with longer input capacities.The review contrasts small models’ 512-subtoken limit with GPT-4’s stated 128k-subtoken capacity.
  • Opportunities: The roadmap also calls for open vulnerability-specialized LLMs, advanced retrieval and agentic adaptation, and deployment features such as feedback, explanations, and IDE integration.Under-explored retrieval approaches include iterative, recursive, and adaptive retrieval augmentation.

9 THREATS TO VALIDITY

The review may have omitted relevant studies because vulnerability and LLM terminology is varied and keyword coverage may be incomplete. The authors mitigated this risk through venue selection, keyword expansion, automated search, and forward-backward snowballing.

  • Threats to Validity: Incomplete keywords and varied terminology could have caused relevant vulnerability-detection and repair studies to be missed.The authors identify study exclusion during literature search and selection as the central validity threat.
  • Mitigation: The review mitigated search omissions by manually selecting 17 high-impact venues, extracting standard keywords, and adding forward-backward snowballing.

10 CONCLUSION AND FUTURE WORK

The paper reviews 58 primary studies on LLMs for vulnerability detection and repair, analyzing employed models and adaptation techniques. It also identifies limitations and outlines future research opportunities.

  • Conclusion: 58 primary studies were reviewed to characterize LLM choices and adaptation techniques for vulnerability detection and repair.
  • Future Work: The review identifies limitations in existing studies and proposes future research directions for LLM-based vulnerability detection and repair.
Loading 2404.02525v3…