PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Pre-trained Language Models for Vulnerability Detection

Youpeng Li, Weiliang Qi, Xuyu Wang, Fuxun Yu, Xinda Wang

arXiv:2507.16887v2cs.CRcs.AIcs.LGcs.SE

TL;DR

Real-world vulnerability detection remains challenging, and prior evaluations have not adequately addressed data preparation and practical settings. RevisitVD evaluates PLMs with newly constructed datasets across adaptation, generalization, and robustness settings; code-syntax-and-semantics pretraining performs best, but practical challenges remain.

  • Problem

    Existing PLM vulnerability-detection evaluations inadequately address data preparation and evaluation settings, while detecting complex, diverse, and subtle real-world vulnerabilities remains challenging.

    Source: §1 · p. 1

  • Method

    RevisitVD evaluates PLMs on newly constructed datasets across fine-tuning, prompting, training and testing settings, and code perturbations.

    Source: §1 · pp. 1–2

  • Results

    PLMs with pretraining designed to capture code syntax and semantics outperform general-language-modeling PLMs, but struggle with complex dependencies, perturbations, and truncation-related labeling errors.

    Source: §1 · p. 2, §4.3 · p. 9, §4.6 · p. 11

  • Takeaways & Limitations

    The findings suggest future research could focus on specialized pretraining tasks for vulnerability detection while considering performance-efficiency trade-offs.

    Source: §1 · p. 2

    Complex-dependency vulnerabilities may require external or cross-procedure information, while truncation can mislead training with noisy labels.

    Source: §4.3 · p. 9, §4.6 · p. 11

Abstract

from arXiv · show

The rapid advancement of pre-trained language models (PLMs) has demonstrated promising results for various code-related tasks. However, their effectiveness in detecting real-world vulnerabilities remains a critical challenge. % for the security community. While existing empirical studies evaluate PLMs for vulnerability detection (VD), their inadequate consideration in data preparation, evaluation setups, and experimental settings undermines the accuracy and comprehensiveness of evaluations. This paper introduces RevisitVD, an extensive evaluation of 17 PLMs spanning smaller code-specific PLMs and large-scale PLMs using newly constructed datasets. Specifically, we compare the performance of PLMs under both fine-tuning and prompt engineering, assess their effectiveness and generalizability across various training and testing settings, and analyze their robustness against code normalization, abstraction, and semantic-preserving transformations. Our findings reveal that, for VD tasks, PLMs incorporating pre-training tasks designed to capture the syntactic and semantic patterns of code outperform both general-purpose PLMs and those solely pre-trained or fine-tuned on large code corpora. However, these models face notable challenges in real-world scenarios, such as difficulties in detecting vulnerabilities with complex dependencies, handling perturbations introduced by code normalization and abstraction, and identifying semantic-preserving vulnerable code transformations. Also, the truncation caused by the limited context windows of PLMs can lead to a non-negligible amount of labeling errors. This study underscores the importance of thorough evaluations of model performance in practical scenarios and outlines future directions to help enhance the effectiveness of PLMs for realistic VD applications.

1 Introduction

RevisitVD addresses weaknesses in vulnerability-detection evaluations with updated datasets and broad model comparisons. Its findings favor code-syntax-and-semantics pretraining, while identifying practical robustness and context challenges.

Evaluation gap

  • Existing evaluations risk data leakage through duplicated code patterns and temporal overlap between test commits and PLM pretraining data.Other studies also use limited model, dataset, or vulnerability-type coverage and neglect practical generalization and robustness.Source: §1 · p. 1

Evaluation approach

  • RevisitVD reconstructs a dataset and uses commit-time partitioning to reduce leakage, while collecting post-pretraining-cutoff C/C++ vulnerability data from NVD.The newly collected samples span vulnerability types and projects.Source: §1 · pp. 1–2
  • The study compares 18 PLMs using fine-tuning and prompt engineering, and tests generalization and robustness across out-of-distribution data and code perturbations.Prompting includes structural and semantic-aware formats, and the study also examines chain-of-thought reasoning.Source: §1 · p. 2

Key findings

  • Code-syntax-and-semantics pretraining significantly outperforms most general-language-modeling PLMs, even when the specialized models have far fewer parameters.The evaluation also finds challenges with complex dependencies, perturbations, and limited context windows.Source: §1 · p. 2

Contributions

  • The paper identifies data leakage, limited evaluation scope, and superficial practical analysis as shortcomings its evaluation is designed to address.It also presents an extensible framework and newly collected datasets for future evaluation.Source: §1 · p. 2

2 Background and Related Work

The section reviews PLM objectives, code-specific pretraining approaches, and weaknesses in prior vulnerability-detection evaluations. RevisitVD responds with broader model, adaptation, and practical robustness comparisons.

2.1 Pre-trained Language Models

  • Encoder PLMs commonly use masked language modeling, while decoder PLMs use causal language modeling to predict tokens from preceding context.The section describes the Transformer as using self-attention to capture dependencies across sequences.Source: §2.1 · p. 2

2.2 Pre-trained Language Models for Code

  • Code PLMs use either code-structure-aware pretraining tasks or general language modeling on large code corpora, with Code SLMs and LLMs distinguished by scale and approach.Examples include AST, data-flow, dependency, and identifier-aware tasks; code LLMs may use fill-in-the-middle training and long-context adaptations.Source: §2.2 · pp. 2–3

2.3 Evaluating PLMs for Vulnerability Detection

  • Prior evaluations have limitations in scope, data partitioning, and analysis of real-world generalization and robustness.Studies may rely on random splits, overlook pretraining-data overlap, or use narrow model and dataset coverage.Source: §2.3 · pp. 3–4
  • RevisitVD evaluates 18 PLMs across fine-tuning and prompt engineering, including structural prompts, out-of-distribution data, and code perturbations.The framework includes open-source models up to 34 billion parameters and examines normalization, abstraction, semantic-preserving transformations, and truncation.Source: §2.3 · p. 4

3 Experimental Setup

The setup combines reconstructed vulnerability data, time- and CWE-aware partitioning, and a post-cutoff dataset collected from NVD. It also includes a balanced training variant and an out-of-distribution test set.

3.1.1 Dataset Selection.

  • The evaluation dataset is based on PrimeVul, whose refined labeling rules improve labeling accuracy by 26%–67%.Existing benchmarks can have limited diversity, synthetic samples, unrealistic class balance, or inaccurate labels.Source: §3.1.1 · p. 4

3.1.2 Dataset Partitioning.

  • The partitioning method groups samples by CWE, sorts them by commit date, and splits within each CWE at an 8:1:1 ratio to preserve type distribution and reduce leakage.The authors report no leakage for the two described cases involving patched counterparts and similar fixes across splits.Source: §3.1.2 · p. 4

3.2 Self-Collected Dataset

  • The self-collected dataset contains 25,536 functions from 99 projects across 28 CWE types, with 646 vulnerable functions, collected from October 2023 to October 2024.Its commit dates postdate the evaluated PLMs’ pretraining cutoffs, and it serves as an out-of-distribution test set.Source: §3.2 · pp. 4–5, §3.3 · p. 5, Figure 1 · p. 5

3.3 Statistics of the Datasets Used in Evaluations

  • A balanced training set is created by undersampling the reconstructed training data to a 1:1 vulnerable-to-non-vulnerable ratio, while validation and test sets remain unchanged.Table 2 presents statistics for the datasets used in the evaluation.Source: §3.3 · p. 5, Table 2 · p. 5

3.4 Evaluated Models

The evaluation covers code-oriented PLMs across encoder, decoder, and encoder-decoder architectures, and tests prompting with different examples and code representations. Prompt variants add structural information, API-call sequences, or data-flow descriptions alongside raw code.

3.4 Evaluated Models

  • The evaluated models span encoder-based, decoder-based, and encoder-decoder architectures, including models from millions to billions of parameters.The selection focuses on PLMs oriented toward code tasks or exposed to substantial code data during pre-training.Source: §3.4 · p. 5

3.5 Prompt Design

  • Prompts use either zero-shot prediction or four-shot in-context learning with randomly selected pre- and post-patched function pairs.Each few-shot example includes a query and its ground-truth answer before the target function.Source: §3.5 · p. 5
  • Four prompt types provide raw code alone or add flattened AST structure, API-call sequences, or data-flow descriptions.Tree-sitter extracts code structure for serialization as plain text; the data-flow description uses relative variable positions to avoid misalignment with LLM tokenization.Source: §3.5 · pp. 5–6
  • Prompt templates pair a code-security-expert system role with a user question, the input function, and the model's default chat format.The user asks whether the function is vulnerable and requests a Yes or No answer.Source: §3.5 · p. 6

3.6 Metrics

The evaluation uses balanced accuracy as its main metric and describes the framework and settings used to run fine-tuning and prompting experiments.

3.6 Metrics

  • Balanced accuracy is the main metric because the evaluation aims to balance true-positive and true-negative predictions.The paper also defines accuracy, recall, precision, true negative rate, and F1.Source: §3.6 · p. 6

3.7.1 Evaluation Framework.

  • The evaluation framework uses full fine-tuning for SLMs, LoRA for LLMs, and the Azure OpenAI API for GPT inference.Open-source models are loaded with Hugging Face Transformers; DeepSpeed ZeRO-3 and Accelerate support LLM training efficiency.Source: §3.7.1 · p. 6

3.7.2 Hyperparameters.

  • Prompting uses top-p 0.9 and temperature 0, except GPT-5 Mini, whose unmodifiable temperature remains at its default 1.0.Reasoning effort is medium and verbosity low; the settings aim for concise, deterministic answers.Source: §3.7.2 · p. 6

4 Research Questions (RQs) and Findings

The research questions examine how fine-tuning, pre-training, data and code changes, context limits, and prompting affect vulnerability detection. The findings highlight performance sensitivities and model-dependent effects of structural information, complex prompts, and few-shot examples.

4.1 RQ1: How do PLMs perform across various training and testing configurations?

  • Balanced training yields better recall–TNR trade-offs than imbalanced training, while balanced accuracy drops about 5.9% on a self-collected test set.Imbalanced training produces over 99% average TNR but below 10% average recall; balanced training on the reconstructed test set yields 73.5% average TNR, 75.3% average recall, and 74.4% average balanced accuracy. PDBERT leads other Code SLMs by about 3.2% to 4.7% in average balanced accuracy, attributed to dependency-focused pre-training; F1 can misleadingly favor imbalanced training.Source: §4.1 · pp. 6–7, Table 4 · p. 7

4.2 RQ2: How do PLMs of varying parameter scales perform under both fine-tuning and prompt engineering?

  • Fine-tuned PLMs outperform prompting-based PLMs by about 14% overall on vulnerability detection.GPT-4o Mini and DeepSeek-Coder-34B perform best in prompting, while PDBERT and DeepSeek-Coder-34B lead fine-tuning; GPT-5 Mini still fails to outperform most fine-tuned PLMs. Incorporating code-specific optimization during pre-training is more beneficial than pre-training solely on large code corpora with standard language modeling.Source: §4.2 · pp. 7–9, Table 5 · p. 8

4.3 RQ3: How do PLMs perform across various types of vulnerabilities?

  • PLMs achieve over 70% average recall across vulnerability types, but struggle with flaws requiring complex dependencies or external context.PDBERT performs significantly better on CWE-119, CWE-200, CWE-189, and CWE-362; its dependency-aware pre-training may help it track control and data flow. CWE-264 and CWE-476 are challenging because detection can require permission-scheme or cross-procedure information.Source: §4.3 · p. 9, Figure 3 · p. 9

4.4 RQ4: How do PLMs perform under different code normalization rules?

  • Inconsistent training and test normalization degrades performance; removing newlines when evaluating PDBERT causes an approximately 6.9% drop.PLMs generally perform best when the same normalization rules are used for training and testing, indicating sensitivity to minor formatting perturbations.Source: §4.4 · pp. 9–10, Figure 4 · p. 9

4.5 RQ5: Are PLMs capable of understanding abstracted vulnerability code?

  • After identifier and string abstraction, PLMs’ average balanced accuracy drops by 1%, with false positives decreasing and false negatives increasing.The results suggest that predictions do not rely solely on textual content; code understanding and learned dependency analysis also support detection.Source: §4.5 · p. 10, Figure 5 · p. 10

4.6 RQ6: Are PLMs capable of detecting vulnerable code after semantic-preserving transformation?

  • Semantic-preserving transformations reduce PLM performance, with the smallest average drop for conditional branch expansion (1.3%) and the largest for branch negation (15.8%).UniXCoder’s average drop is only 0.1%, attributed to multimodal contrastive pre-training; most PLMs are not yet reliable for detecting transformed vulnerable clones and reused code.Source: Figure 6 · p. 10, Figure 7 · p. 10, §4.6 · p. 11

4.7 RQ7: To what extent do implicit labeling errors caused by token truncation affect the performance of Code SLMs?

  • Token truncation creates opposite labels for identical visible inputs in 1,473 of 5,480 PrimeVul patch pairs (27%); slicing improves average performance by 4%.Slicing uses vulnerability-related anchors and bidirectional data/control-flow analysis to retain relevant code within the 512-token limit; truncation-related labeling errors also remain a concern for repository-level detection.Source: §4.6 · pp. 11–12, Figure 9 · p. 11, Figure 8 · p. 11

4.8 RQ8: How do PLMs perform under different prompt types and settings?

  • Prompt-type benefits depend on model capability, while few-shot examples can help GPT-4o Mini but disrupt strong zero-shot Code LLMs.GPT-4o Mini performs best with API calls and few-shot examples; DeepSeek-Coder typically benefits from API-call and data-flow information, whereas CodeLlama and StarChat-𝛽 can struggle with added logic chains. Limited examples may encourage DeepSeek-Coder or WizardCoder to overfit simplistic patterns.Source: §4.8 · p. 12, Figure 10 · p. 12
  • Prompt effectiveness depends on code understanding: structural information can benefit models such as GPT-4o Mini, while complex prompts and few-shot examples may hinder weaker models through superficial-pattern overfitting.Limited few-shot examples may not capture the complex and diverse patterns of vulnerabilities.Source: §4.8 · p. 12

5 Discussion

RevisitVD argues that biased prior evaluations widen the gap between reported vulnerability-detection performance and practical effectiveness. It also identifies limits from residual data leakage, model coverage, and context windows.

5 Discussion

  • The authors argue that biased prior evaluations yield conclusions that do not reflect practical effectiveness.They identify a significant gap between reported performance and practical effectiveness.Source: §5 · p. 12
  • Some undetectable data leakage may remain because addressing it is beyond the study’s scope.The dataset is limited to commits dated after the evaluated LLMs’ knowledge cutoffs.Source: §5 · p. 12
  • Resource and budget constraints prevented evaluation of all state-of-the-art LLMs, while recent model cutoffs and scarce recent vulnerabilities limit evaluation data.The authors say their framework can help future work evaluate stronger LLMs on more recent vulnerability data.Source: §5 · p. 12
  • Code SLMs such as PDBERT generally outperform general-language-model LLMs, but their applicability is constrained to function-level vulnerability detection.LLMs offer larger context windows, yet longer function-level and cross-procedure detection still needs substantial improvement.Source: §5 · p. 12

6 Conclusion

The conclusion reports that code models with pre-training designed to capture code syntax and semantics outperform models trained only on general language modeling. It also highlights persistent challenges involving normalization, transformation, complex dependencies, and context limits.

6 Conclusion

  • PLMs with pre-training designed to capture code syntax and semantics outperform those trained solely on general language modeling.The evaluation also finds sensitivity to code normalization and transformation, and difficulty with complex dependencies and code exceeding context windows.Source: §6 · p. 12
  • PLMs remain sensitive to code normalization and transformation.The conclusion names these as continuing challenges for vulnerability detection.Source: §6 · p. 12
  • Complex dependencies and code exceeding model context windows remain challenging for vulnerability detection.Source: §6 · p. 12

A Ethical Considerations

The study evaluates existing PLMs using publicly available data, models, and source code. It reports that it generates or identifies no new vulnerabilities and therefore raises no ethical issues.

A Ethical Considerations

B.1 Small Language Models for Code (Code SLMs)

This subsection describes code-focused pre-trained models and the objectives used to learn code representations. It covers models designed around general code-language alignment, code structure, dependency analysis, and identifier-aware learning.

B.1 Small Language Models for Code (Code SLMs)

  • PDBERT [40] uses statement-level control-dependency and token-level data-dependency prediction to support dependency analysis.The passage reports strong results on vulnerability detection, classification, assessment, and program dependence analysis.Source: §B.1 · p. 15
  • CodeBERT [19] learns general-purpose programming- and natural-language representations using replaced-token detection.The passage notes performance in code search and documentation generation.Source: §B.1 · p. 15
  • GraphCodeBERT [25] uses data flow and edge-prediction and node-alignment objectives to incorporate code structure.Source: §B.1 · p. 15
  • UniXCoder [24] combines mask-attention matrices and prefix adapters with cross-modal content such as ASTs and comments.Source: §B.1 · p. 15
  • CodeT5 [67] uses identifier-aware pre-training and comment-informed dual generation to leverage code semantics.Source: §B.1 · p. 15

B.2 Large Language Models for Code (Code LLMs)

The section describes code LLMs built from code-focused base models, with varied pre-training and instruction-tuning approaches. Several are designed to handle long contexts or improve coding-assistant behavior.

B.2 Large Language Models for Code (Code LLMs)

  • CodeLlama combines autoregressive and causal-infilling prediction, while RoPE parameter adjustments support long input contexts.CodeLlama-Instruct is further fine-tuned on proprietary instruction data and a machine-generated self-instruct dataset.Source: §B.2 · p. 15
  • DeepSeek-Coder uses fill-in-the-middle pre-training on project-level code and English and Chinese natural-language corpora, with RoPE changes extending sequence length.Its instruction variant incorporates human instruction data; the passage reports improved code-benchmark performance for CodeLlama-Instruct and DeepSeek-Coder-Instruct, respectively.Source: §B.2 · p. 15
  • StarChat-β builds on StarCoderPlus, which was trained on an uncensored OpenAssistant-Guanaco variant to improve its coding-assistant capabilities.StarCoderBase was pre-trained on the Stack dataset, and StarCoder was fine-tuned on 35B Python tokens.Source: §B.2 · p. 15
  • WizardCoder applies Evol-Instruct to construct its instruction-following fine-tuning set and showed strong results against open-source code LLMs on four code-generation benchmarks.The benchmarks are HumanEval, HumanEval+, MBPP, and DS1000.Source: §B.2 · p. 15
Loading 2507.16887v2…