PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformer-Based Language Models for Software Vulnerability Detection

Chandra Thapa, Seung Ick Jang, Muhammad Ejaz Ahmed, Seyit Camtepe, Josef Pieprzyk, Surya Nepal

arXiv:2204.03214v2cs.CRcs.AIcs.LG

TL;DR

The paper addresses how transformer-based language models can be used for software vulnerability detection and how well they perform relative to contemporary RNN-based models. It presents a systematic framework and evaluates models on C/C++ vulnerability datasets, finding stronger performance while also examining platforms and practical training constraints.

  • Problem

    The paper studies how to leverage transformer-based language models for software vulnerability detection and how they perform compared with contemporary RNN-based models.

  • Method

    The paper presents a systematic framework covering source-code translation, model preparation, inference, data cleaning, comparative experiments, and platform analysis for transformer models on C/C++ vulnerability datasets.

  • Results

    Transformer-based models outperformed BiLSTM and BiGRU across reported performance metrics for buffer and resource management errors, with GPT-2 Large and GPT-2 XL achieving the best binary and global-average multi-class F1-scores, respectively.

  • Takeaways & Limitations

    The study supports using transformer-based language models for software vulnerability detection while considering model size, data preparation, and platform requirements.

  • Takeaways & Limitations

    Platform use is constrained by hardware capacity, administrative restrictions, and framework limitations such as Horovod’s inability to train models that do not fit on a single GPU.

Abstract

from arXiv · show

The large transformer-based language models demonstrate excellent performance in natural language processing. By considering the transferability of the knowledge gained by these models in one domain to other related domains, and the closeness of natural languages to high-level programming languages, such as C/C++, this work studies how to leverage (large) transformer-based language models in detecting software vulnerabilities and how good are these models for vulnerability detection tasks. In this regard, firstly, a systematic (cohesive) framework that details source code translation, model preparation, and inference is presented. Then, an empirical analysis is performed with software vulnerability datasets with C/C++ source codes having multiple vulnerabilities corresponding to the library function call, pointer usage, array usage, and arithmetic expression. Our empirical results demonstrate the good performance of the language models in vulnerability detection. Moreover, these language models have better performance metrics, such as F1-score, than the contemporary models, namely bidirectional long short-term memory and bidirectional gated recurrent unit. Experimenting with the language models is always challenging due to the requirement of computing resources, platforms, libraries, and dependencies. Thus, this paper also analyses the popular platforms to efficiently fine-tune these models and present recommendations while choosing the platforms.

1 INTRODUCTION

The paper investigates whether transformer-based language models can extend beyond NLP to software vulnerability detection, addressing framework, performance, and platform questions for C/C++ code. It motivates this direction through transfer learning, similarities between natural and programming languages, and practical needs for scalable vulnerability detection.

  • Motivation: Transformer-based models outperform RNN-based architectures in NLP and support parallel computation and transfer learning to related domains.These properties motivate applying them beyond NLP, including software vulnerability detection.
  • Motivation: Source-level vulnerability detection is needed because software vulnerabilities can let adversaries control execution or steal and modify data.The paper cites buffer overflow vulnerabilities affecting devices, operating systems, and applications.
  • Motivation: Transformer-based vulnerability detection can automate detection beyond heuristic static-analysis methods and reduce extensive feature-engineering requirements.The paper contrasts these benefits with manual operations and classical machine learning requirements.
  • Research questions: The study asks how to build a systematic framework, how transformer models compare with contemporary RNN models, and which platforms run them efficiently.These questions arise because performance beyond BERT and the practical demands of training and fine-tuning other models remain unclear.
  • Study setting: C/C++ is selected because programming languages share structure, syntax, contextual meaning, and semantics with natural languages.The paper presents these similarities as factors enabling transformer-based language models for vulnerability detection.
  • Contributions: The framework covers source-code translation, model descriptions and preparation, inference, comparative evaluation, data cleaning, fine-tuning time, and platform analysis.Experiments cover binary and multi-class C/C++ vulnerability tasks involving multiple vulnerability categories.

2 SYSTEMATIC FRAMEWORK

The framework translates C/C++ source into code gadgets, cleans and preprocesses the data, partitions datasets, and prepares transformer and recurrent models for vulnerability classification. It uses dependency-based gadgets, vectorized representations, and a range of model architectures and sizes.

  • Code gadgets and extraction: The framework loads and normalizes C/C++ files, extracts function and variable relationships, and backtracks library/API calls to assemble gadgets.Normalization removes comments and non-ASCII characters before dependency-based extraction.
  • Code gadgets and extraction: Code gadgets are assembled from semantically related statements connected by data or control dependencies and labeled vulnerable or non-vulnerable.The extraction process backtracks library/API calls through related variables and callers.
  • Data preparation: Duplicate gadgets are identified with SHA256 hashing; conflicting-label gadgets are removed, while same-label copies are deduplicated to reduce leakage and training problems.The cleaning step addresses both test-set leakage and negative effects from label conflicts.
  • Data preparation: Preprocessing removes comments, replaces user-defined names with symbolic tokens, and creates vulnerability-specific subsets for binary and multi-class classification.The subsets include vulnerability types with their non-vulnerable counterparts.
  • Data preparation: Datasets are partitioned into groups for separate and combined vulnerability experiments, including Buffer Error and Resource Management Error cases.The VulDeePecker example uses separate groups for each error type and a combined group.
  • Models and representations: The study compares BiLSTM, BiGRU, BERT-family, CodeBERT, GPT-2 variants, GPT-J, and Megatron models using model-specific tokenization and embedding methods.Transformer models are examined across different architectures and parameter scales, while recurrent baselines use Word2Vec.

3 EXPERIMENTS AND RESULTS

Experiments compare transformer-based models with recurrent baselines across binary and multi-class vulnerability detection, while examining data cleanliness, model size, and fine-tuning cost. Transformer models generally perform better, but the strongest model and trade-offs vary by task and dataset.

  • Experimental setup: Training used 10 epochs with learning_rate = 1.0e-05, weight_decay = 0.06, and warmup_steps = 500, while BiLSTM used 100 epochs.The experiments used batch size 16 and a linear learning-rate scheduler.
  • Data preparation: Clean data was required because redundant samples in the original VulDeePecker dataset created a high possibility of test-set leakage and apparently better performance.The study removed duplicate gadgets and conflicting labels before evaluation.
  • Binary classification: GPT-2 Large achieved a 95.51% F1-score for Buffer Error compared with 86.6% for the originally reported BiLSTM, while GPT-2 XL reduced FNR to 4.72% from 18%.For Resource Management Error, several transformer models improved all reported metrics over the original BiLSTM results.
  • Multi-class classification: In multi-class classification, GPT-2 XL had the best global-average Recall and F1-score, whereas GPT-J had the best precision; performance generally increased with model size despite fluctuations.The same increasing-with-size trend was observed in the binary-classification results, although GPT-J did not outperform smaller models there.
  • Fine-tuning cost: GPT-J required around 65 hours for 10-epoch multi-class fine-tuning on one RTX A6000, motivating model choices based on performance–time trade-offs.The paper recommends GPT-2 Large when time is unconstrained and CodeBERT when a trade-off is needed for Group 1 F1-score.
  • SeVC dataset: For the SeVC dataset, BERTBase reached an 88.34% global-average F1-score in multi-class classification, while multi-class performance remained lower because multiple vulnerabilities increase task complexity.In binary classification, BERTBase and GPT-2 Base generally improved over BiLSTM across the reported metrics.

4 PLATFORM ANALYSIS

The platform analysis compares parallelism frameworks, identifies engineering constraints in fine-tuning large models, and recommends DeepSpeed-integrated HuggingFace when models exceed one GPU.

  • 4 PLATFORM ANALYSIS: Large transformer models can exceed single-GPU memory, motivating data, pipeline, tensor, and other parallelism approaches.GPT-2 Large with 774M parameters could not run on a 16GB NVIDIA V100 even with batch size 1.
  • 4 PLATFORM ANALYSIS: Horovod supports data parallelism and model partitioning but not model or pipeline parallelism, so models must fit within one GPU without implementation changes.The authors therefore exclude Horovod as a development framework for their experiments.
  • 4 PLATFORM ANALYSIS: Megatron supports 3D parallelism, while DeepSpeed provides memory- and communication-efficient model training through techniques such as ZeRO.HuggingFace supports data parallelism and integrates DeepSpeed, whereas Megatron offers fewer pre-trained language models.
  • 4 PLATFORM ANALYSIS: HuggingFace pipeline parallelism caused GPU idling and long processing times when fine-tuning GPT-2 Extra Large with four GPUs.Limited access to computing resources delayed the experiments.
  • 4 PLATFORM ANALYSIS: The recommendations are to use data parallelism when a model fits one GPU and HuggingFace with DeepSpeed otherwise.DeepSpeed ZeRO Stage 2 enabled GPT-2 XL fine-tuning on a single 16GB GPU, although GPT-J fine-tuning remained impractical.
  • 4 PLATFORM ANALYSIS: 26.05% performance gain and 57.77% lower GPU RAM usage were achieved for single-GPU GPT-2 XL fine-tuning with DeepSpeed.The two-GPU comparison produced a 12.45% processing speed gain and 65.64% memory gain.

5 RELATED WORK

Related work spans transformer, dynamic-analysis, pattern-matching, and machine-learning approaches to vulnerability detection, while this paper focuses on standard code gadgets with transformer models.

  • 5 RELATED WORK: CodeBERT is trained on paired natural-language descriptions and programming-language code to support program-document generation and natural-language code search.Its architecture combines BERT with generators for code instructions and natural language descriptions.
  • 5 RELATED WORK: Prior vulnerability-detection research used BERTBase and BERT models with recurrent heads, reporting 93.49% highest detection accuracy on its dataset.The present work differs by considering a larger dataset and multiple model architectures.
  • 5 RELATED WORK: Existing vulnerability-detection techniques include dynamic analysis, pattern matching, and machine learning with CNN- and RNN-based models.Examples include fuzzing, taint analysis, code similarity, rule-based methods, and code-property graphs.
  • 5 RELATED WORK: Classical machine learning requires difficult and labor-intensive feature engineering, while much deep-learning research has focused on CNN and RNN models.These models require formatted data to capture vulnerability-related features.
  • 5 RELATED WORK: This work uses standard code-gadget extraction rather than improved representations, minimum intermediate representation learning, or graph neural networks.The authors state that improved extraction techniques could be transferred by updating the code gadget.

6 CONCLUSION

The paper presents a transformer-based vulnerability-detection framework, compares transformer and recurrent models across datasets and task types, and analyzes platforms for fine-tuning.

  • 6 CONCLUSION: Transformer models outperformed BiLSTM and BiGRU across reported metrics on the two-vulnerability dataset.GPT-2 Large achieved the best global-average binary F1-score, while GPT-2 XL achieved the best global-average multi-class F1-score.
  • 6 CONCLUSION: F1-score generally increased with model size on the two-vulnerability dataset.The conclusion reports separate performance patterns for a second dataset containing 126 vulnerability types across 341 CWE IDs.
  • 6 CONCLUSION: On the second dataset’s binary tasks, BERTBase and GPT-2 achieved better F1-scores than the compared recurrent models.The supplied conclusion passage ends before stating the complete qualification of this result.

A.1 VulDeePecker Data

The VulDeePecker dataset contains vulnerability examples collected from academic flaws and open-source projects for source-code vulnerability detection.

  • A.1 VulDeePecker Data: The VulDeePecker dataset contains two common vulnerability types collected from syntactic and academic security flaws and popular open-source projects.The cited passage names sources including the Linux kernel, Thunderbird, and Wireshark.

A.2 SeVC Data

The SeVC dataset contains diverse C/C++ vulnerability samples organized into four categories based on vulnerability cause. It supports analysis of vulnerabilities involving library/API calls, arrays, pointers, and arithmetic expressions.

  • SeVC covers 126 vulnerability types collected from open-source C/C++ programs.The sources include 1,591 NVD programs and 14,000 SARD programs.
  • The dataset contains 56,395 vulnerable samples and 364,232 clean samples.
  • Library/API Function Call vulnerabilities concern defects related to library or API function calls.
  • Array Usage vulnerabilities include address transfer as a function parameter and improper array-element access.
  • Pointer Usage vulnerabilities include improper pointer arithmetic and wrong referencing.
  • Arithmetic Expression vulnerabilities include defects such as integer overflow.

B.2 Bidirectional Gated Recurrent Unit

BiGRU is a bidirectional recurrent model built from gated recurrent units, using fewer gates and parameters than LSTM while retaining a similar architecture.

  • BiGRU uses forward and backward recurrent processing with GRU cells replacing the LSTM cells in the depicted architecture.
  • A GRU resembles an LSTM but has a forget gate, fewer parameters, and no output gate.
  • Each GRU cell receives the previous memory component C_t−1 and the word embedding x_t at time t.
  • The GRU computations use bias and weight matrices, a sigmoid function, a hyperbolic tangent function, and element-wise multiplication.

B.3 Transformers

Transformers use attention to weight token context and combine positional encoding, multi-head attention, and fully connected layers. The section introduces encoder-decoder processing and several transformer-based models used in the paper.

  • Transformer architectures use attention to differentially weight the importance and context of input tokens.
  • Positional encoding adds relative or exact token positions to embeddings represented in d_model dimensions.
  • Scaled dot-product attention combines query, key, and value matrices using softmax weighting, with d_k specifying key-vector dimension.
  • Multi-head attention projects queries, keys, and values into h representations, computes attention in parallel, then concatenates and linearly projects the outputs.
  • The decoder uses masked self-attention to ignore future inputs, then attends to encoder outputs before producing probability-space outputs.
  • The paper considers BERT, DistilBERT, RoBERTa, CodeBERT, GPT, and Megatron-LM as transformer-based models.

C PERFORMANCE METRICS

The paper evaluates vulnerability classifiers using confusion-matrix outcomes and metrics that characterize correctness, false alarms, missed vulnerabilities, precision, recall, and their harmonic mean.

  • Positive samples are vulnerable code gadgets, while negative samples are non-vulnerable code gadgets.
  • True positives and true negatives are correctly classified positive and negative samples, whereas false positives and false negatives are misclassified samples.
  • False Positive Rate measures the proportion of negative samples misclassified as positive, using FP divided by FP plus TN.
  • False Negative Rate measures the proportion of vulnerable samples misclassified as non-vulnerable, using FN divided by FN plus TP.
  • Precision measures resistance to labeling negative samples as positive, while Recall measures correct classification of positive samples.
  • F1-score is the harmonic mean of Precision and Recall and considers false positives and false negatives together.

D THREE FOLDS RESULTS

The paper reports three-fold evaluation results across binary and multi-class classification on the clean VulDeePecker dataset. Tables 14–17 organize these test-performance results by task.

  • Three-fold individual results are reported for the VulDeePecker dataset.
  • Tables 14 and 15 present test performance for binary classification on the clean VulDeePecker dataset.
  • Tables 16 and 17 present test performance for multi-class classification on the clean VulDeePecker dataset.
Loading 2204.03214v2…