PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatic feature learning for vulnerability prediction

Hoa Khanh Dam, Truyen Tran, Trang Pham, Shien Wee Ng, John Grundy, Aditya Ghose

arXiv:1708.02368v1cs.SE

TL;DR

Vulnerability prediction is challenging because vulnerabilities are rare and existing analysis and handcrafted representations have limitations. This paper uses LSTMs to learn semantic and syntactic features from code automatically. On 18 Android applications, it reports improvements for within-project and cross-project prediction over comparison approaches.

  • Problem

    Vulnerabilities are rare, while static analysis commonly produces many false positives and dynamic analysis requires representative test cases; prediction can help prioritize inspection.

    Source: §I · p. 1

  • Method

    The approach uses LSTMs to automatically learn semantic and syntactic code features, including method-level sequential features and context-based token features.

    Source: §I · p. 2, D. Generating syntactic features · p. 5, E. Generating semantic features · pp. 5–6

  • Results

    On 18 Android applications, within-project prediction improved 58% over software metrics and 3% over BoW and DBN on average, while cross-project applicability improved 85% over BoW or DBN models.

    Source: 4) Cross-project prediction (RQ3) · p. 9, §IX · p. 10

  • Takeaways & Limitations

    The evaluation supports using automatically learned features for vulnerability prediction within the studied Android applications, including cross-project prediction.

    Source: 4) Cross-project prediction (RQ3) · p. 9, §IX · p. 10

    The vulnerability labels cover Java source files, and the Android dataset may not represent all Android applications or other application types and languages.

    Source: §VII · p. 10

Abstract

from arXiv · show

Code flaws or vulnerabilities are prevalent in software systems and can potentially cause a variety of problems including deadlock, information loss, or system failure. A variety of approaches have been developed to try and detect the most likely locations of such code vulnerabilities in large code bases. Most of them rely on manually designing features (e.g. complexity metrics or frequencies of code tokens) that represent the characteristics of the code. However, all suffer from challenges in sufficiently capturing both semantic and syntactic representation of source code, an important capability for building accurate prediction models. In this paper, we describe a new approach, built upon the powerful deep learning Long Short Term Memory model, to automatically learn both semantic and syntactic features in code. Our evaluation on 18 Android applications demonstrates that the prediction power obtained from our learned features is equal or even superior to what is achieved by state of the art vulnerability prediction models: 3%--58% improvement for within-project prediction and 85% for cross-project prediction.

I. INTRODUCTION

Vulnerability prediction is difficult because vulnerabilities are rare, conventional tools and handcrafted features have limitations, and code with different behavior can share the same metrics or token frequencies. The paper proposes LSTM-based automatic feature learning, illustrated by Java locking code that exposes a weakness in those existing representations.

I. INTRODUCTION

  • Vulnerabilities are rare relative to other defects, while static analysis commonly produces many false positives and dynamic analysis requires representative test cases.The introduction says practice therefore still relies heavily on domain knowledge to identify code for intensive inspection.Source: §I · p. 1
  • Handcrafted metrics can miss semantic differences between code regions and may not generalize across projects; Bag-of-Words also misses token relations and code sequence structure.The paper notes that feature selection is manual and can reflect outdated experience or expert bias.Source: §I · pp. 1–2
  • LSTM-based automatic feature learning aims to capture both code-token semantics and sequential structure, avoiding manual feature engineering.The approach targets long-distance relationships such as Java try/catch and API lock()/unlock() pairs.Source: §I · p. 2

II. MOTIVATION

  • In the motivating example, an exception can bypass unlock(), leaving the lock unreleased and potentially causing deadlock and denial-of-service attacks.The corrected listing places unlock() in a finally block so release occurs whether or not an exception occurs.Source: §II · p. 2
  • The two Java listings have identical software metrics and Bag-of-Words token frequencies, although only the left-hand listing contains the vulnerability.The example illustrates why those representations cannot distinguish the vulnerable and corrected code.Source: §II · p. 2

III. BACKGROUND

Vulnerability prediction represents each source file with features and predicts whether it is vulnerable, focusing on files for prioritizing attention rather than pinpointing a line. RNNs process sequences, and LSTMs add a memory mechanism for learning long-term dependencies.

A. Vulnerability prediction

  • Prediction operates at the source-file level: a file is represented as a feature vector, and the model predicts whether it is vulnerable.The stated goal is to alert engineers to code deserving attention, not identify an exact vulnerable line.Source: A. Vulnerability prediction · pp. 2–3

B. Recurrent Neural Network and Long Short Term Memory

  • An RNN maps an input sequence to an output sequence, computing each step's output state from the current input and previous state.It shares parameters across steps, reducing the total number of parameters to learn.Source: B. Recurrent Neural Network and Long… · p. 3
  • An LSTM adds a memory cell controlled by input, forget, and output gates, allowing its output state to depend on current memory and the previous memory state.The paper identifies this memory path as the key feature that allows learning long-term dependencies.Source: B. Recurrent Neural Network and Long… · p. 3

IV. APPROACH

The approach feeds method token sequences into an LSTM to produce method vectors and context-sensitive token states. It pools method vectors into file-level syntactic features and uses project-wide token-state clusters to address cross-project generalization.

A. Overview

  • Each Java file is parsed into method token sequences that an LSTM transforms into fixed-size method vectors and context-sensitive token states.The header is treated as a special method.Source: A. Overview · p. 3
  • Pooling aggregates method feature vectors into a single file representation, producing syntactic features.The overview gives mean-pooling as a simple example.Source: A. Overview · pp. 3–4
  • Because syntactic features can be project-specific, the approach clusters token states from all studied projects into a codebook for semantic feature generation.The paper presents this second feature set as addressing generalization across projects.Source: A. Overview · p. 4

B. Parsing source code

Source files are parsed into token sequences, with comments and blank lines ignored and uncommon or unseen tokens normalized. An LSTM learns token representations by predicting the next code token from preceding context.

B. Parsing source code

  • Comments and blank lines are ignored; numeric and string constants are normalized, and rare or unseen tokens map to <unk>.The vocabulary is fixed to the most popular tokens.Source: B. Parsing source code · p. 4

C. Learning code token semantics

  • Each code token is mapped to a fixed-length continuous vector through a vocabulary-indexed embedding matrix.The embedding matrix acts as a lookup table, with one column per vocabulary token.Source: C. Learning code token semantics · p. 4
  • The LSTM uses a token’s output state to predict the next token from the preceding code context, learning token embeddings and prediction weights.This training objective estimates a code language model and is described as automatically learning a code grammar.Source: C. Learning code token semantics · p. 4

1) Model training:

The LSTM is trained on code sequences to predict each true next token, minimizing sequence log-loss. Its learned states retain information from preceding tokens and are used to generate file features.

1) Model training:

  • Training minimizes the sum of log-losses for the true next token at each position in a code sequence.The sequence loss is essentially the negative log probability of the full token sequence.Source: 1) Model training · p. 4
  • The model parameters, including M and U, are updated by stochastic gradient descent using the loss gradient from backpropagation.Source: 1) Model training · p. 5

2) Generating output token states:

  • After training, the LSTM computes a state vector for every code token, with each state containing information from preceding tokens.The states are used to generate two distinct sets of file features.Source: 2) Generating output token states · p. 5

D. Generating syntactic features

Syntactic features aggregate LSTM token states first within methods and then across a file. Semantic features instead represent files by counts of token states assigned to regions of a shared semantic space.

D. Generating syntactic features

  • Method-level syntactic features aggregate token states across a method, using pooling that is invariant to variable method length.The resulting method vectors are pooled again across the file’s methods to form file-level syntactic features.Source: D. Generating syntactic features · p. 5
  • Mean pooling, variance pooling, or their concatenation can summarize the token states within a method.Source: D. Generating syntactic features · p. 5

E. Generating semantic features

  • A codebook built with k-means on training-set token states defines semantic-space regions; each file is represented by counts of its token states assigned to each centroid.With three centroids, the example file’s feature vector is [3, 2, 5].Source: E. Generating semantic features · p. 5
  • The semantic representation characterizes a file by the types of token usage contexts it contains, abstracting over its code tokens.The paper motivates this abstraction by noting that usage-context types may be fewer than the tokens in the dataset.Source: E. Generating semantic features · pp. 5–6

F. Vulnerability prediction

The approach generates syntactic and semantic features for training and test files, then uses the features to train vulnerability prediction models. The implementation uses an LSTM language model and a Random Forest classifier, with Noise Contrastive Estimation to reduce vocabulary costs.

F. Vulnerability prediction

  • Both syntactic and semantic features are generated for training and test files before training vulnerability prediction models and evaluating them on test files.Source: F. Vulnerability prediction · p. 6

V. IMPLEMENTATION

  • The approach is implemented in Python using Theano and Keras; Theano supports automatic differentiation of the loss and adaptive gradient-descent methods.Source: §V · p. 6

A. Training details

  • The LSTM is selected by validation perplexity from a dataset of more than 240K sequences, and a Random Forest classifier is used for prediction.200K sequences are used for training and the remainder for validation.Source: A. Training details · p. 6

B. Handling large vocabulary

  • Noise Contrastive Estimation approximates vocabulary-wide probability evaluation with a randomly sampled subset of 100 words.The approximation addresses the high cost of iterating through the large vocabulary.Source: B. Handling large vocabulary · p. 6

C. Handling long methods

The study handles variable-length methods by splitting long methods into fixed-length sequences, and evaluates the approach on retrieved Android source files across within-project, cross-version, and cross-project settings.

C. Handling long methods

  • Long methods are split into non-overlapping sequences of length 100, and their method features are averaged to avoid padding waste while enabling fixed-size mini-batches.The sequence length was chosen for faster learning.Source: C. Handling long methods · p. 6

A. Datasets

  • Two unavailable repositories and unmaintained older releases left 18 applications with incomplete version histories in the retrieved dataset.The original dataset contained 20 applications; the source files were retrieved using its file names and version numbers.Source: A. Datasets · p. 6

B. Research questions

  • Within-project evaluation used 10-fold cross-validation on each app’s first version; cross-version and cross-project tests trained on first versions and tested on later versions or other apps.For cross-version tests, subsequent versions of the same application were tested; for cross-project tests, first versions of other applications were tested.Source: B. Research questions · p. 7

C. Benchmarks

The evaluation compares the approach with software metrics, Bag of Words, and Deep Belief Networks using a shared classifier and standard metrics. It also illustrates how LSTM-learned token states group semantically related code tokens.

C. Benchmarks

  • Three benchmark families are used: 60 software metrics, token-frequency Bag of Words, and a two-hidden-layer Deep Belief Network built from word counts.The DBN hidden layers have sizes 500 and 128.Source: C. Benchmarks · p. 7
  • All prediction models use Random Forests and are evaluated with precision, recall, and F-measure.The shared classifier supports comparison across the approach and benchmarks.Source: C. Benchmarks · p. 7

1) Learned code token semantics:

C. Benchmarks

  • The within-project results table compares the three benchmarks with syntactic-only, semantic-only, and joint-feature variants.Joint features combine syntactic and semantic features.Source: Table II · p. 8

2) Within-project prediction (RQ1):

Across the evaluated settings, the approach outperforms its benchmarks, with the strongest reported relative improvement in cross-project applicability. Semantic features are more useful for cross-version and cross-project generalization, while syntactic features lead within projects.

2) Within-project prediction (RQ1):

  • The within-project comparison includes syntactic-only, semantic-only, and joint-feature variants alongside the three benchmarks.The reported variants use specified pooling choices, while additional pooling variations are not included here.Source: 2) Within-project prediction (RQ1) · p. 8
  • Within-project, syntactic-only features are best on average across 18 applications, with 58% F-measure improvement over software metrics and approximately 3% over BoW and DBN.All three approach variants exceeded 80% in precision, recall, and F-measure in every application.Source: 2) Within-project prediction (RQ1) · p. 8

3) Cross-version prediction (RQ2):

  • In cross-version prediction, semantic-only or joint features are the best variants, slightly outperforming BoW and DBN as models predict later versions of the same apps.The authors attribute the increased usefulness of semantic measures in this setting to improved generalization.Source: 3) Cross-version prediction (RQ2) · p. 8, Table III · p. 9

4) Cross-project prediction (RQ3):

  • 85% improvement over BoW or DBN in cross-project applicability: joint-feature models applied to around 4 other applications on average.All joint-feature models applied to at least one other application; some applied to eight.Source: Table IV · p. 9, 4) Cross-project prediction (RQ3) · p. 9

E. Remarks and implications

Within-project prediction benefits from BoW's simplicity and repeated information across project versions, while cross-project prediction favors learned representations that can generalize across domains. The study also notes limits involving label coverage, possible version overlap, statistical testing, and representativeness.

E. Remarks and implications

  • Within-project BoW performs strongly in part because versions of the same project may repeat code, token statistics, conventions, and style.BoW is also a strong text-classification representation, and flexible classifiers such as Random Forests can work well with expressive representations.Source: E. Remarks and implications · p. 9
  • Cross-project BoW statistics may differ between projects, whereas the LSTM language model and codebook learn a common representation across domains without vulnerability-label supervision.The authors suggest that external unlabeled data could therefore be used.Source: E. Remarks and implications · p. 9
  • The authors recommend BoW for within-project prediction and LSTM language modeling with codebook-based semantic structure discovery for cross-project generalization.The recommendation distinguishes the simpler within-project setting from transfer between projects.Source: E. Remarks and implications · p. 9

VII. THREATS TO VALIDITY

  • The authors acknowledge that they did not apply statistical tests to verify the significance of their conclusions and plan to investigate this in future work.They used standard vulnerability-prediction performance measures.Source: §VII · p. 10
  • The dataset labels only Java source files, and allowing identical files across versions may have inflated cross-version results, although every compared model benefits from that overlap.Other files, including XML manifests containing access-right information, may also contain security information.Source: §VII · p. 10
  • Although the study covers diverse Android applications, its dataset may not represent all Android applications, and findings need further examination on other applications and platforms.The authors specifically identify web applications and software written in PHP or C++ as future validation settings.Source: §VII · p. 10

VIII. RELATED WORK

Prior vulnerability prediction work has used manually selected metrics, text-based BoW features, and deep learning approaches for software tasks. The paper reports improvements for its LSTM-based vulnerability features and proposes broader application and end-to-end extensions as future work.

VIII. RELATED WORK

  • Earlier vulnerability models used complexity, coupling, cohesion, churn, developer activity, dependency, and organizational metrics, which require domain experts to select predictors.Examples include McCabe’s cyclomatic complexity, coupling and cohesion metrics, and code churn and developer activity metrics.Source: §VIII · p. 10
  • BoW vulnerability models use source-code term frequencies, but disregard syntactic information such as code order; BoW has also produced higher recall than software metrics for PHP applications.Source: §VIII · p. 10
  • Related software-defect work has adapted defect models for vulnerability prediction and used diverse metrics and deep belief networks to learn defect features.Source: §VIII · p. 10
  • Software-engineering research has applied LSTMs and other recurrent networks to code modeling, clone detection, API-sequence generation, and C-program repair.CNNs have also been used for bug localization.Source: §VIII · p. 10

IX. CONCLUSIONS AND FUTURE WORK

  • 58% average improvement over software metrics and 3% over BoW and Deep Belief Network for within-project prediction; cross-project prediction improved 85% on average over state-of-the-art techniques.The evaluation used 18 Android applications from a public dataset.Source: §IX · p. 10
  • Future work includes testing other application types and languages, predicting vulnerabilities at method and code-change levels, and extending the approach to defects and safety-critical hazards.The authors also plan a fully end-to-end system from raw code tokens to vulnerability outcomes.Source: §IX · pp. 10–11
Loading 1708.02368v1…