PaperlayerView exact revision on arXiv

Source-linked AI summary

D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis

Yunhui Zheng, Saurabh Pujar, Burn Lewis, Luca Buratti, Edward Epstein, Bo Yang, Jim Laredo, Alessandro Morari, Zhong Su

arXiv:2102.07995v1cs.SEcs.AIcs.LG

TL;DR

Static analysis provides scalable vulnerability findings but generates many false positives, and existing training datasets often lack realistic context, scale, or coverage. D2A labels issues by comparing static-analysis results before and after presumed bug-fixing commits, producing a large dataset from real-world projects. Manual review improved label accuracy from 7.8% to 53%, and a demonstrated model reduced false alarms to help prioritize likely real bugs.

  • Problem

    Existing vulnerability datasets often lack context, scale, or realistic source-code diversity, while static analysis produces many false positives that complicate issue review.

  • Method

    D2A compares static-analysis issues in versions before and after presumed bug-fixing commits, labeling disappearing issues as likely fixed bugs and generating a dataset from real-world projects.

  • Results

    7.8% to 53% label accuracy was observed in manual review, while D2A produced more than 1.3M examples and supported a false-positive reduction model.

  • Takeaways & Limitations

    D2A can help developers prioritize static-analysis issues more likely to be true positives and provides detailed data for AI-based vulnerability detection research.

Abstract

from arXiv · show

Static analysis tools are widely used for vulnerability detection as they understand programs with complex behavior and millions of lines of code. Despite their popularity, static analysis tools are known to generate an excess of false positives. The recent ability of Machine Learning models to understand programming languages opens new possibilities when applied to static analysis. However, existing datasets to train models for vulnerability identification suffer from multiple limitations such as limited bug context, limited size, and synthetic and unrealistic source code. We propose D2A, a differential analysis based approach to label issues reported by static analysis tools. The D2A dataset is built by analyzing version pairs from multiple open source projects. From each project, we select bug fixing commits and we run static analysis on the versions before and after such commits. If some issues detected in a before-commit version disappear in the corresponding after-commit version, they are very likely to be real bugs that got fixed by the commit. We use D2A to generate a large labeled dataset to train models for vulnerability identification. We show that the dataset can be used to build a classifier to identify possible false alarms among the issues reported by static analysis, hence helping developers prioritize and investigate potential true positives first.

I. INTRODUCTION

Static analysis scales to complex software but produces many false positives, while existing vulnerability datasets often lack realistic scale, context, and coverage. D2A addresses these gaps by labeling analyzer issues through differential analysis of bug-fixing version pairs and demonstrates utility for false-positive reduction.

  • Motivation: Existing vulnerability datasets often lack bug context, have limited size, or rely on synthetic patterns that do not represent diverse real-world behavior.Function-level datasets may omit traces, bug types, locations, and root causes; confirmed-bug datasets may also be too small for training.
  • Motivation: Static analysis scales to millions of lines and can detect subtle bugs, but approximation for scalability creates excess false alarms.Path-insensitive analysis commonly assumes all paths are feasible, introducing false positives.
  • D2A Approach: D2A labels static-analysis issues as likely true or false positives by comparing analyzer results before and after presumed bug-fixing commits.Issues that disappear after a commit are treated as very likely real bugs fixed by that commit.
  • Dataset and Evaluation: 349,373,753 reported issues yielded 18,653 positive and 1,276,970 negative unique examples across thousands of version pairs from six open-source projects.The analyzed projects were OpenSSL, FFmpeg, libav, httpd, NGINX, and libtiff.
  • Dataset and Evaluation: D2A improved manually reviewed label accuracy from 7.8% to 53% and supported a model that reduced false alarms and helped prioritize likely real bugs.The dataset preserves inter-procedural bug types, locations, traces, and analyzer outputs, and was made publicly available with its generation pipeline.

II. MOTIVATION

The paper motivates building high-quality labeled datasets because AI-based vulnerability detection requires many examples, while static-analysis-derived labels can support this need. It presents D2A as a basis for demonstrating why such datasets are useful for AI-based vulnerability detection.

  • AI-based vulnerability detection requires large labeled datasets of buggy and non-buggy code examples, especially for advanced neural models.

A. Existing Datasets for AI on Vulnerability Detection Task

Existing vulnerability-detection datasets trade off label quality, scale, realism, and contextual information. D2A is positioned to complement them by providing large, real-world examples with inter-procedural context and analysis metadata.

  • Dataset landscape: Public vulnerability datasets include synthetic, manually labeled, commit-derived, and confirmed-vulnerability examples with different strengths and limitations.The paper compares these datasets to highlight D2A’s contribution.
  • Dataset limitations: Synthetic datasets can be sizable but rely on predefined patterns and therefore lack the behavioral diversity of real-world programs.Juliet, Choi et al., and S-babi are cited as examples.
  • Dataset limitations: Manually reviewed or confirmed-bug datasets generally offer better labels, but their sample counts may be insufficient for model training.The paper identifies this limitation for Draper and CDG.
  • Dataset limitations: Several datasets provide limited context because examples are function-level, omit original-code links or traces, or assume all functions in a fixing commit are buggy.These constraints hinder additional compilation, analysis, or precise bug characterization.
  • D2A contribution: D2A examples are generated from inter-procedural analysis and include bug types, locations, traces, repository and commit links, analyzer outputs, and compiler arguments.The dataset is intended to address the need for real-world examples with context beyond function boundaries.

1) Manual Case Study:

The manual case study evaluates Infer on large C/C++ programs and reviews security-relevant reports to measure true and false positives. It finds that most reviewed reports are false positives, while bug-report features separate the two classes.

  • Manual Case Study: Infer was selected because it supports industrial-scale C/C++ programs and inter-procedural analysis, whereas Clang Static Analyzer lacks cross-translation-unit support.The experiments use Infer with its default setting.
  • Manual Case Study: Infer reported 492 issues across four bug types, and 166 security-related issues remained after excluding DEAD_STORE reports.The excluded DEAD_STORE category was considered not directly related to security vulnerabilities.
  • Manual Case Study: Eight C/C++ developers reviewed each issue through multiple rounds, including test-case construction when both initial reviewers agreed the bug could occur.The review process was time-consuming, especially for complex cryptographic programs.
  • Manual Case Study: 13 of 166 security-related issues were confirmed true positives, leaving 92.2% as false positives.This manual review quantifies the false-positive problem for the studied Infer reports.

2) Feature Exploration for False Positive Reduction:

The feature exploration tests report-derived indicators of issue complexity and finds that normalized true-positive and false-positive averages are separable across all eight features. The section then introduces D2A’s differential-analysis workflow for generating labeled examples.

  • Feature Exploration: The explored features describe error location and trace complexity through line numbers, columns, unique lines, files, packages, branches, and functions.They comprise eight features grouped into three categories.
  • Feature Exploration: After normalization, average values for 13 true positives and 153 false positives differed significantly and were easily separable across all eight features.The result suggests that a false-positive reduction classifier may perform well.
  • D2A Workflow: D2A takes a Git repository URL as input and produces examples using static differential analysis.The pipeline is designed to generate examples without relying on manually written labels.
  • D2A Workflow: A commit-message analyzer filters commits likely to fix bugs before expensive analysis of consecutive versions.This preprocessing step removes commits such as documentation improvements.
  • D2A Workflow: For each selected commit, the analyzer compares issue sets from before-commit and after-commit versions to identify issues fixed by the commit.The auto-labeler aggregates these findings across multiple consecutive version pairs and filters noise using commit history.
  • D2A Workflow: The resulting positives are issues likely to be real bugs, while issues never fixed by a commit are labeled negatives likely to be false positives.Function bodies are extracted according to the corresponding bug traces to create dataset examples.

B. Commit Message Analysis

D2A uses the Commit Message Analyzer to identify likely vulnerability-fixing commits, then compares static-analysis results across commit versions to label and consolidate issues.

  • CMA combines semantic similarity and snippet-sample methods learned from NVD to rank commits likely associated with vulnerability fixes.
  • CMA categorizes vulnerabilities from semantic distributions and ranks associated commits by confidence scores.
  • Infer analyzes both versions surrounding each selected bug-fixing commit, with the pipeline choosing it for security coverage, cross-translation-unit analysis, and scalability.
  • Fixed issues are those detected before but absent after a commit; pre-existing and introduced issues are tracked separately.
  • D2A merges and deduplicates issues across version pairs, then labels filtered fixed issues positive and other issues negative.
  • A fixed issue reappearing later is relabeled negative, while an issue whose trace does not overlap the commit diff is also treated as negative.

D. After-fix Examples

D2A adds after-fix examples by pairing positive fixed-issue examples with corresponding post-fix code, creating balanced negatives that remain closely related to the bugs.

  • After-fix examples are extracted from versions in which positive issues no longer appear, providing negative examples paired with positive examples.
  • This pairing balances the positive and negative classes and helps models focus on code changes associated with bug fixes.
  • Each dataset example records a label and label source, while static analysis supplies bug type, location, trace, and raw report information.
  • Trace steps can contribute the involved function bodies, and commit-diff overlap marks whether a function was touched by the fix.
  • Compiler arguments are retained to support additional compilation-dependent analyses.
  • D2A demonstrates use as a static-analysis false-positive reduction model that helps developers prioritize potential true positives.

A. Problem Statement

The problem is to prioritize likely true positives because static analyzers produce many false alarms, while features from their reports may capture issue complexity for classification.

  • Excessive false positives reduce developers’ trust in static analyzers and motivate prioritizing issues more likely to be true positives.
  • The model treats the static analyzer as a black box and uses only bug reports as input.
  • The intended balance is many predicted positives together with a high false-positive reduction rate.
  • Bug traces describe paths from the suspected error origin through the steps leading to the reported error, often across functions and files.
  • The experiments use positive samples and before-fix negatives because after-fix negatives lack bug traces, and negative labels substantially outnumber positives.
  • Features are motivated by the assumption that complex code may be more likely to contain bugs or receive bug classifications from static analysis.
  • Trace size, file and line locations, and code nesting provide indicators of issue complexity.

D. Model Selection

D2A compares machine-learning models for false-positive reduction using metrics suited to imbalanced data and combines classifier scores to address variation across datasets.

  • The study evaluates 13 machine-learning models and selects four top models based on AUC and F1 scores.
  • Random Forest trains independent decision trees on random data and feature samples, while boosting sequentially improves predictions from earlier trees.
  • AUC evaluates threshold-independent trade-offs between false-positive and true-positive rates in the imbalanced dataset.
  • The ROC operating point is selected by minimizing distance to the top-left corner, then F1-score summarizes average class performance.
  • Soft voting combines classifier scores because performance can vary across projects with different sizes and class imbalance.

A. Dataset Generation Results

D2A’s dataset-generation pipeline analyzes six open-source programs through bug-fixing version pairs and validates auto-labeler quality with manual review. The experiment also defines project-level splits and restricts analysis to production-ready security-related error types.

  • Dataset construction: The pipeline analyzes six open-source programs and runs Infer on before-commit and after-commit versions for selected bug-fixing commits.Commits are dropped when Infer cannot analyze either version.
  • Label validation: Manual review of 57 examples found that auto-labeler accuracy increased from 35% to 53%, compared with 7.8% without auto-labeling in an OpenSSL security-example study.The reviewed sample contained 41 positives and 16 negatives and intentionally focused on positives.
  • Dataset organization: The released data uses project-specific train, development, and test splits while preserving bug-type distributions.The split ratio is 80:10:10.
  • Dataset limitations: FFmpeg examples were removed from the combined experiment because some examples resemble libav examples, and their greater class imbalance was left for future work.The similarity reflects libav’s origin as a fork of FFmpeg.
  • Dataset scope: The experiments consider 18 security-related bug types enabled by default in Infer, although the pipeline collected examples for additional non-production-ready types.This filtering determines the error types used in the reported experiment.

2) Feature Importance:

Feature ranking identifies report location and size as important signals, while several model families reduce static-analysis false positives across projects. No single model dominates consistently, but voting performs well in the combined experiment.

  • Feature importance: The line number of the reported error and the number of lines in its bug report are important features across many models.The final feature set contains 25 features selected using feature-importance rankings.
  • Model setup: The evaluation uses Random Forest, Extra Trees, LightGBM, CatBoost, and related boosting configurations to compare false-positive reduction models.Random Forest uses 1000 estimators, while Extra Trees uses 500.
  • False-positive reduction: Most models achieve a False Positive Reduction Rate above 70% for each project without substantially reducing true bugs.The paper defines FPRR from the reduction in false positives relative to Infer’s negative examples.
  • Model comparison: No single model performs best across all projects, whereas voting can outperform individual models in the combined experiment.The authors associate this result with the potential benefit of using more data in the voting system.

VII. CONCLUSION

The paper presents D2A as an automated dataset-building approach for real-world vulnerability-detection research, based on differential analysis of static-analysis results across version pairs. Its dataset supports false-positive reduction models that can help prioritize potential true positives.

  • Conclusion: D2A labels static-analysis issues through differential analysis and builds a labeled dataset from real-world programs for AI-based vulnerability detection.The approach was run on six large programs and incorporates bug-related information from static analysis, code, and commit history.
  • Conclusion: The released project-level dataset splits define train, development, and test sets, with combined splits formed from corresponding project sets.The models are trained on train plus development data and tested on the test set.
  • Conclusion: The dataset contains more than 1.3M examples generated from six large programs.The examples include detailed information obtained from inter-procedural static analysis, the code base, and commit history.
  • Conclusion: The false-positive reduction model effectively suppresses false positives and helps developers prioritize and investigate potential true positives.The reported false-positive reduction results are summarized in Table VII.
Loading 2102.07995v1…