PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepBugs: A Learning Approach to Name-based Bug Detection

Michael Pradel, Koushik Sen

arXiv:1805.11683v1cs.SEcs.PL

TL;DR

Existing bug detectors often ignore identifier names, missing bugs whose intended semantics is visible in names. DeepBugs learns semantic identifier representations and classifiers from transformed training examples, achieving 89%–95% accuracy and finding 102 real-world problems with 68% precision.

  • Problem

    Most bug detectors ignore identifier names, while existing name-based approaches use lexical reasoning and manually designed algorithms despite names conveying useful semantic information.

  • Method

    DeepBugs learns identifier embeddings and binary classifiers from likely-correct code plus likely-incorrect examples generated by simple transformations.

  • Results

    89%–95% accuracy and 68% precision were reported across 150,000 JavaScript files, with 102 real-world bugs and code-quality problems found.

  • Takeaways & Limitations

    Artificially seeded bugs can train detectors that identify real-world name-related bugs in dynamically typed JavaScript code.

  • Takeaways & Limitations

    Name-based detection can produce false positives when developers choose variable names that diverge from common practice.

Abstract

from arXiv · show

Natural language elements in source code, e.g., the names of variables and functions, convey useful information. However, most existing bug detection tools ignore this information and therefore miss some classes of bugs. The few existing name-based bug detection approaches reason about names on a syntactic level and rely on manually designed and tuned algorithms to detect bugs. This paper presents DeepBugs, a learning approach to name-based bug detection, which reasons about names based on a semantic representation and which automatically learns bug detectors instead of manually writing them. We formulate bug detection as a binary classification problem and train a classifier that distinguishes correct from incorrect code. To address the challenge that effectively learning a bug detector requires examples of both correct and incorrect code, we create likely incorrect code examples from an existing corpus of code through simple code transformations. A novel insight learned from our work is that learning from artificially seeded bugs yields bug detectors that are effective at finding bugs in real-world code. We implement our idea into a framework for learning-based and name-based bug detection. Three bug detectors built on top of the framework detect accidentally swapped function arguments, incorrect binary operators, and incorrect operands in binary operations. Applying the approach to a corpus of 150,000 JavaScript files yields bug detectors that have a high accuracy (between 89% and 95%), are very efficient (less than 20 milliseconds per analyzed file), and reveal 102 programming mistakes (with 68% true positive rate) in real-world code.

1 INTRODUCTION

DeepBugs addresses missed name-related bugs by learning semantic representations of identifiers and classifying correct versus incorrect code. It generates likely incorrect examples through code transformations and detects three bug patterns with strong reported accuracy and real-world precision.

  • Motivation: Identifier names convey intended semantics, but conventional bug detectors largely ignore them, causing them to miss mistakes apparent to humans.This issue is especially relevant in JavaScript, where static types are absent and names help reveal incompatible or misplaced values.
  • Challenges: Name-based detection must infer fuzzy identifier meaning and decide whether code is correct or incorrect.Existing approaches address these challenges with lexical similarity and manually designed algorithms, which require substantial tuning effort.
  • Approach: DeepBugs uses learned identifier embeddings and binary classification to distinguish correct from incorrect code without manually designed heuristics.The semantic representation captures relationships such as similarity between length and count that lexical matching may miss.
  • Training Data: DeepBugs creates likely incorrect training examples by applying simple transformations to existing likely-correct code.This supplies both classes of examples without manually labeling thousands of bugs.
  • Framework: The extensible framework supports detectors for swapped function arguments, incorrect binary operators, and incorrect binary operands.New detectors reuse the same identifier embedding and require a training-data generator plus a vector mapping for code examples.
  • Evaluation: 89%–95% accuracy and 68% precision were reported across evaluations of 150,000 JavaScript files, with 102 real-world bugs and code-quality problems found.The corpus contained 100,000 training files and 50,000 search files; manual inspection covered 150 warnings.

2 A FRAMEWORK FOR LEARNING TO FIND NAME-RELATED BUGS

DeepBugs learns name-based bug detectors from positive code examples and artificially transformed negative examples. It uses semantic identifier embeddings and a classifier to detect likely bugs without manually designed heuristics.

  • Framework: DeepBugs frames name-related bug detection as classification between likely correct and likely incorrect code.The framework trains a classifier for a particular bug pattern and applies it to previously unseen code.
  • Training data: The framework extracts likely correct examples from a corpus and creates likely incorrect examples through simple AST-based transformations.This avoids relying on manually labeled bug datasets, which are difficult to scale.
  • Model: The classifier is a feedforward neural network trained on vector representations of positive and negative examples.Identifier names and selected literals are mapped into vectors before classification.
  • Semantic representations: DeepBugs represents identifiers with learned embeddings so semantically similar names can receive similar vectors.This supports generalization beyond identical names, unlike one-hot representations.
  • Framework: Unlike prior name-based detectors, DeepBugs learns detectors and warning exceptions rather than relying on manually designed lexical heuristics or filters.The approach infers exceptions from training data instead of hard-coding lists of function names.

3 NAME-BASED BUG DETECTORS

The framework instantiates three name-based detectors for swapped arguments, wrong binary operators, and wrong binary operands. Each uses AST extraction, transformed negative examples, and vectorized code representations.

  • Detector coverage: DeepBugs provides detectors for accidentally swapped function arguments, incorrect binary operators, and incorrect binary operands.The latter two patterns are presented as new targets for name-based detection in this work.
  • Common design: Each detector combines an AST-based training-data generator with a vector representation that a machine-learning model classifies as benign or buggy.The same framework structure supports all studied bug patterns.
  • Swapped function arguments: The swapped-argument detector extracts callee, argument, base-object, literal-type, and formal-parameter names from calls with at least two arguments.Calls with unavailable required names are ignored, and negative examples swap the original argument order.
  • Incorrect binary operators: The incorrect-operator detector extracts operand names, the operator, literal types, and AST context, then replaces the operator with a different randomly selected binary operator.For example, i <= length may become i < length or i % length.
  • Incorrect binary operands: The incorrect-operand detector replaces one binary-operation operand with an alternative occurring in the same file.This creates likely incorrect examples while preserving the surrounding operation and context.

4 IMPLEMENTATION

DeepBugs uses simple AST traversals with the Acorn JavaScript parser to generate training data, then uses TensorFlow and Keras to build the detectors.

  • Implementation: The implementation extracts code through AST traversals based on the Acorn JavaScript parser.The training-data generator writes extracted information to text files for the detector implementation.
  • Implementation: The detector implementation builds on TensorFlow and Keras, while each individual detector requires about 100 lines of code.Most implementation resides in the generic framework rather than in individual detectors.

5 EVALUATION

The evaluation trains and validates DeepBugs on a large JavaScript corpus and examines accuracy, real-world warnings, efficiency, and embedding usefulness. The generated datasets contain millions of balanced examples for each detector.

  • Research questions: The evaluation asks whether DeepBugs distinguishes correct from incorrect code, finds production bugs, operates efficiently, and benefits from learned embeddings.These questions cover effectiveness, real-world utility, runtime, and representation quality.
  • Experimental setup: 150,000 JavaScript files containing 68.6 million lines are divided into 100,000 training files and 50,000 validation files.The corpus comes from cleaned open-source projects with duplicate files removed.
  • Training data: Each detector learns from several million examples, with half positive and half negative examples.The authors identify automated generation as important because manually creating this quantity of negative data would be impractical.

5.3 Warnings in Real-World Code

DeepBugs applies learned name-based detectors to real-world JavaScript warnings, finding bugs and code-quality problems that depend on identifier semantics. Manual inspection shows useful precision, alongside false positives from misleading or unconventional names.

  • Two evaluation sets assess DeepBugs: real-world warnings are manually inspected, while artificially created bugs support accuracy, recall, and precision analysis.The real-world evaluation applies detectors to unmodified code; the large-scale evaluation uses hundreds of thousands of artificially created bugs.
  • 68% of 150 inspected warnings were actual problems, comprising 95 bugs and 7 code-quality problems.The inspection distinguishes runtime bugs, code-quality problems, and false positives.
  • Examples of Bugs: DeepBugs finds mistakes that traditional name-unaware analyses may miss because their intended semantics become apparent only through identifiers and literals.Examples include swapped Promise.done arguments, a meaningless binary operation, and a reversed assertEquals argument order.
  • Examples of Code Quality Problems: Using !== instead of < in a loop termination condition is flagged as error-prone because increments or assignments can make the loop run out of bounds.The detector identifies the conventional use of < in termination conditions.
  • Examples of Code Quality Problems: Using bitwise | to compare boolean-valued expressions may be inefficient or produce unexpected behavior for non-boolean operands.DeepBugs flags the expression because the is identifiers suggest boolean return values normally compared with a logical operator.
  • Examples of False Positives: False positives arise when generic wrappers or poorly named variables make correct code appear inconsistent with common naming conventions.The approach reports unusual operands such as each added to a width and is compared with length, although close inspection shows the code is correct.

5.4 Accuracy and Recall of Bug Detectors

DeepBugs is evaluated with automatically seeded bugs using accuracy and threshold-dependent recall, while manual inspection complements these estimates with real-world findings. Across detectors, accuracy is high, and higher warning thresholds trade recall for fewer warnings.

  • Accuracy: 89.06%–94.70% accuracy shows that all three bug detectors effectively distinguish correct from artificially incorrect code examples.The evaluation uses automatically seeded bugs, with positive and negative examples classified by the detector.
  • Evaluation assumptions: Recall and false-positive counts are estimates because they assume seeded transformations create actual bugs and original code is correct.These assumptions enable evaluation with hundreds of thousands of artificial bugs and complement manual inspection of real-world warnings.
  • Warning thresholds: 116,941 warnings at t = 0.5 correspond to roughly one warning per 196 lines of code.Lower thresholds report more warnings and therefore tend to reveal more bugs while also increasing false positives.
  • Warning thresholds: 11,292 warnings at t = 0.9 correspond to roughly one warning per 2,025 lines of code.In practice, developers are expected to inspect only the highest-ranked warnings.

5.5 Efficiency

DeepBugs requires substantial corpus-wide training and prediction time, but its per-file prediction cost is low.

  • Prediction: Below 20 milliseconds per JavaScript file is the average prediction time for DeepBugs.Prediction time includes extracting code examples and querying the classifier for each example.
  • Corpus-wide processing: 36–73 minutes per bug detector is required to run training and prediction across all 150,000 files.Training includes gathering examples and classifier training; the authors characterize the total training time as reasonable.

5.6 Usefulness of Embeddings

Learned embeddings improve DeepBugs by representing semantic similarities between identifiers and code examples, while the broader approach remains effective with random embeddings.

  • Quantitative evaluation: Learned embeddings yield a more accurate classifier than random embeddings.Table 5 compares the two representations using detector accuracy and recall.
  • Quantitative evaluation: Learned embeddings increase recall for all three bug detectors compared with randomly assigned identifier vectors.They allow detectors to reason about semantic similarities between syntactically different examples and predict bugs across similar examples.
  • Quantitative evaluation: The overall approach retains relatively high accuracy and recall even with randomly created embeddings.This indicates value from the approach beyond the learned representation itself.
  • Qualitative evaluation: The embeddings capture abbreviation, lexical, and semantic similarities, including msg/message and wrapper/container.They also capture similarities between lexically dissimilar identifiers such as name/Identifier.

5.7 Vocabulary Size

Vocabulary size controls the coverage, resource requirements, and potential accuracy of learned embeddings. DeepBugs uses the 10,000 most frequent tokens to cover many identifier occurrences while replacing rarer tokens with an unknown placeholder.

  • Vocabulary choice: 10,000 tokens is the experimental vocabulary size, selected from about 2.4 million unique training tokens.All tokens outside the 10,000 most frequent are replaced with an unknown placeholder.
  • Trade-offs: Larger vocabularies cover more code locations but increase resource requirements and may reduce accuracy for uncommon identifiers.Rare identifiers may lack sufficient training data for effective embedding learning.
  • Coverage: A small number of frequent tokens covers a large percentage of token occurrences.Figure 4 plots included identifier occurrences against vocabulary size on a logarithmic horizontal axis.

6 RELATED WORK

Related work applies learning to code understanding, bug detection, and specification mining, but DeepBugs differs by learning name-based detectors from artificially seeded negative examples. It also targets buggy locations and name-related mistakes rather than only modeling correct code or labeling whole files.

  • Learning-based program analysis: Learning-based approaches use code regularities for tasks including completion, compilation-error fixing, fuzz-input generation, and code adaptation.These approaches learn from large collections of publicly available code.
  • Identifier-aware learning: Identifier-focused systems recover, summarize, or recommend names, whereas DeepBugs uses identifier representations to detect bugs.Prior systems address tasks such as recovering names from minified code, summarizing code, and predicting variable names or uses.
  • Learning-based bug detection: Existing learning-based bug detectors model API usage, vulnerabilities, or file-level defects, while DeepBugs targets name-related bugs and pinpoints buggy locations.Some prior models learn from positive examples only, others use manually created labels, and file-level defect prediction does not localize bugs.
  • Specification mining and seeded examples: Specification-mining approaches learn from correct examples and flag unusual code, unlike DeepBugs, which distinguishes correct from incorrect code using seeded negatives.DeepBugs frames artificial negative-example construction as a program-oriented variant of noise-contrastive estimation.
  • Name-based bug detection: DeepBugs is motivated by manually developed name-based analyses, especially swapped-argument detection, but extends name-based detection to additional identifier-related bug patterns.The authors report no known identifier-based approaches for the other targeted detectors and relate their transformations to mutation operators.
  • Bug seeding: Automated bug seeding has also been used to evaluate security-vulnerability detectors, whereas DeepBugs seeds name-related bugs to train detectors.The distinction is the bug domain and the purpose of seeding.

7 CONCLUSIONS

DeepBugs learns name-based bug detectors from semantic identifier representations and artificially seeded bugs. On a large JavaScript corpus, the detectors achieve high accuracy and find programming mistakes in real-world code.

  • 7 CONCLUSIONS: DeepBugs learns classifiers that distinguish correct from incorrect code using semantic identifier representations and simple transformations that seed artificial bugs.The framework targets dynamically typed code, where identifier names provide information about intended semantics in the absence of static types.
  • 7 CONCLUSIONS: The authors envision DeepBugs complementing manually designed detectors and reducing the human effort required to create bug detectors.This long-term vision follows from learning detectors from existing code rather than writing all detector logic manually.
Loading 1805.11683v1…