Source-linked AI summary
A deep tree-based model for software defect prediction
Hoa Khanh Dam, Trang Pham, Shien Wee Ng, Truyen Tran, John Grundy, Aditya Ghose, Taeksu Kim, Chul-Joo Kim
TL;DR
Existing defect-prediction features often fail to capture code syntax and levels of semantics, limiting how source code is represented. The paper uses a tree-structured LSTM aligned with an AST to learn features automatically. Its evaluation reports effectiveness on Samsung and PROMISE datasets for within- and cross-project prediction.
Problem
Existing metric- and token-based defect-prediction features do not sufficiently capture source-code syntax and different levels of semantics.
Source: §1 · p. 1Method
Results
The evaluation on Samsung projects and the PROMISE repository demonstrates effectiveness for both within-project and cross-project defect prediction.
Source: §1 · p. 2Takeaways & Limitations
The system takes an AST as input, predicts whether a file is defective or clean, and learns features without manual feature engineering.
Source: §8 · p. 9The authors note that their dataset may not represent all software applications and call for investigation of other application types and programming languages.
Source: §6.4 · p. 8
Abstract
from arXiv · showhide
Defects are common in software systems and can potentially cause various problems to software users. Different methods have been developed to quickly predict the most likely locations of defects in large code bases. Most of them focus on designing features (e.g. complexity metrics) that correlate with potentially defective code. Those approaches however do not sufficiently capture the syntax and different levels of semantics of source code, an important capability for building accurate prediction models. In this paper, we develop a novel prediction model which is capable of automatically learning features for representing source code and using them for defect prediction. Our prediction system is built upon the powerful deep learning, tree-structured Long Short Term Memory network which directly matches with the Abstract Syntax Tree representation of source code. An evaluation on two datasets, one from open source projects contributed by Samsung and the other from the public PROMISE repository, demonstrates the effectiveness of our approach for both within-project and cross-project predictions.
1 INTRODUCTION
Defect prediction helps prioritize inspection, but existing metric- and token-based features do not sufficiently capture code syntax and semantics. The paper proposes a tree-structured LSTM model that learns source-code features and evaluates it on Samsung and PROMISE projects.
1 INTRODUCTION
- Metric features may not reflect code syntax or semantics and may not generalize well across projects.Prior work uses features such as code size, complexity, churn, and process metrics.Source: §1 · p. 1
- Bag-of-Words misses differences in code order and syntax, while recent structure-aware work may still miss syntax and token semantics.The passage gives “for” and “while” as tokens whose semantic relation may be overlooked.Source: §1 · p. 1
- The model adapts a tree-structured LSTM to match an input source file’s Abstract Syntax Tree, with each AST node corresponding to an LSTM unit.The approach is intended to capture long-context relationships among code elements.Source: §1 · pp. 1–2
- The prediction system learns features automatically from raw AST input to classify files as defective or clean, avoiding manual feature engineering.The paper reports evaluation on Samsung open-source projects and the PROMISE repository.Source: §1 · p. 2
2 MOTIVATING EXAMPLE
The motivating example shows why defect predictors need to distinguish code structure and semantics, not just metrics or token frequencies. The paper proposes modeling source code with a deep tree-based LSTM over Abstract Syntax Trees.
2 MOTIVATING EXAMPLE
- The two listings have identical software metrics, so metric-based features would not distinguish the defective code from the corrected version.The defect is a possible underflow when pop is called on an empty stack; the second listing checks that the stack is not empty.Source: §2 · p. 2
- The listings also have identical code tokens and frequencies, so term-frequency features would miss their different defectiveness.The difference is where the if statement appears in the code.Source: §2 · p. 2
- The example motivates representing syntactic structure because the if statement’s location changes behavior, while n-grams cover only a few code elements and impose sequence patterns.Some code elements can be swapped without changing behavior, and dependent elements may be far apart.Source: §2 · p. 2
- The example also motivates capturing token semantics: “for” and “while” can be behaviorally similar, yet existing approaches may overlook that relation.The paper proposes a deep tree-based LSTM to model ASTs and preserve syntactic and structural information.Source: §2 · p. 2
3 APPROACH
The approach represents a source file as an AST, learns a whole-tree vector with a tree-structured LSTM, and uses a classifier to predict defectiveness. It is trained from labeled files and can also use attention to identify likely defect-causing parts.
3 APPROACH
- The model learns a classification function from examples of files labeled defective or clean.It predicts defectiveness for new files within the same project or in a different project.Source: §3 · p. 3
- An attention mechanism lets the model locate source-file parts, such as code lines, that are likely to cause a defect.The paper says this can help explain what the model considers for specific defects.Source: §3 · p. 3
- The approach parses source code into an AST, maps its nodes to embeddings, and inputs those embeddings to a tree-based LSTM.The network produces a whole-AST vector that a traditional classifier uses to predict defect outcomes.Source: §3 · p. 3
- A traditional classifier, such as Logistic Regression or Random Forests, predicts defect outcomes from the whole-AST vector.The prediction function approximates whether a source file is defective or clean.Source: §3 · pp. 2–3
4 MODEL BUILDING
The model represents source files as ASTs, embeds AST-node labels as vectors, and processes the trees with Tree-LSTM units to predict file defectiveness.
4.1 Parsing source code
- Each source file is parsed into an AST whose nodes represent code constructs, with identifiers labeled by name and constants represented by their AST type.Comments, blank lines, punctuation, and delimiters are ignored.Source: §4.1 · p. 3
4.2 Embedding AST nodes
- The ast2vec embedding maps each AST-node label to a fixed-length vector, placing nodes that frequently occur in similar contexts near one another.The embedding matrix is randomly initialized and adjusted during training.Source: §4.2 · p. 4
4.3 Defect prediction model
- For prediction, the root AST's Tree-LSTM representation feeds a classifier, which labels the file defective when its predicted probability is at least 0.5.The prediction function returns 1 for defective and 0 otherwise.Source: §4.3 · p. 4
- Each Tree-LSTM unit recursively combines child states with the AST node embedding, using child-specific forget gates and input and output gates to update and expose memory.The child forget gates selectively include information from each child.Source: §4.3 · p. 4
5 MODEL TRAINING
The model learns Tree-LSTM parameters by predicting AST parent labels from children, then trains defect classifiers on automatically generated file features. Validation guides model selection and early stopping.
5.1 Training Tree-LSTM
- Tree-LSTM units are trained unsupervised on AST branches to predict each parent node’s label from child-node information.The model uses child hidden states to predict the parent label through a softmax function.Source: §5.1 · p. 5, Figure 5 · p. 5
- Training iteratively adjusts embeddings and weight matrices to minimize prediction-error loss, using stochastic gradient descent with RMSprop for optimization.The loss is based on the difference between predicted and actual outcomes.Source: §5.1 · p. 5
- The best validation-performing model is evaluated on the test set, and training stops when validation performance worsens.Validation guides hyperparameter assessment but is not used to learn model parameters.Source: §5.1 · pp. 5–6
- Dropout randomly zeros input and output state elements during training, while parameter averaging is used during testing.Perplexity, based on log loss, helps choose the best Tree-LSTM model and guide early stopping.Source: §5.1 · p. 6
5.2 Training defect prediction model
- Automatically generated file features and labels train Logistic Regression and Random Forest classifiers to predict whether files are defective.Random Forests combine estimates from many decision trees.Source: §5.2 · p. 6
6 EVALUATION
The evaluation tests the approach on Samsung and PROMISE data, using within-project and cross-project prediction settings. Results show strong recall, alongside precision and validity caveats.
6.2 Performance measures
- Because the data are imbalanced and defective files matter more, evaluation focuses on defective-class precision, recall, F-measure, and threshold-insensitive AUC.The study treats predicting defective files as more important than predicting clean files.Source: §6.2 · pp. 6–7
6.3 Results
- For Samsung, Random Forests scored well above 0.9 on all four indicators—F-measure, precision, recall, and AUC—while Logistic Regression achieved very high recall but much lower precision than Random Forests.The passage attributes AUC values of 0.98 and 0.60 to Random Forests.Source: §6.3.1 · p. 7
6.4 Threats to validity
- PROMISE lacked source files, comparison with Wang et al. relied on reported results, and statistical significance tests were left for future work.The authors also caution that the datasets may not represent all software applications.Source: §6.4 · pp. 7–8
6.3 Results
- For Samsung within-project prediction, LR achieved recall of 0.86 averaged across 16 cases, 23% above Wang et al.’s approach; lower precision reduced F-measure by 17% versus that approach.Average AUC was 0.6 across 16 cases, well above random prediction; high recall is generally preferable because the cost of missing defects is much higher than having false positives.Source: §6.3.1 · p. 7, Figure 7 · p. 8
- Across 22 cross-project cases, average recall was 0.8, with 15 cases above 0.8; average F-measure was 0.5 amid low precision.Average AUC remained above the 0.5 random-prediction threshold.Source: §6.3.2 · p. 7, Figure 8 · p. 8
6.4 Threats to validity
- Samsung defect labels came from internal static-analysis warnings, which may include false positives.The authors identify validation of those warnings as future work.Source: §6.4 · p. 8
7 RELATED WORK
Prior defect-prediction work used engineered code and process metrics, while deep-learning approaches learned features but had limits in capturing source-code structure. Related code-modeling work explored recurrent and other neural architectures; this paper uses Tree-LSTM to match the AST representation.
7.1 Defect Prediction
- Traditional defect-prediction models use static code and process features, including size, complexity, and development-change activity.Process metrics are language-independent, and models using these features have been evaluated within and across projects.Source: §7.1 · p. 9
- Cross-project models train on other projects’ historical data, but accurate prediction is difficult when data are insufficient.Prior work evaluated 622 cross-project models across 12 applications; a universal model reported performance similar to within-project prediction and on five other projects.Source: §7.1 · p. 9
- Prior deep-learning defect-prediction work used DBNs, which improved predictive performance but do not naturally capture source-code order and long-term dependencies.The cited DBN approach outperformed software-metric and Bag-of-Words approaches; finer method- or line-level prediction would require datasets labeled at those levels.Source: §7.1 · p. 9
7.2 Deep learning in code modeling
- Earlier software-engineering work proposed DeepSoft, an LSTM framework for modeling software and its development and evolution processes.The authors describe this paper as realizing one of DeepSoft’s proposed applications.Source: §7.2 · p. 9
- Prior neural code-modeling work includes RNNs, encoder–decoders, and CNNs; this paper instead uses Tree-LSTM to match source code’s AST structure.The cited examples include code modeling, clone detection, API-sequence generation, error fixing, and bug localization.Source: §7.2 · p. 9
8 CONCLUSIONS AND FUTURE WORK
The paper presents an AST-input Tree-LSTM model that predicts whether a source file is defective or clean, automatically learning features and capturing code dependencies. Evaluation on Samsung and PROMISE datasets produced promising results; future work proposes broader application, finer prediction levels, additional defect types, and a practical tool.
8 CONCLUSIONS AND FUTURE WORK
- The Tree-LSTM takes a source-file AST and predicts whether the file is defective or clean, automatically learning features and capturing long-term code dependencies.The authors state that matching the tree-structured network to the AST captures syntax and different levels of semantics, avoiding manual feature engineering.Source: §8 · p. 9
- Evaluation on Samsung and PROMISE datasets produced promising results, which the authors say indicate the approach can be applied in practice.Source: §8 · p. 9
- Future work proposes testing other applications and languages, predicting defects at method and code-change levels, and targeting specific defect types.The authors also plan to build a tool for software engineers and testers to use in real-life settings.Source: §8 · p. 10