PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Use of Deep Learning in Software Defect Prediction

Görkem Giray, Kwabena Ebo Bennin, Ömer Köksal, Önder Babur, Bedir Tekinerdogan

arXiv:2210.02236v1cs.SE

TL;DR

Manual feature extraction makes conventional software defect prediction time-consuming and limited in capturing semantic information, motivating deep-learning approaches. This study systematically synthesizes DL use in SDP and finds modest performance gains over traditional ML, alongside reproducibility and data challenges.

  • Problem

    Conventional software defect prediction requires manually extracted features that are time-consuming and may poorly capture semantic information from bug reports.

  • Method

    The study systematically analyzes the current use of deep-learning algorithms for software defect prediction across the surveyed literature.

  • Results

    Deep-learning models improved prediction performance, but the improvement was not significant compared with traditional machine-learning models.

  • Takeaways & Limitations

    Future work should capture features from diverse software artifacts, address class imbalance with data augmentation, and publish replication packages.

  • Takeaways & Limitations

    Hyperparameter sensitivity and incomplete reporting create serious challenges for transparency and reproducibility.

Abstract

from arXiv · show

Context: Automated software defect prediction (SDP) methods are increasingly applied, often with the use of machine learning (ML) techniques. Yet, the existing ML-based approaches require manually extracted features, which are cumbersome, time consuming and hardly capture the semantic information reported in bug reporting tools. Deep learning (DL) techniques provide practitioners with the opportunities to automatically extract and learn from more complex and high-dimensional data. Objective: The purpose of this study is to systematically identify, analyze, summarize, and synthesize the current state of the utilization of DL algorithms for SDP in the literature. Method: We systematically selected a pool of 102 peer-reviewed studies and then conducted a quantitative and qualitative analysis using the data extracted from these studies. Results: Main highlights include: (1) most studies applied supervised DL; (2) two third of the studies used metrics as an input to DL algorithms; (3) Convolutional Neural Network is the most frequently used DL algorithm. Conclusion: Based on our findings, we propose to (1) develop more comprehensive DL approaches that automatically capture the needed features; (2) use diverse software artifacts other than source code; (3) adopt data augmentation techniques to tackle the class imbalance problem; (4) publish replication packages.

1 INTRODUCTION

Software defect prediction aims to identify faulty software early, reducing the time and effort required by fully manual testing and reviews. This study systematically examines how deep learning is being used to improve SDP models and guide future research.

  • Early defect prediction can guide developers toward improved code quality at lower cost than fully manual approaches.
  • Machine-learning SDP approaches require manually extracted features that are time-consuming to construct and poorly capture semantic information from bug-reporting tools.
  • Deep-learning techniques automatically extract higher-level features and learn from complex, high-dimensional data.
  • The study uses a systematic literature review of 102 peer-reviewed studies with quantitative and qualitative analysis.

2 BACKGROUND AND RELATED WORK

SDP pipelines collect software artifacts, extract and balance data, train prediction models, predict faulty parts, and evaluate performance across varied scenarios and architectures. This study distinguishes itself through a systematic review focused specifically on DL for SDP, covering 102 primary studies through the end of 2021.

  • Software defect prediction: SDP typically collects clean and defective samples, extracts features, optionally balances data, trains a model, predicts faulty parts, and evaluates performance.
  • Software defect prediction: Software artifacts for SDP include source code, commit messages, bug reports, and other repository data.
  • Software defect prediction: Feature extraction converts software artifacts into model inputs such as metrics, change histories, assembly code, or source code.
  • Software defect prediction: Class balancing addresses datasets with fewer faulty than non-faulty parts, which can otherwise produce misleading evaluation results.
  • Software defect prediction: SDP scenarios include within-project, cross-project, heterogeneous, and change-level prediction settings.
  • Related work: The review covers DL for SDP through a systematic literature review of 102 primary studies published by the end of 2021.

3 RESEARCH OBJECTIVES AND METHOD

The study uses a systematic literature review to classify and synthesize evidence on deep learning for software defect prediction. Its research questions cover the SDP lifecycle, study characteristics, modeling choices, evaluation, reproducibility, and reported challenges.

  • Method: The authors adopted a systematic literature review approach based on established guidelines to synthesize knowledge about DL algorithms for SDP.The protocol was informed by Kitchenham and Charters, Wohlin, and prior SLR examples.
  • Eligibility criteria: Studies were included when they were English scholarly publications reporting empirical results from at least one DL algorithm applied to SDP.Studies without available full text, secondary studies, and studies using only traditional ML or statistical techniques were excluded.
  • Research objectives: The review examines SDP scenarios, ML categories, datasets, source-code representations, prediction granularity, class-imbalance techniques, DL algorithms, evaluation, reproducibility, and challenges.These topics are organized as research questions spanning the ML model lifecycle.
  • Study selection: The search strategy combined five online databases, manual searches of related reviews, and forward snowballing using Google Scholar.The databases were ACM, IEEE Xplore, ScienceDirect, Springer, and Wiley.
  • Quality assessment and extraction: Primary studies underwent quality assessment and structured data extraction, with disagreements resolved through author discussion during extraction.Quality assessment used a three-point Likert scale, while the extraction form was based on the research questions.

4 RESULTS

The reviewed literature shows growing use of DL for SDP, with studies spanning multiple prediction scenarios. Within-project prediction dominates, while cross-version and heterogeneous prediction receive less attention.

  • Publication trend: DL-based SDP research showed an increasing publication trend, indicating a recent and ongoing research focus as of 2019.The number of primary studies increased over the years.
  • SDP scenarios: 82 studies included within-project defect prediction, 42 involved cross-project defect prediction, and 22 examined both scenarios.These counts describe overlap across the reported SDP scenarios.
  • SDP scenarios: Ten studies focused on cross-version defect prediction, including six classified under within-project prediction.One study evaluated 32 cross-version pairs from 45 versions of 13 software projects.
  • SDP scenarios: Just-in-time defect prediction studies trained models within individual projects to predict defect-prone changes, while other work addressed limited labeled data in mobile applications.The reviewed scenarios therefore included change-level prediction and application-specific data constraints.
  • SDP scenarios: Three studies addressed heterogeneous defect prediction using neural networks, deep adversarial learning, or autoencoders for heterogeneous or multi-source data.The autoencoder extracted intermediate features rather than simply removing redundant and unrelated features.

4.2 ML CATEGORIES

Supervised learning overwhelmingly dominates DL-based SDP studies, while unsupervised and semi-supervised approaches appear in smaller subsets of the literature. Unsupervised learning emerged later and subsequently declined in use.

  • Supervised learning: 94% of the primary studies, corresponding to 96 studies, applied supervised deep learning.77 studies used supervised learning without another ML category.
  • Unsupervised learning: 23 studies involved unsupervised deep learning, with most combining it with supervised learning.Three studies used unsupervised learning alone, while 19 combined supervised and unsupervised learning.
  • Semi-supervised learning: Semi-supervised learning was used in studies by Shi et al., Sun et al., and Xu et al.Sun et al. and Xu et al. used only semi-supervised learning, whereas Shi et al. combined unsupervised and semi-supervised learning.
  • Publication trend: Unsupervised learning first appeared in the reviewed studies in 2018, with 10 of 23 such studies published in 2019.The review reports four studies using unsupervised learning in 2020 and five in 2021.

4.3 TRAINING AND TESTING DATASETS

The studies used many datasets, but frequently relied on public Java, C, and C++ repositories, especially PROMISE. Commercial data and newer programming languages were comparatively uncommon.

  • Dataset sources: Researchers used more than eight datasets on average per study, and frequently used datasets developed in Java, C, or C++.The 11 most frequently used datasets came from the PROMISE repository and were developed using Java.
  • Dataset sources: The frequently used public datasets include Java projects such as Synapse, Jedit, Ant, Ivy, Velocity, and Eclipse JDT.These datasets were among those listed as used in ten or more primary studies.
  • Dataset diversity: Some studies used projects written in C++, Java, and Go, while others constructed datasets from Android applications and online repositories.Examples include QT, OpenStack, JDT, Platform, Gerrit, Wikipedia, Chess, GitHub, and Codeforces.
  • Commercial data: Only one study included commercial projects, combining six open-source projects with five commercial projects.The study by Albahli used both project types for model construction and defect prediction.

4.4 REPRESENTATION OF SOURCE CODE

Studies represented source code for DL-based defect prediction using metrics, intermediary forms such as ASTs, and direct numeric vectors. Metrics were the most frequent representation, while AST-based representations supported semantic and structural modeling.

  • Intermediary representations: 40 studies used an intermediary representation, while some studies combined metrics with intermediary representations.Intermediary representations convert textual source code into a form processable by DL algorithms while attempting to limit information loss.
  • Metric-based representations: 67 studies used software metrics to represent source code for defect prediction.The metrics included software size, structure, process, and product measures.
  • Intermediary representations: 38 studies used Abstract Syntax Trees as intermediary representations for constructing numeric vectors.AST tokens could be mapped to fixed-length vectors and supplied to sequence models such as LSTMs.
  • Alternative representations: Other representations included program and data flow graphs, assembly instruction sequences, and source-code images.These choices were used to model dependencies, program behavior, or source-code character values for image classification.
  • Combined representations: Eight studies combined AST-based inputs with software metrics.These combinations included AST word embeddings or node vectors alongside metric vectors and, in one approach, change-semantic information.
  • Direct representations: Two studies converted source code directly into numeric vectors without an intermediary representation.The direct conversions used word vectors for commit messages and code changes or fixed-length Word2vec representations.

4.5 GRANULARITY LEVEL OF PREDICTION

DL-based defect prediction models operated at multiple granularity levels, with file and module prediction most common. Recent studies increasingly examined finer-grained levels such as classes, statements, procedures, and changes.

  • Reported granularity: 37 studies predicted defects at file level, while 32 studies predicted defects at module level.These were the most frequently reported prediction granularities in the surveyed studies.
  • Reported granularity: 12 studies built models to identify buggy changes.The surveyed granularity levels also included class, function, procedure, and statement prediction.
  • Fine-grained prediction: Since 2019, nine studies reported class-, statement-, or procedure-level prediction experiments.The nine studies comprised two in 2019, four in 2020, and three in 2021.
  • Fine-grained prediction: Starting in 2018, studies also reported change-level defect prediction using DL algorithms.Eleven such studies were published during the following three years.
  • Multi-granularity prediction: Three studies reported experiments at more than one granularity level.One DBN-based approach learned semantic features from ASTs for file-level models and from source-code changes for change-level models, with reported improvements from 2.9% to 13.3%.

4.6 TECHNIQUES FOR DEALING WITH THE CLASS IMBALANCE PROBLEM

Class imbalance is a central concern in SDP because defective samples are usually the minority class and DL models may neglect them. Researchers mainly addressed it through data-level sampling, especially oversampling, with some algorithm-level methods and comparative studies of alternatives.

  • Imbalance-handling techniques: 63 studies used data-level techniques to address class imbalance, while nine used algorithm-level techniques.Six studies applied more than one technique, so the total number of techniques exceeded the number of studies.
  • Data-level techniques: Oversampling was the most frequently used data-level technique, appearing in 42 studies.Oversampling duplicates or synthetically creates minority-class instances, including defective samples.
  • Data-level techniques: 21 studies used SMOTE for oversampling.ADASYN was also used in some studies as an extension of SMOTE.
  • Synthetic data generation: GAN, VAE, and WGAN-GP were used to generate additional defective training instances.Some experiments combined generative approaches with SMOTE.
  • Algorithm-level techniques: Algorithm-level approaches included cost-sensitive learning and ensemble learning.Cost-sensitive methods assigned different misclassification costs to classes, while ensemble methods included two-stage and bagging-based approaches.
  • Comparative findings: Comparative studies reported improved DL performance from oversampling, fuzzy sampling, and synthetic-sample generation with VAE on imbalanced datasets.One study evaluated 240 data points across 24 datasets, while another evaluated ten imbalanced datasets.

4.7 DL APPROACHES

CNN was the most frequently used DL algorithm in the surveyed SDP studies, followed by recurrent architectures, MLP, and DBN. Less common approaches included generative, hybrid, hierarchical, Siamese, extreme-learning, and graph-based models.

  • Frequently used approaches: CNN was the most frequently used DL algorithm in the surveyed SDP studies.RNN/LSTM/GRU, MLP, and DBN were also widely used, while 13 studies used encode-decoder architectures, mostly autoencoders.
  • Frequently used approaches: CNN, RNN/LSTM/GRU, and MLP were the three most frequently used DL approaches across the surveyed studies.The paper notes that these algorithms are well known and have performed well across many tasks.
  • Less frequent approaches: Less frequently used algorithms included GANs, hybrid and hierarchical neural networks, Siamese networks, extreme learning machines, and graph neural networks.Reported motivations included synthetic-data generation, combining supervised and unsupervised learning, and extracting semantic features.

4.8 EVALUATION METRICS AND VALIDATION APPROACHES

The studies used varied evaluation metrics and validation approaches for defect prediction. F-measure was most frequent, while hold-out and cross-validation were the dominant validation strategies.

  • 68% of studies (69) used F-measure, making it the most frequently reported evaluation metric.
  • 48 studies used Recall, while 40 studies used AUC and 40 used Precision.Recall measures successfully predicted defects; AUC measures class discrimination, and Precision measures correctly predicted defects among predictions.
  • 13 studies used effort-aware performance measures, including Popt, PofB20, and EARecall.These measures account for differences in the cost of applying quality assurance activities to code.
  • G-measure, ROC curve, MSE, and TNR/Specificity were also reported among the evaluation metrics.G-measure balances detection and false alarms, while ROC compares true-positive and false-positive rates.
  • 60 studies used hold-out validation, 43 used cross-validation, and one used both approaches.

4.9 REPRODUCIBILITY PACKAGE

The review examined reproducibility packages in the primary studies and found limited sharing, with no upward trend over time. Only a small number of studies shared packages in the later years examined.

  • Reproducibility requires other researchers to obtain the reported results using the authors’ artifacts and experimental setup.
  • Only one study shared a reproducibility package in 2017 and one in 2018.
  • In the last three years, only three studies shared packages among sets of 33, 26, and 30 studies.

4.10 CHALLENGES AND PROPOSED SOLUTIONS

The review organizes challenges in deep-learning software defect prediction around data engineering, model development, and general lifecycle concerns. Proposed solutions address heterogeneous or limited data, representation issues, model sensitivity, and prediction constraints.

  • The qualitative analysis extracted challenges and proposed solutions from 50 primary studies and mapped them to three ML lifecycle categories.The categories are Data Engineering, Model Development, and General concerns.
  • Data Engineering: Heterogeneous data across projects, versions, and features can degrade classifier performance.
  • Data Engineering: Limited training data makes software defect prediction difficult; proposed solutions include suitable architectures and adding labelled data.
  • Data Engineering: Invalid defect instances can enter training data, while manual validation may introduce bias.
  • Data Engineering: Incomplete code snippets challenge change-level prediction, particularly for AST-based approaches; heuristic extraction is proposed.
  • Model Development: AST granularity, feature context, redundancy, fixed-length vectors, and sequential architectures create representation or modeling challenges.Proposed responses include multi-kernel transfer CNNs, LSTMs, feature selection, latent or varying-length features, and non-sequential networks.
  • Model Development: Traditional source-code metrics may miss program semantics and lead to poor prediction performance.
  • Model Development: Hyperparameter sensitivity can produce very different performance results, and incomplete reporting creates transparency and reproducibility challenges.

5 DISCUSSION

The review identifies persistent challenges in applying deep learning to software defect prediction, including limited gains over traditional machine learning, class imbalance, feature-representation complexity, and weak reproducibility. It recommends broader data sources, augmentation, comprehensive workflows, replication packages, and stronger attention to industrial adoption and study validity.

  • Findings: DL models improved prediction performance, but their improvement over traditional ML models was not significant.The authors suggest that limited data may partly explain this result.
  • Findings: WPDP remained the dominant defect scenario, although almost a third of studies conducted CPDP, which can leverage heterogeneous datasets for DL training.The review states that WPDP models perform significantly better than CPDP models because their data are more similar and homogeneous.
  • Findings: Most studies relied on manually extracted metrics or source-code representations such as ASTs, limiting DL’s ability to learn richer features automatically.AST conversion also adds tooling complexity, especially for non-Java source code.
  • Findings: Only 11% of studies provided source code and datasets, underscoring a reproducibility gap for DL-based SDP experiments.The review attributes this partly to complex experiments, manual parameters, and time-consuming optimization.
  • Recommendations: Industrial adoption remains weak: the literature offers little evidence from commercial datasets, and one survey found only 7.8% of practitioners willing to use defect-prediction tools.The review links adoption challenges to data privacy, security, complexity, and incompatibility with organizational environments.
  • Recommendations: The review recommends comprehensive DL approaches, diverse software artifacts, data augmentation for class imbalance, identification of key defect attributes, and public replication packages.Recommended artifacts include bug reports, requirements, tests, and graphical documents; augmentation is conditioned on efficiently converting data into images.

6 CONCLUSION AND FUTURE WORK

The study reviews deep-learning-based software defect prediction to characterize the field’s state of the art, challenges, and research directions. It highlights reproducibility gaps and proposes broader representations, data augmentation, common evaluation criteria, practical integration, and open science.

  • Conclusion: The study systematically reviews existing software defect prediction techniques using deep learning to characterize the state of the art.Its analysis covers scenarios, datasets, representations, prediction granularity, class imbalance, algorithms, evaluation, validation, reproducibility, and challenges.
  • Findings: Research has increased in recent years, using diverse techniques, datasets, and validation approaches, but most surveyed articles lack reproducibility packages.The authors identify the absence of reproducibility packages as problematic for transparency and further field advancement.
  • Future work: Future work should develop comprehensive deep-learning approaches that learn richer features from heterogeneous software artifacts and use data augmentation for limited datasets and class imbalance.The proposed artifacts include source code, bug reports, and other sources.
  • Future work: The authors also call for common evaluation criteria, greater attention to cross-project defect prediction, improved tool usability, daily-practice integration, reproducibility, and open science.These directions are presented as priorities for advancing deep-learning-based software defect prediction.
  • Implications: The review is intended to help newcomers understand the landscape and established researchers prioritize future efforts, with meta-analysis identified as planned future work.The proposed meta-analysis would investigate factors influencing reported prediction performance.

8 APPENDICES

The appendices document the review’s search and selection materials, surveyed publication venues and studies, and selected challenges in deep-learning-based software defect prediction. They also show that limited training data, semantic feature limitations, and early prediction remain important concerns.

  • Search strategy: The appendices record database-specific search queries used to identify studies on software faults, defects, bugs, quality, prediction, estimation, and deep learning.Queries were adapted to the search interfaces of ACM, Wiley, IEEE Xplore, ScienceDirect, and Springer.
  • Study selection: Fourteen studies were excluded because they did not meet the review’s quality-assessment criteria.The appendices list the excluded studies and their bibliographic information.
  • Study distribution: IEEE Access contributed 10 primary studies, while IET Software contributed 5; several other listed venues contributed between 2 and 4 studies.The venue table reports the number of primary studies and associated references.
  • Challenges and solutions: Insufficient training data makes successful defect-prediction model development difficult, motivating architectures that learn from few samples, including Siamese networks.The cited proposals address limited sample availability rather than treating it solely as a model-selection issue.
  • Challenges and solutions: Traditional features such as lines of code and McCabe complexity may miss program semantics, while early defect prediction remains challenging.The review connects these concerns to the motivation for deep-learning approaches and richer representations.
  • Challenges and solutions: A genetic algorithm combined with a deep neural network was reported to improve performance through feature optimization.The reported improvement is attributed to applying the optimization technique before classification.
Loading 2210.02236v1…