PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Deep Learning for Software Engineering

Yanming Yang, Xin Xia, David Lo, John Grundy

arXiv:2011.14597v1cs.SE

TL;DR

Applying deep learning in software engineering involves choices among heterogeneous architectures and methods for controlling overfitting. This systematic literature review analyzed 142 primary studies to characterize these applications, finding rapid research growth and widespread use of RNN, CNN, and FNN architectures.

  • Problem

    Heterogeneous deep neural networks and overfitting make selecting suitable networks and controlling generalization important concerns in software engineering applications.

  • Method

    The study conducted a systematic literature review of 142 primary studies, with two researchers extracting data and another researcher checking the results.

  • Results

    The review found rapid growth in deep-learning-for-software-engineering research, with RNN, CNN, and FNN among the most widely used architectures and CNN employed in almost 90% of studies.

  • Takeaways & Limitations

    The review organizes deep-learning architectures, model-selection strategies, and training concerns while identifying challenges and opportunities for future software-engineering research.

  • Takeaways & Limitations

    Data collection may contain bias because disagreements can arise when extracting and classifying content from primary studies, despite researcher discussion and checking.

Abstract

from arXiv · show

In 2006, Geoffrey Hinton proposed the concept of training ''Deep Neural Networks (DNNs)'' and an improved model training method to break the bottleneck of neural network development. More recently, the introduction of AlphaGo in 2016 demonstrated the powerful learning ability of deep learning and its enormous potential. Deep learning has been increasingly used to develop state-of-the-art software engineering (SE) research tools due to its ability to boost performance for various SE tasks. There are many factors, e.g., deep learning model selection, internal structure differences, and model optimization techniques, that may have an impact on the performance of DNNs applied in SE. Few works to date focus on summarizing, classifying, and analyzing the application of deep learning techniques in SE. To fill this gap, we performed a survey to analyse the relevant studies published since 2006. We first provide an example to illustrate how deep learning techniques are used in SE. We then summarize and classify different deep learning techniques used in SE. We analyzed key optimization technologies used in these deep learning models, and finally describe a range of key research topics using DNNs in SE. Based on our findings, we present a set of current challenges remaining to be investigated and outline a proposed research road map highlighting key opportunities for future work.

3.1 Research Questions

This survey examines how deep learning is being applied in software engineering by tracking research trends, classifying models, and analyzing model-selection and training factors. It finds rapid growth, broad use of several DNN architectures, and strong concentration in recent conference and journal publications.

  • 3.1 Research Questions: The survey analyzes 142 primary studies to answer four questions about DL publication trends, techniques, training difficulties, and supported SE tasks.Two researchers extracted study data together, and another researcher double-checked the results.
  • 4.1 Publication trends of DL techniques for SE: No DL-based SE studies were found before 2015, while publications increased to 58 papers in 2019.The cumulative publication fit reached R^2 = 0.99447, and the authors project continued growth.
  • 4.2 Distribution of publication venues: 69% of publications appeared in conferences and symposiums, compared with 31% in journals; SANER had the most studies among conference venues with 22.ASE, ICSE, and MSR were the other conference venues with more than 10 studies.
  • 4.2 Distribution of publication venues: 70.4% of conference papers were published in 2018 and 2019, while journal publications also increased especially from 2018 to 2020.TSE accounted for the largest journal proportion in 2018 and 2019, while IST represented a large proportion in 2019 and 2020.
  • 5.2 DL technique selection strategy: Characteristic-based model selection was used by 69% of studies, compared with 25% for the second strategy and 6% for the third.The survey identifies three DL technique selection strategies for heterogeneous SE applications.
  • 5.1 Classification of DNNs in SE: The survey classifies 30 DNNs across layered, Encoder-Decoder, and AutoEncoder architectures and five model families.Layered architectures were most popular; RNN-based models appeared in 72 studies, CNN-based models in 48, and FNN-based models in 25.

6.1.1 What were the sources of datasets used for training DNNs?

Primary studies most often used open-source datasets, while some constructed new datasets or collected data from multiple sources. Industry real-world datasets were rarely used.

  • 45% of primary studies trained DNNs using open-source datasets.Widely accepted datasets were one reason for this choice.
  • 18% of primary studies constructed new datasets because suitable datasets were unavailable.
  • 4% of studies used real-world datasets from industry.
  • 33% of studies used large-scale datasets to verify model scalability and robustness.Many collected multiple small datasets from different sources to do so.
  • Among collected datasets, GitHub was the most frequent source at 51%, followed by different systems and projects at 26%.The app store accounted for 11%, followed by Stack Overflow and YouTube.

6.1.2 What are the types of SE datasets used in prior DL studies?

The survey grouped SE datasets into code-based, text-based, software repository-based, and user-based categories. Source code dominated, while documentation and repository information were also used extensively.

  • SE datasets were classified into code-based, text-based, software repository-based, and user-based data types.
  • 104 primary studies collected data from source code, commonly for software testing and maintenance.
  • Text-based data were the second most popular category, with bug reports and requirements documentation used most often.Logs, certifications, and design documentation appeared rarely.
  • 12% of studies crawled software repositories such as Stack Overflow and GitHub for useful information and patterns.Examples included questions and answers, tags, issues, commits, and pull requests.
  • Only 5 studies adopted user-based data, including user behavior, reviews, and interactions.

6.1.3 What input forms were datasets transformed into when training DNNs?

DL studies transformed SE data into token, tree or graph, feature or metric, pixel, and combined input forms. Token-based inputs were most common, while multiple forms were uncommon.

  • 6.1.3 What input forms were datasets transformed into when training DNNs?: One-hot encoding and Word2vec were frequently used to transform source data into vectors.Only 5 studies used one-hot encoding.
  • 6.1.3 What input forms were datasets transformed into when training DNNs?: Input forms included token-based, tree or graph-based, feature or metric-based, pixel-based, and combined representations.
  • 6.1.3 What input forms were datasets transformed into when training DNNs?: 8 studies converted image-based datasets into pixels, while only 7 processed datasets into multiple forms.
  • 6.1.3 What input forms were datasets transformed into when training DNNs?: 64 studies used token-based inputs, including source code and documentation converted into token sequences.This represented over 45% of studies.
  • 6.1.3 What input forms were datasets transformed into when training DNNs?: 25 studies used tree-based inputs and 4 used graph-based structures to analyze source-code structure.
  • 6.1.3 What input forms were datasets transformed into when training DNNs?: 33 studies used embedding techniques to generate feature-based vectors.

6.2.1 What learning algorithms are used in order to optimize the models?

The surveyed studies used multiple optimization algorithms and overfitting countermeasures. Adam was most common, while cross-validation, pooling, regularization, and dropout were frequently reported for overfitting control.

  • 131 of 142 studies identified their optimization method, while 11 did not report one.
  • Adam was the most commonly used optimizer; SGD and GD appeared in 21 and 15 studies, respectively.Back-propagation and fine-tuning were used in 14 and 11 studies.
  • Cross-validation was used in 40 studies to combat overfitting, followed by pooling in 25 studies.
  • Regularization and dropout were each used in 20 studies to combat overfitting.
  • Data enlargement, early stopping, and data balancing were used in 8, 6, and 5 studies, respectively.
  • Ensembling was least frequent, appearing in 1 study, while 4 studies proposed new algorithms for overfitting.
  • The surveyed studies reported no strong association between overfitting techniques and data types or input forms.

6.2.3 What measures are used to evaluate DL-based models?

The survey reports widely used metrics for evaluating DL-based SE models and highlights substantial variation in evaluation practice and reproducibility.

  • Precision, recall, F1-measure, and accuracy were widely accepted evaluation metrics for DL-based models.
  • Running time was used by some studies because DNN construction, training, and testing require substantial computational time.
  • Over 20 metrics were used across primary studies, including task-specific measures such as MRR and BLEU.
  • 53 studies provided public model links, while 62.7% of proposed models were difficult to reproduce because source code was unavailable.

7.2.1 Requirements analysis.

Deep learning supports requirements analysis by extracting structured information from natural-language requirements and automating verification-method classification.

  • User requirements are natural-language documents describing users’ specific needs or a software product’s services.
  • A self-trained Named-entity Recognition model combining Bi-LSTM and CNN extracted system states from requirements specifications.
  • A CNN classified requirements according to six predefined verification methods using multiclass and multilabel prediction.
  • Imperfect training data affected classifier performance, although testing results remained good.

7.3.1 Software design patterns detection.

Deep learning approaches address software design tasks through learned representations, image-based UI analysis, API-sequence generation, and semantic code search.

  • A CNN-based wireframe image autoencoder automatically generated labels for large-scale Android UI design datasets.
  • Feature Maps compressed abstract semantic graph subtrees into feature maps for identifying design-pattern instances with Random Forest and CNN models.
  • DeepAPI generated functional API usage sequences from natural-language queries using an attention-based GRU Encoder-Decoder.
  • Code-search methods embedded source code and queries into vector space to calculate their semantic correlation.

7.5.1 Defect prediction.

Defect prediction is the most extensive and active DL research topic in software maintenance, spanning metrics-based, semantic, just-in-time, and security-related approaches.

  • Almost 30% of primary studies focused on defect identification, making defect prediction the most extensive maintenance topic.
  • SDAEsT used stacked denoising autoencoders for robust feature extraction and a two-stage ensemble to address class imbalance and overfitting.
  • DBN-based semantic defect prediction learned features from token vectors extracted from programs’ abstract syntax trees.
  • DeepJIT extracted features from code changes and commit messages with a CNN, achieving improvements over 10% in AUC on two open-source datasets.
  • DL-based studies also addressed aging-related bugs, text-layout bugs, vulnerabilities, bug localization, and syntax-error localization.

7.5.7 Testing techniques.

Deep learning supports diverse software testing and maintenance-related tasks, including fuzzing, game testing, code clone detection, and automated code commenting.

  • Testing techniques: Wuji applies an FNN model to automate game testing after analyzing 1,349 real bugs.
  • Testing techniques: DeepSmith uses an LSTM-based generative model to produce tens of thousands of realistic compiler-input programs for differential testing.The generated programs support compiler validation and bug discovery.
  • Testing techniques: Software maintenance studies apply deep learning to improve source code, logging information, and software energy consumption.
  • Testing techniques: Code clone detection studies use RNNs, FNNs, tree-based convolution, and graph-based representations to capture lexical, syntactic, semantic, and structural information.
  • Testing techniques: DL-based approaches also generate code comments by learning program structure with LSTM models.

7.6.3 Program repair.

Deep learning is applied across program repair, source-code modeling, software categorization, code-quality analysis, and maintainability prediction, with outcomes varying by task and approach.

  • Program repair: A neuro-symbolic repair approach corrected syntax errors in 60% of submissions and found functionally correct repairs for 24%.
  • Program repair: Deep learning language models significantly outperformed traditional language models for code suggestion across 16,221 Java projects.
  • Program repair: DNNs support cross-language software categorization by jointly encoding program syntax and semantics from bilateral implementations.
  • Program repair: DL-based code-quality work includes linguistic anti-pattern detection, SATD classification, code readability classification, and maintainability prediction.
  • Program repair: Feature reduction techniques achieved better maintainability-prediction results than using DNNs alone.

7.7.1 Effort estimation.

The surveyed work covers effort estimation and software-repository mining, organizing SE tasks into regression, classification, ranking, and generation categories.

  • Effort estimation: Only 39% of software projects are finished and published on time, motivating development-cost assessment for reliable delivery within schedule and budget.
  • Effort estimation: MLP and RBFNN achieved higher effort-estimation accuracy than the MLR model.
  • Software repository mining: Deep learning studies mine Stack Overflow, GitHub, and YouTube repositories for software-engineering information.
  • Software repository mining: A CNN and AutoEncoder identified Java code in YouTube videos with 85.6%-98.6% accuracy using learned syntactic and contextual features.
  • Software repository mining: The survey groups SE activities into six areas and classifies tasks as regression, classification, ranking, or generation; most studies are classification tasks.
  • Software repository mining: Seventeen specific SE research topics use deep learning techniques.

8 LIMITATIONS

The survey identifies potential data-extraction bias and the possibility of missing relevant studies during literature search and selection.

  • Limitations: Data extraction may contain classification disagreements, although researcher discussion and adjudication were used to mitigate this bias.
  • Limitations: The literature search may miss relevant deep-learning studies because many publication venues publish software-engineering research.

9 CHALLENGES AND OPPORTUNITIES

The survey identifies limited coverage, low transparency, scarce industry validation, and data dependence as challenges for deep learning in software engineering. It outlines broader research opportunities and reports rapid growth in this research area.

  • Using DL in more SE activities: Deep learning remains underused in software requirements and design despite their importance in the software development life cycle.The survey suggests applying DL to new research topics and classical requirements and design problems.
  • The transparency of DL: Few studies explain their architecture choices or the necessity of individual DNN layers, limiting transparency because DL decisions are difficult to interpret.The survey attributes this difficulty to the black-box nature of DL.
  • DL in real scenarios: Only 4% of studies used industry datasets for training and evaluation, leaving proposed models insufficiently validated on real industry data.The survey calls for greater industry-academia collaboration because industry data can differ substantially from open-source data.
  • Data hungry: Deep learning research in software engineering depends heavily on large public labeled datasets, although such data are rarely available for most SE tasks.The survey notes that DNN training requires massive data and cannot observe every possible labeled sample.
  • Research landscape: Research interest in deep learning for software engineering grew rapidly, while the survey investigated applications across 142 primary studies from 21 publication venues.The conclusion frames the survey as a systematic literature review spanning multiple SE tasks and DL architectures.
Loading 2011.14597v1…