PaperlayerView exact revision on arXiv

Source-linked AI summary

A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research

Cody Watson, Nathan Cooper, David Nader Palacio, Kevin Moran, Denys Poshyvanyk

arXiv:2009.06520v2cs.SEcs.AIcs.LGcs.NE

TL;DR

Deep learning’s rapid adoption in software engineering makes it difficult to distinguish established successes, failures, and opportunities, especially as traditional machine learning requires laborious feature engineering. This paper conducts a systematic literature review organized around the components of learning, synthesizing prior DL4SE research into a roadmap of applications, challenges, and best practices. The review finds diverse task coverage and task-specific relationships among data, loss functions, and architectures, while identifying limitations including scarce data, interpretability concerns, and search-methodology threats.

  • Problem

    Rapid DL adoption makes it difficult to distill successes, failures, and opportunities in software engineering, while conventional ML requires tedious feature engineering.

  • Method

    The paper conducts a systematic literature review and organizes its research questions around the components of learning to develop a taxonomy and roadmap for applying DL to SE tasks.

  • Results

    SE researchers apply DL to diverse tasks, and SE task is the most informative feature, correlating strongly with data, loss function, and architecture choices.

  • Takeaways & Limitations

    The roadmap identifies task-specific DL component combinations, open challenges, and considerations for applying complex models to automate software engineering tasks.

  • Takeaways & Limitations

    The review’s external validity may be threatened by the search string and filtering process used to identify relevant DL4SE work.

Abstract

from arXiv · show

An increasingly popular set of techniques adopted by software engineering (SE) researchers to automate development tasks are those rooted in the concept of Deep Learning (DL). The popularity of such techniques largely stems from their automated feature engineering capabilities, which aid in modeling software artifacts. However, due to the rapid pace at which DL techniques have been adopted, it is difficult to distill the current successes, failures, and opportunities of the current research landscape. In an effort to bring clarity to this crosscutting area of work, from its modern inception to the present, this paper presents a systematic literature review of research at the intersection of SE & DL. The review canvases work appearing in the most prominent SE and DL conferences and journals and spans 128 papers across 23 unique SE tasks. We center our analysis around the components of learning, a set of principles that govern the application of machine learning techniques (ML) to a given problem domain, discussing several aspects of the surveyed work at a granular level. The end result of our analysis is a research roadmap that both delineates the foundations of DL techniques applied to SE research, and highlights likely areas of fertile exploration for the future.

1 INTRODUCTION

Software engineering research increasingly applies deep learning to automate tasks over growing, complex software data, but the rapid adoption makes the field difficult to assess. This review maps the DL4SE literature and develops a roadmap of practices, omissions, and opportunities.

  • Growing software repositories contain large amounts of unstructured, unlabeled data, creating opportunities for machine-learning-based automation of SE tasks.Examples include requirements, design documents, source code, test cases, and defect reports.
  • Deep learning automatically extracts task-relevant features, reducing reliance on manually engineered attributes when modeling software artifacts.
  • The review systematically maps DL and SE research to identify completed work, successful approaches, improvements, and open challenges.It synthesizes a research roadmap and discusses reproducibility and replicability concerns.
  • The analysis is organized around five components of learning to identify best practices and promising directions for applying DL in SE.
  • SE4DL was identified as a related area, but its small and emerging literature is reserved for future work.
  • The review provides guidelines addressing commonly omitted issues such as sampling bias, data snooping, and model over- and under-fitting.

2 RESEARCH QUESTION SYNTHESIS

The review derives its research questions from the components of learning, using them to organize a taxonomy of DL4SE tasks, data, models, evaluation, and reporting. This structure also supports guidelines for clearer and more reproducible DL research.

  • Framework: The components of learning provide the conceptual framework for synthesizing research questions and constructing the review’s DL4SE taxonomy.They formalize conditions for computational learning and guide the review’s analysis.
  • Target function: The target-function questions identify which SE tasks DL approaches address and what inputs, outputs, learning types, features, and architectures they imply.
  • Data: The data questions examine which software artifacts are used, how they are extracted and preprocessed, and how exploratory analysis informs model design and training.
  • Models: The model questions examine architectures, learning algorithms, training processes, and methods for addressing over- and under-fitting.
  • Evaluation: The evaluation questions study model effectiveness through baselines, benchmarks, metrics, impact, and generalizability to practical scenarios.The review frames comparisons partly through the least-complex-model principle.
  • Reproducibility: The final questions assess whether studies describe learning components sufficiently for reproducibility and replicability, then derive flexible guidelines and checkpoints.

3 METHODOLOGY FOR SYSTEMATIC LITERATURE REVIEW

The review follows a staged systematic-literature-review process: formulate questions, search and filter studies, supplement the corpus, extract and synthesize data, and derive a taxonomy. Its search covered a defined period, databases, venues, and eligibility criteria.

  • Process: The methodology follows Kitchenham’s systematic-review guidelines and organizes the review into searching, filtering, sampling, extraction, synthesis, taxonomy derivation, and exploratory analysis.
  • Search scope: The search covered January 1, 2009 through June 1, 2019 to capture work before, during, and after AlexNet’s 2012 influence on neural-network research.
  • Search scope: The authors queried IEEE Xplore, the ACM Digital Library, Springer Link, and DBLP using empirically evaluated search strings, selecting ("Deep" OR "Learning" OR "Neural") as the most proficient.
  • Search scope: Advanced database searches applied year and venue filters, while Google Scholar, snowballing, and manual additions supplemented the primary search.
  • Study selection: Authors applied inclusion and exclusion criteria through divided classification, independent review, and manual alignment analysis of how studies used DL for SE tasks.

3.3 Comparison Between Probability Sampling of Papers and Non-Probability (Expert) Sampling Based on Exclusion Criteria

The review compares expert-criteria selection with stratified probability sampling to assess whether its 128-paper corpus represents the eligible literature. The authors report statistical representativeness under the specified time-period and venue conditions.

  • Sampling comparison: Stratified sampling estimated required conference-level samples across 18 venues using 90% and 99% confidence intervals and a 7% margin of error.
  • Sampling comparison: The relative difference compares expert-criteria and stratified sample sizes using Δ_ci = (r−s)/f(r−s), with f(r,s) = max(r,s).
  • Results: 125 papers were required at 90% confidence and 266 at 99% for the 1,217-paper eligible population, while expert filtering retained 128 papers.
  • Results: The authors conclude that the 128-paper target set is statistically representative at 90% confidence under the stated period and conference-quality conditions.

3.4 Data Extraction, Synthesis, and Taxonomy Derivation

The review extracted and organized evidence from 128 primary studies using a grounded taxonomy-building process. Descriptive statistics and exploratory analysis then synthesized trends and relationships into a holistic taxonomy for DL4SE research.

  • Data Extraction: 128 primary studies were manually analyzed and represented in an extraction form aligned with the review’s research questions.A separate author confirmed the extracted results to ensure important details were captured and papers were properly represented.
  • Taxonomy Derivation: The taxonomy was derived through open coding based on constructivist grounded theory, with the implementation adapted to the review’s research problem and questions.The process began by establishing the research problem and questions before coding the study data.
  • Synthesis: Descriptive statistics enumerated taxonomy trends, while exploratory data analysis examined statistical relationships among taxonomy features.Together, these analyses supported answers to the review’s research questions.
  • Synthesis: The resulting taxonomy helps researchers identify SE tasks suited to particular DL approaches and potential applications for modeling complex software artifacts.It also provides a basis for exploring future DL4SE applications.

3.5 Exploratory Data Analysis

The exploratory analysis converted extracted study attributes into normalized features and mined their relationships using KDD-inspired correlation and association methods. The resulting analyses supported interpretations of how DL techniques and SE tasks co-occur across the surveyed literature.

  • Data Mining: The KDD-inspired data-mining stage used correlation discovery and association rule learning to uncover relationships among normalized features.The process proceeded from selection and preprocessing to data mining and interpretation/evaluation.
  • Data Preparation: 55 explicit features were extracted from the primary studies and organized for subsequent analysis.These features represented paper attributes collected during the review’s data-extraction process.
  • Data Preparation: Taxonomy categories were normalized into more detailed nominal features, including Tokenization, Neural Embedding, I/O Vectors, and Hyper-Tuning.The same preprocessing approach was applied to categories such as SE Task Addressed and Evaluation Metrics Used.
  • Correlation Discovery: Mutual information measured feature dependencies because the nominal data did not support classic Pearson correlations.Feature-level correlation analysis examined values above 1.0 B, while association rules operated at the class or category level.
  • Association Rule Learning: Association rules were retained only when both support and confidence exceeded 0.8.The rules were generated from frequent itemsets using FP-Growth and organized into an interconnected network to reduce spurious or non-logical interpretations.

4 RQ1: WHAT TYPES OF SE TASKS HAVE BEEN ADDRESSED BY DL-BASED APPROACHES?

Across 128 papers, DL was applied to 23 diverse SE tasks, with program synthesis, code comprehension, and source code retrieval and traceability among the most prevalent. Task choice was strongly associated with other learning components, while several SE areas remained underexplored because of data, problem-definition, or architecture constraints.

  • Task coverage: 128 papers covered 23 SE tasks, with source code generation, code comprehension, source code retrieval and traceability, bug fixing, and feature location appearing first.These early applications primarily used source code as their data.
  • Publication trends: 2017 marked extensive DL use in SE, with papers more than doubling from 9 to 25 over the previous year.The task set also became more diverse, adding code smell detection, software security, and software categorization.
  • Prevalent tasks: Program synthesis was most popular, comprising 22 of the 128 papers, supported by accessible input-output examples and a clear mapping to target programs.These properties provide comparatively accessible, high-quality training data and predictable targets.
  • Task diversity: 70% of the task distribution consisted of distinct topics occurring at approximately 3–5% each, indicating broad dispersion across SE applications.The analysis therefore characterized DL-for-SE research as spanning a diverse set of tasks rather than concentrating exclusively on a few topics.
  • Learning components: SE task was the most informative extracted feature at approximately 4.04B and correlated strongly with data (1.51B), loss function (1.14B), and architecture (1.11B).This suggests that particular learning-component combinations may suit particular SE tasks.
  • Opportunities and barriers: Software performance, program analysis, cloud computing, human aspects of SE, parallel programming, feature location, and defect prediction remained unexplored or underrepresented.Suggested barriers include insufficient clean data, poorly defined problems, and architectures that do not fit available data; transformers such as BERT were proposed as a future direction for clone detection and program repair.

5 RQ2: HOW ARE SOFTWARE ARTIFACTS BEING EXTRACTED, PREPARED, AND USED IN DL-BASED APPROACHES FOR SE TASKS?

DL-based SE approaches use diverse artifacts, with data selection and preprocessing closely tied to the task and model. The review identifies substantial reporting gaps and opportunities to broaden data types, combine representations, and formalize preprocessing practices.

  • 5.1 RQ2𝐴: Data Types: Data selection depended strongly on the SE task: source-code tasks commonly used code, whereas program synthesis primarily used input-output examples.The review also reports that requirements, dependencies, and licenses had not yet been analyzed with DL-based approaches.
  • 5.1 RQ2𝐴: Data Types: Unlabeled repository data can support unsupervised approaches but requires target labels before supervised learning, sometimes through manual labeling.Examples include buggy-file localization, specification mining, code-clone identification, method and class naming, API-sequence learning, and source-code summarization.
  • 5.1 RQ2𝐴: Data Types: Source code comprised ≈59.78% of analyzed data uses, while I/O examples, natural language, repository metadata, and visual data were also prominent.Across the review, 152 data uses were identified, including 86 attributed to source code.
  • 5.2 RQ2𝑏: Preprocessing: Preprocessing was tightly coupled to data type, SE task, publication venue, and DL architecture, and the review organized these relationships into a taxonomy.Transformations such as abstraction can address open-vocabulary problems but may remove complex hierarchical features.
  • 5.2 RQ2𝑏: Preprocessing: Tokenization (≈51%) and neural embeddings (≈35%) were the most prevalent preprocessing techniques, while neural embeddings alone represented 23.85% of the reported distribution in one analysis.The review suggests standardizing common pipelines and exploring less-used representations such as directed graphs, lookup tables, and execution-trace vectors.
  • 5.3 Exploratory Analysis: ≈52% of papers reported exploratory dataset analysis, but approximately 2% addressed sampling bias and data snooping; control of these issues was indeterminate in 90% and 82% of studies, respectively.Reported mitigation strategies included limiting source-data filtering, evaluating projects excluded from training, balancing classes, and removing overlapping data.

6 RQ3: WHAT DEEP LEARNING MODELS ARE USED TO SUPPORT SE TASKS?

DL4SE studies use several architecture families and learning components, with architecture choices closely associated with SE task and data type. The review also identifies reporting gaps and open questions around interpretability, loss functions, and training choices.

  • Architectures: RNNs (≈45%), encoder-decoder models (≈22%), and CNNs (≈21%) are the most prevalent DL architectures in surveyed DL4SE studies.Seven major architecture types were identified overall, including FNNs, autoencoders, siamese networks, and custom architectures.
  • Architecture–task relationships: RNNs and encoder-decoder models dominate source-code generation, analysis, synthesis, traceability, and repair, while CNNs commonly support image- and media-based SE tasks.The review links these patterns to architectures’ suitability for sequential, spatial, or structural features.
  • Architecture–task relationships: SE task features and DL architecture have mutual information of 1.11𝐵, while RNNs appear in ≈45% of papers, partly reflecting the prevalence of text-based SE data.The review reports that task type strongly influences the architecture adopted.
  • Interpretability: Current DL4SE research cannot reliably validate or quantify extracted implicit features, limiting model transparency and motivating work on interpretability.The review connects interpretability to understanding predictions and developing usable tools around them.
  • Learning and loss functions: ≈76% of primary studies use gradient descent, whereas cross entropy is the most common loss function at ≈20%.Other reported learning approaches include gradient ascent at ≈2% and policy-based learning at ≈2%; negative log likelihood is another common loss paired with gradient descent.
  • Learning and loss functions: Loss function and overfitting mitigation have mutual dependence of 1.00𝐵, but ≈33% of papers omit the loss function and the optimal architecture–data pairing remains open.The review also reports mutual dependence of 1.14𝐵 between loss function and SE task.

7 RQ4: HOW WELL DO DL TASKS PERFORM IN SUPPORTING VARIOUS SE TASKS?

DL4SE evaluations use varied metrics and often pursue automation, but reusable baselines, benchmarks, and consistent analyses remain limited. The review therefore emphasizes standardized evaluation and scrutiny of model complexity and generalizability.

  • Baselines and benchmarks: A majority of DL4SE evaluations use self-generated baselines, while many implementations are not publicly available or reusable.The review identifies a general lack of well-documented, reusable baselines and benchmarks.
  • Reported impact: ≈43% of papers target increased automation efficiency, ≈24% advance or introduce a novel architecture, and ≈14% seek improved performance over a prior technique.These categories summarize the primary objectives assigned to DL implementations.
  • Model complexity: ≈16% of DL4SE studies do not consider Occam’s Razor, while ≈23% adjust model complexity when applying the principle.The review cites a case where a simpler optimizer for an SVM outperformed a DL model on the same task.
  • Evaluation consistency: The review reports a confidence level of ≈0.95 for an association between omitting ROC or AUC evaluations and studies using overfitting countermeasures, excluding data augmentation.It also reports mutual information of 0.71𝐵 between SE task and claimed automation impact.
  • Evaluation consistency: Evaluation analysis is inconsistent even within individual SE applications, and testing sets may underrepresent real-world data classes.The review calls for clearer comparison infrastructure and better assessment of test-set representativeness.
  • Evaluation metrics: Accuracy (≈46%), precision (≈35%), recall (≈33%), and F1-measure (≈26%) are the most prominent evaluation metrics.Metric choice often corresponds to the type of learning used in the evaluated approach.

8 RQ5: WHAT COMMON FACTORS CONTRIBUTE TO THE DIFFICULTY WHEN REPRODUCING OR REPLICATING DL4SE STUDIES?

DL4SE studies are difficult to reproduce or replicate because crucial implementation, data-processing, and computational-learning details are often omitted. Open implementations and datasets, along with complete reporting, are presented as central remedies.

  • ≈14% of primary studies provided enough online-repository information to reasonably replicate the approach.Most studies lacked either the implementation or the dataset needed to train and test the approach.
  • Only 11 of the 63 reproducible studies were also replicable, showing limited overlap between textual reproducibility and artifact-based replication.
  • Open-source implementations and datasets can support replication, future work, and more verifiable comparisons of DL-based SE approaches.The review notes that few publicly available SE datasets are tailored for DL techniques.
  • Only 19 studies were conceivably replicable, while 63 were reasonably reproducible from their descriptions.The review distinguishes replication through available artifacts from reproduction based on publication details.
  • Missing hyperparameter details affected 45 papers, while missing data-filtering details affected 55 papers.Other omissions included learning-algorithm information, data-extraction details, and data-formatting information.

9 THREATS TO VALIDITY

The review identifies threats to validity from its search and sampling choices, venue selection, taxonomy construction, and data-synthesis procedures. The authors describe mitigation steps but retain boundaries around coverage and interpretation.

  • The search string and filtering process may have missed relevant DL4SE studies, creating a potential external-validity threat.The authors tested multiple search strings and selected the widest-net formulation to limit selection bias.
  • Predefined inclusion and exclusion criteria, snowballing, probability sampling, and a systematic extraction process were used to mitigate selection and construct-validity threats.The authors report that probability sampling supported the representativeness of their taxonomy.
  • The review focused on top SE, PL, and AI venues with at least a C CORE ranking, potentially excluding pertinent studies elsewhere.
  • The taxonomy classifications may still reflect the collective views and opinions of the authors despite review by multiple evaluators.The classification process used open-coding-inspired review by at least three authors.
  • Rule association mining provided correlations between DL-solution attributes and SE tasks, but the authors treat these associations as prompts for further investigation.

10 GUIDELINES FOR FUTURE WORK ON DL4SE

The proposed DL4SE guidelines organize future work around a five-step checklist covering essential aspects of applying DL to SE. They emphasize explicit reporting and public artifacts to improve reproducibility, replication, and comparison.

  • The five-step checklist is intended to guide researchers through applying DL in SE while acknowledging that implementations have task-specific nuances.
  • Step 3: Select a Deep Learning Architecture: Step 3 requires selecting an architecture while specifying the learning algorithm, hyperparameters, optimization methods, and training process.Hyperparameter selection should be explicitly stated, including empirical tuning when applicable.
  • Step 3: Select a Deep Learning Architecture: Training reports should include the number of iterations, training time, and techniques used to combat overfitting.
  • Data preparation and training-process errors were ranked as the first- and second-highest effort issues among the related developer-reported pitfalls.Hyperparameter issues ranked third in required effort to address.
  • DL4SE publications should make datasets, models, and implementation scripts available online to facilitate reproduction, replication, and meaningful comparisons.

11 FRINGE RESEARCH NOT INCLUDED IN THIS REVIEW

The review excludes several adjacent research strands because they do not meet its operational definition of deep learning or fall outside its primary DL4SE focus. These include embedding-only studies and the emerging SE4DL area.

  • The review defines deep learning models as automatically extracting complex, hierarchical features through multiple nonlinear transformations across hidden neural-network layers.
  • Word2Vec-based studies were commonly used for preprocessing or embedding sequential software data and were treated as outside the review’s DL implementation scope.
  • SE4DL applies software testing and maintenance concepts to developing software based on DL algorithms, including concolic testing and interoperability work.

12 RELATED STUDIES AND LITERATURE REVIEWS

Earlier reviews examined machine learning or deep learning in software engineering with narrower scopes or less methodological detail. This review broadens the task coverage and analyzes applied deep learning through learning components and additional methodological attributes.

  • Prior literature reviews: Earlier studies reviewed machine learning applications in software engineering with varying scope and detail.The literature includes broader ML reviews and studies analyzing implementation components of deep learning more generally.
  • Prior literature reviews: Task-specific reviews examined deep learning applications in areas such as defect prediction and anomaly detection.These studies provide detailed analyses within individual software engineering tasks rather than across multiple tasks.
  • Prior literature reviews: Related broad reviews analyzed deep learning studies, common software engineering tasks, architectures, and application strengths and weaknesses.Li et al. analyzed 98 studies, while Ferreira et al. also described studied works and surveyed implemented architectures.
  • This review: This review adds a learning-component analysis, exploratory data analysis, and coverage of data, modeling, evaluation, and reproducibility factors.It also examines correlations among attributes of deep learning approaches applied to software engineering.

13 CONCLUSIONS

The paper presents a systematic literature review of deep learning applied to software engineering, following established review guidance and extracting information into a taxonomy. Its intended outcome is practical information and intuitions for future software engineering researchers.

  • Review process: The study reviews primary DL4SE research from top software engineering venues using Kitchenham et al.’s systematic-review guidelines.The process established research questions, developed a search, and classified relevant work using inclusion and exclusion criteria.
  • Review outcomes: Information extracted from relevant papers generated a taxonomy of aspects involved in applying deep learning to software engineering tasks.The taxonomy emerged through the extraction process and the study’s research questions.
  • Review outcomes: The review aims to provide future software engineering researchers with information and intuitions for applying deep learning approaches.

14 APPENDIX - SUPPLEMENTAL INFORMATION

The appendix provides supplemental information through tables covering search terms, inclusion and exclusion criteria, tested search strings, and software-engineering task columns.

  • Search materials: Supplemental tables list software terms used as additional search parameters for deep learning and machine-learning venues.
  • Review protocol: Supplemental tables document the systematic literature review’s inclusion and exclusion criteria.
  • Search materials: Supplemental tables record top search strings tested and the search-string terms considered.
  • Task information: A supplemental table describes columns for software engineering tasks.
Loading 2009.06520v2…