PaperlayerView exact revision on arXiv

Source-linked AI summary

Software Engineering Meets Deep Learning: A Mapping Study

Fabio Ferreira, Luciana Lourdes Silva, Marco Tulio Valente

arXiv:1909.11436v3cs.SEcs.LG

TL;DR

Recent research combining Deep Learning and Software Engineering lacks a consolidated map. This study maps 81 papers across the intersection and examines their problems, techniques, and comparisons with traditional machine learning. The literature is growing, with documentation, defect prediction, and testing as the top three research problems, while DL also carries substantial computational and data requirements.

  • Problem

    Recent cross-pollination between DL and SE lacks a clear map of the research conducted at their intersection.

    Source: §1 · p. 1

  • Method

    The study reviews 81 papers and analyzes their SE problems, DL techniques, traditional-ML comparisons, strengths, and drawbacks.

    Source: §1 · p. 1

  • Results

    The literature is gaining momentum, with documentation, defect prediction, and testing as the top three problems and CNNs and RNNs the most common network types.

    Source: §6 · p. 7

  • Takeaways & Limitations

    The mapping provides researchers and practitioners a clearer picture of problems, solutions, and opportunities in DL and SE.

    Source: §1 · p. 1

    DL-based techniques require extensive training data and high computational cost, with training reported to take hours to weeks even on advanced hardware.

    Source: §5.2 · p. 7

Abstract

from arXiv · show

Deep Learning (DL) is being used nowadays in many traditional Software Engineering (SE) problems and tasks. However, since the renaissance of DL techniques is still very recent, we lack works that summarize and condense the most recent and relevant research conducted at the intersection of DL and SE. Therefore, in this paper, we describe the first results of a mapping study covering 81 papers about DL & SE. Our results confirm that DL is gaining momentum among SE researchers over the years and that the top-3 research problems tackled by the analyzed papers are documentation, defect prediction, and testing.

1 INTRODUCTION

The study addresses the lack of a clear map of recent research combining Deep Learning and Software Engineering. It reviews 81 papers to characterize research problems, techniques, traditional-ML comparisons, and DL's strengths and drawbacks.

  • It identifies documentation, defect prediction, and testing among the main SE problems addressed by DL research.Source: §1 · p. 1
  • The mapping study analyzes 81 recent papers on applying DL to SE problems and vice versa.Source: §1 · p. 1
  • The study reports the growth of DL-and-SE publications and the most common DL techniques used by SE researchers.Source: §1 · p. 1
  • It highlights cases where traditional machine-learning techniques achieved the same results and discusses DL's strengths and drawbacks.Source: §1 · p. 1

2 DEEP LEARNING IN A NUTSHELL

DL uses multiple neural-network layers to learn high-level representations, with architectures suited to different data and prediction tasks. The section outlines MLP, CNN, RNN, and hybrid architectures relevant to SE.

  • DL uses multiple neural-network layers to model high-level representations, unlike traditional ML's handcrafted features.Source: §2 · p. 1
  • MLPs support classification and regression across image, text, and time-series data, with layered inputs, abstractions, and predictions.Source: §2 · p. 1
  • CNNs extract features through convolution and pooling layers and can process images, text, and sequences for classification or regression.Source: §2 · p. 1
  • RNNs use historical inputs as memory for sequence prediction, while LSTMs learn long-term dependencies.Source: §2 · p. 1
  • Hybrid neural-network architectures combine two or more neural-network types, commonly using CNNs and RNNs together.Source: §2 · p. 2

3 STUDY DESIGN

The mapping study defines research questions about SE problems, DL techniques, and comparisons with traditional ML, then searches, filters, and evaluates relevant literature. Its search identified 141 papers before selection produced a final dataset of 81 studies.

3.1 Research Questions

  • The study asks which SE problems DL addresses, which DL techniques are used, and whether DL improves results over traditional ML.Source: §3.1 · p. 2

3.2 Search process

  • Researchers searched six digital libraries for “deep learn*” in titles, keywords, and abstracts, restricting results to SE venues indexed by CSIndexbr.Source: §3.2 · p. 2
  • The search found 141 distinct papers published from August 2019 to December 2019.Source: §3.2 · p. 2

3.3 Study selection

  • After excluding papers below the page threshold and studies unrelated to the goals, the final dataset contained 81 papers.Source: §3.3 · p. 2

4 RESULTS

The selected literature is concentrated in recent years and major SE venues, indicating growing interest in applying DL to SE. Most studies appeared at conferences, especially ICSE and FSE, while journals contributed a smaller share.

4.1 Overview

  • Only one selected paper was published in 2015, while more papers appeared in 2019 than in 2018 despite covering only nine months.Source: §4.1 · p. 3
  • 75.3% of the 81 studies came from conferences and 24.7% from journals, with ICSE and FSE together accounting for 29.6%.Source: §4.1 · p. 3
  • Twelve papers, or 14.8%, had at least one author affiliated with industry.Source: §4.1 · p. 3

(RQ1) What SE problems are solved by DL?

The mapping study groups DL-and-SE research into applying SE techniques to DL-based software, applying DL to SE problems, and position papers or tutorials. Among DL applications to SE, documentation, testing, and defect prediction dominate, alongside a broad range of additional problems.

(RQ1) What SE problems are solved by DL?

4.1.1 Using Software Engineering Techniques in DL-based Software.

  • Ten papers (12.3%) apply SE techniques to DL-based software, mainly addressing testing, metrics, deployment, and bugs in DL systems.Source: §4.1.1 · p. 3

4.1.2 Using DL Techniques in SE Problems.

  • Documentation has the largest concentration, with 13 papers (16%), including Stack Overflow linking, code-comment generation, developer-tip extraction, and source-code identification in videos.Source: §4.1.2 · pp. 3–4
  • Testing includes seven papers (8.6%) on fuzzing, fault localization, mutation testing, and mobile-app testing, with LSTM-based approaches automating test generation and protocol analysis.Source: §4.1.2 · pp. 4–5
  • Defect prediction includes seven papers (8.6%), using DBN, LSTM, change-sequence, commit-message, and deep-forest models to learn semantic or structural features.Source: §4.1.2 · p. 5
  • Other studied problems include code search, security, language modeling, bug localization, clone detection, code smells, mobile development, program repair, sentiment analysis, and type inference.Source: §4.1.2 · p. 5
  • Thirteen additional problems receive one paper each, including anomaly detection, API migration, bug-report summarization, decompilation, effort estimation, and traceability.Source: §4.1.2 · p. 5
  • Three position or tutorial papers address automotive software challenges and opportunities or provide introductory material on machine learning and DL.Source: §4.1.3 · p. 5

(RQ2) What DL techniques are used in SE problems?

The study identifies the DL techniques most often used to support SE problems and examines how their use is distributed across research problems.

(RQ3) How does DL compare with other machine learning techniques used in SE problems?

The comparison shows that DL is not consistently superior to traditional machine learning for SE problems. In several cases, optimized or conventional approaches match or outperform DL while using fewer resources or less time.

5 DISCUSSION

The discussion identifies strengths that make DL useful for SE problems, especially with unstructured or sequential data, while also emphasizing substantial costs and reproducibility constraints. These drawbacks include computational demands, large dataset requirements, replication difficulty, and overfitting risks.

5.1 Strengths

  • DL can handle unstructured data, including programming screencasts, to extract developer actions and relevant code frames.The reviewed studies demonstrate practical use with visual and video data.Source: §5.1 · p. 6
  • DL reduces manual Feature Engineering by automatically extracting and combining correlated features across multiple layers.CNNs can extract image features from screen changes caused by diverse developer actions, environments, and languages.Source: §5.1 · p. 6
  • DL supports learning without labeled data, with LSTM detecting likely vulnerability locations while learning syntactic and semantic code features.This is presented as an advantage over approaches requiring extensive manual labeling.Source: §5.1 · p. 6
  • RNNs are suited to sequence prediction because their internal memory incorporates historical inputs when predicting subsequent values.This supports tasks involving sequential data such as text and speech.Source: §5.1 · p. 6
  • Once trained correctly, DL can perform thousands of repetitive tasks with better performance than traditional ML techniques.The claimed benefit depends on correct training.Source: §5.1 · p. 6

5.2 Drawbacks

  • Training DL models can require expensive high-performance hardware and take from hours to weeks, with high computational cost reported in at least 22 papers.The training process is described as the most challenging and time-consuming part of using DL.Source: §5.2 · p. 7
  • DL effectiveness depends on extensive, high-quality training data that many researchers cannot access, limiting approaches such as DeepSims.The cited study describes building a large and representative dataset as challenging.Source: §5.2 · p. 7
  • DL models may require retraining for similar problems, while reproducing results is difficult because implementations and data are not commonly shared.Small differences in implementations or data may produce large differences in results.Source: §5.2 · p. 7

6 CONCLUSION

This mapping study analyzed 81 recent papers applying DL to SE problems or applying SE to DL-based systems. It finds increasing adoption, with documentation, defect prediction, and testing as the top research problems, and CNNs and RNNs as the most common neural network types.

6 CONCLUSION

  • 35 of 81 papers (43.2%) were published in 2019, compared with only one paper in 2015, indicating growing momentum for DL among SE researchers.The study’s analyzed papers cover applications of DL to SE problems or SE to DL-based systems.Source: §6 · p. 7
  • Documentation, defect prediction, and testing were the top three research problems, with 13, 7, and 7 papers respectively.These counts summarize the most frequently tackled problems in the analyzed literature.Source: §6 · p. 7
  • CNN and RNN were the most common neural network types used in the analyzed papers.The conclusion reports this pattern across the mapped studies.Source: §6 · p. 7
Loading 1909.11436v3…