PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Study on Deep Learning Bug Characteristics

Md Johirul Islam, Giang Nguyen, Rangeet Pan, Hridesh Rajan

arXiv:1906.01388v1cs.SEcs.LG

TL;DR

Deep learning software increasingly relies on libraries, but the characteristics and causes of bugs in their usage are not well characterized. This study analyzes Stack Overflow posts and Github bug-fix commits across five libraries, finding that data and logic bugs dominate and that buggy patterns are strongly correlated.

  • Problem

    The study asks what types, causes, impacts, vulnerable pipeline stages, and common antipatterns characterize bugs in software using deep learning libraries.

  • Method

    The authors study 2716 qualified Stack Overflow posts and 500 Github bug-fix commits across Caffe, Keras, Tensorflow, Theano, and Torch.

  • Results

    Data Bugs and Logic Bugs are the most severe bug types, while Incorrect Model Parameter and Structural Inefficiency are major root causes; bugs are strongly correlated across libraries.

  • Takeaways & Limitations

    The findings support attention to data verification tools and recurring antipatterns when developing and debugging software using deep learning libraries.

  • Takeaways & Limitations

    The study’s validity is constrained by possible classification and labeling bias and by the trustworthiness of its collected dataset.

Abstract

from arXiv · show

Deep learning has gained substantial popularity in recent years. Developers mainly rely on libraries and tools to add deep learning capabilities to their software. What kinds of bugs are frequently found in such software? What are the root causes of such bugs? What impacts do such bugs have? Which stages of deep learning pipeline are more bug prone? Are there any antipatterns? Understanding such characteristics of bugs in deep learning software has the potential to foster the development of better deep learning platforms, debugging mechanisms, development practices, and encourage the development of analysis and verification frameworks. Therefore, we study 2716 high-quality posts from Stack Overflow and 500 bug fix commits from Github about five popular deep learning libraries Caffe, Keras, Tensorflow, Theano, and Torch to understand the types of bugs, root causes of bugs, impacts of bugs, bug-prone stage of deep learning pipeline as well as whether there are some common antipatterns found in this buggy software. The key findings of our study include: data bug and logic bug are the most severe bug types in deep learning software appearing more than 48% of the times, major root causes of these bugs are Incorrect Model Parameter (IPS) and Structural Inefficiency (SI) showing up more than 43% of the times. We have also found that the bugs in the usage of deep learning libraries have some common antipatterns that lead to a strong correlation of bug types among the libraries.

1 INTRODUCTION

This study examines bugs in software using deep learning libraries, comparing five popular libraries across multiple bug characteristics and data sources. It finds that data and logic bugs predominate, with structural inefficiency and incorrect model parameters among the primary root causes.

  • Study scope: The study investigates bugs in software using Caffe, Keras, Tensorflow, Theano, and Torch.These libraries have different design goals, enabling comparison of how those goals relate to bugs in their usage.
  • Research questions: The research questions cover bug frequency, root causes, impacts, vulnerable pipeline stages, common patterns, and bug evolution.The study compares findings across the five subject libraries.
  • Key findings: Data Bugs and Logic Bugs are the most common severe bug types, while Structural Inefficiency and Incorrect Model Parameter are primary root causes.The study also examines bug impacts, pipeline stages, and recurring antipatterns.
  • Key findings: Bugs in deep learning library usage show strong correlation in both their distributions and antipatterns across libraries.The study presents this correlation as evidence of commonality in buggy code patterns.

2 METHODOLOGY

The authors combine manually analyzed Stack Overflow posts and Github commits, then classify bugs by type, cause, effect, and pipeline stage. They use established taxonomies, open coding, pilot studies, independent labeling, and reconciliation to study bugs across five libraries.

  • Data sources: The dataset combines Stack Overflow posts with Github bug fix commits.The two sources capture developer-encountered bugs and bugs found and fixed in open-source software, respectively.
  • Stack Overflow collection: Candidate Stack Overflow posts were selected using tags for Caffe, Keras, Tensorflow, Theano, and Torch.The selected counts were 183, 567, 1558, 231, and 177 posts, respectively.
  • Stack Overflow collection: Researchers manually screened candidate posts and retained those whose accepted answers fixed deep learning API usage.The resulting bug counts were 35, 162, 166, 27, and 25 for Caffe, Keras, Tensorflow, Theano, and Torch, respectively.
  • Classification scheme: Bugs were classified by type, root cause, effect, and deep learning pipeline stage using an existing taxonomy supplemented through open coding.The pipeline stages were data collection, data preparation, choice of model, training, evaluation, hyperparameter tuning, and prediction.
  • Labeling procedure: Two authors independently labeled posts, monitored agreement with Cohen’s Kappa, and reconciled disagreements under supervision.The process included training and reconciliation at labeling intervals.

3 FREQUENT BUG TYPES

Data bugs are the most frequent bug type across the studied libraries, while structural logic and API bugs also show library-specific prevalence and practical consequences. Stack Overflow and GitHub distributions generally exhibit similar bug-type patterns.

  • 3.1 Data Bugs: Data Bugs appear 26% of the time overall and are the most frequent bug type across the studied libraries.They account for 30% of TensorFlow, 24% of Keras, 36% of Torch, 35% of Theano, and 9% of Caffe Stack Overflow posts.
  • 3.1 Data Bugs: Data bugs mostly arise from missing preprocessing activities such as feature engineering, validation, and shuffling.The study suggests data verification tools using abstractions such as DataFrame and model properties could help address these difficulties.
  • 3.2 Structural Logic Bugs: 43% of Caffe bugs are Structural Logic Bugs, indicating problems in model construction and logical organization.Other libraries also show Structural Logic Bug proportions ranging from 0% to 27%.
  • 3.3 API Bugs: Torch, Keras, and TensorFlow have 16%, 11%, and 11% API bugs, respectively.More than 81% of API bugs come from Keras and TensorFlow, where dependency and version changes can break software using backend libraries.
  • 3.3 API Bugs: Keras API version changes can cause training failures when keyword names change, such as replacing epochs with nb_epoch.The cited example reports that epochs is unavailable in Keras version 2+ and produces an unknown-keyword error.
  • Cross-source comparison: All bug types except Non Model Structural Logic Bug show similar distributions in GitHub and Stack Overflow at the 5% significance level.The t-test analysis fails to reject equal distributions for the other bug types.

4 ROOT CAUSE

Incorrect Model Parameter (IPS) and Structural Inefficiency (SI) are prominent root causes, with their effects differing across libraries and model-related bug types. Other recurring causes include unaligned tensors, absent type checking, API changes, and API misuse.

  • 4.1 Incorrect Model Parameter (IPS): 24%: IPS is the average share of bugs across the studied libraries, making it the most malicious root cause.IPS causes runtime crashes in which execution does not succeed.
  • 4.2 Structural Inefficiency (SI): 25% in Keras and 37% in Caffe: SI accounts for substantial shares of bugs in these libraries.SI bugs typically produce suboptimal model performance rather than program crashes.
  • 4.3 Unaligned Tensor (UT): 28%: Torch bugs are attributed to UT, the highest UT share among the five libraries.Tensor dimensions are important for successful model construction; corresponding shares are 16% for Tensorflow, 12% for Keras, 7% for Theano, and 3% for Caffe.
  • 4.4 Absence of Type checking: 30%: Theano bugs are attributed to the absence of type checking, compared with 8% in Keras and 15% in Tensorflow.The study relates this issue to the dynamic nature of Python.
  • 4.5 API Change and API Misuse: 9% in Tensorflow and 7% in Keras: bugs are attributed to API change, while API misuse has a different distribution between Github and Stack Overflow.The study reports that all other root causes have similar patterns across the two data sources.

5 IMPACTS FROM BUGS

Crashes are the dominant and most severe effect of bugs, while other bugs cause bad performance or incorrect functionality. These effects show similar distributions in Stack Overflow and Github.

  • 5.1 Crash: More than 66%: bugs cause program crashes on average across the studied libraries.Crash is the top impact in every library, ranging from 40% to 77%.
  • 5.2 Bad Performance: 31%, 16%, 8%, 11%, and 8%: Caffe, Keras, Tensorflow, Theano, and Torch bugs cause bad performance, respectively.A model may train successfully yet achieve poor accuracy during evaluation or prediction.
  • 5.3 Incorrect Functionality: 12%: bugs cause Incorrect Functionality on average across the libraries.The study describes this effect as unexpected software behavior relative to the model’s logical organization or the developer’s prior experience.
  • 5.4 Effects in Github: The study confirms that bug effects in Stack Overflow and Github are similar for all libraries.This conclusion is based on P-value tests of the effects distributions.
  • 5.4 Effects in Github: 79%: the P value indicates that Bad Performance has similar distributions in Stack Overflow and Github.Crash has a 50% P value, and none of the impacts rejects the null hypothesis at the 95% significance level.

6 DIFFICULT DEEP LEARNING STAGES

Bugs cluster most heavily in data preparation, followed by training and choice of model, with model-stage defects commonly rooted in parameter, structural, and utility problems.

  • Data preparation: 32% of bugs occur during data preparation, making it the most bug-prone deep learning pipeline stage.The study identifies data preparation as the stage where most deep learning programming bugs happen.
  • Training: 27% of bugs occur during the training stage.Many training-stage bugs are related to Incorrect Model Parameter and Structural Inefficiency root causes.
  • Choice of model: 23% of bugs occur during the choice-of-model stage.This stage constructs the model and selects the learning algorithm.
  • Choice of model: Choice-of-model bugs are primarily caused by Incorrect Model Parameter, Structural Inefficiency, and Usage Trap.These causes arise while constructing the model and choosing the appropriate algorithm.

7 COMMONALITY OF BUG

Bug-type distributions are strongly correlated across most studied libraries, while Torch differs; antipattern distributions similarly distinguish Torch from TensorFlow and Caffe.

  • Bug-type correlation: Libraries show strong correlation coefficients close to 1 in their bug-type distributions, except Torch, which correlates weakly with the others.The analysis tested whether similar library tasks correspond to similar bug distributions.
  • Antipatterns: TensorFlow and Caffe have similar antipattern distributions, whereas Torch has a different antipattern distribution.The antipattern analysis investigated the strong TensorFlow–Caffe correlation and weak Torch–Caffe correlation.
  • Antipatterns: The identified antipatterns are Continuous Obsolescence, Cut-and-Paste Programming, Dead Code, Golden Hammer, Input Kludge, Mushroom Management, and Spaghetti Code.These antipattern categories were identified through deeper analysis of buggy Stack Overflow code.

8 EVOLUTION OF BUGS

Over time, structural logic bugs increased in Keras, Caffe, and TensorFlow, while data bugs generally declined except in Torch.

  • Structural logic bugs: Structural logic bugs show an increasing trend in Keras, Caffe, and TensorFlow.The study reports this trend as a cross-library temporal finding.
  • Structural logic bugs: Caffe structural logic bugs rose from 30% in 2015 to 100% in 2018.The intermediate yearly values were 32% in 2016 and 67% in 2017.
  • Data bugs: Data bugs slowly decreased since 2015 except in Torch.Torch data bugs remained close to 50% during 2016–2018, while other libraries declined toward near zero.
  • Data bugs: Keras data bugs declined from 27% to 15%, and TensorFlow data bugs declined from 30% to 10% between 2015 and 2018.The paper suggests specialized data libraries and tensor type-and-shape information as possible reasons for the decline.

9 THREATS TO VALIDITY

The study identifies classification and dataset trustworthiness as threats to validity, and describes independent labeling, reconciliation, and filtering procedures to mitigate them.

  • Internal threat: Bug classification and labeling may bias the findings despite using a vetted taxonomy and open coding.Two trained Ph.D. students independently labeled posts, disagreements were reconciled with expert monitoring, and pilot studies monitored agreement.
  • External threat: Dataset trustworthiness is an external threat because Stack Overflow post quality and user reputation may vary.The study limited analysis to posts scoring at least 5 and selected highly scored posts from users ranging from novices to experts.

10 DISCUSSION

The discussion identifies data and structural logic bugs as major practical concerns and points toward verification, model-analysis, and recommendation tools as remedies.

  • Data Bugs: Data bugs are frequent and can cause crashes, poor performance, or failures from mismatched formats, encoding, missing data, splits, and shuffling.The discussion emphasizes developers’ limited access to data-verification tools.
  • Antipatterns: Figure 8 presents an example of a similar antipattern in TensorFlow and Caffe.The figure is used to illustrate cross-library commonality in buggy code patterns.
  • Bug Evolution: Figure 9 presents a timeline of the evolution of bugs.The figure frames bug patterns as changing over time.
  • Data–Model Coupling: Model-analysis tools could assess whether a particular deep learning model fits the available data.This addresses the strong coupling between data and model-related problems.
  • Structural Logic Bugs: Structural logic bugs arise from incorrect model organization, hidden layers, or code usage, motivating automated model and parameter recommendations.The authors suggest mining qualified code repositories for recurring patterns and examples.

11 RELATED WORKS

Prior empirical studies examined machine-learning implementation bugs, TensorFlow applications, specialized bug classes, and Stack Overflow discussions, whereas this work studies usage bugs across five deep learning libraries.

  • Empirical Studies: Thung et al. studied bugs in Mahout, Lucene, and OpenNLP, including frequencies, severity, duration, effort, and impact.Their study differs from this work’s focus on bug types, root causes, and impacts in five deep learning libraries.
  • Empirical Studies: Zhang et al. analyzed TensorFlow application bugs from Stack Overflow and GitHub across root cause, bug type, and impact.The present study extends the library scope and adds pipeline-stage and antipattern analyses.
  • Study Scope: This study compares five libraries with different design constraints and examines data preparation, modeling, training, evaluation, tuning, prediction, and common antipatterns.The libraries are TensorFlow, Keras, Torch, Caffe, and Theano.
  • Related Bug Studies: Other empirical work has focused on concurrency bugs, recovery bugs, and API-change problems rather than bugs from deep learning-library usage.The distinction places this study within a broader empirical software-bug literature.
  • Bug Classification: Earlier defect-classification work includes IEEE-based classifications and a Python predictive-analysis tool able to detect 46 bugs.These efforts provide broader classification context rather than the specific empirical focus of this paper.
  • Stack Overflow Studies: Prior Stack Overflow studies analyzed platform impact, Android APIs, and developer discussions but did not study bugs in deep learning software.This work applies the platform to a previously unexamined deep learning setting.

12 CONCLUSION

The paper empirically studies bugs in software using deep learning libraries through Stack Overflow posts and GitHub bug-fix commits, including their types, causes, impacts, pipeline stages, and antipatterns. It finds that data and logic bugs are most severe, incorrect model parameters and structural inefficiency are major causes, and bug types are strongly correlated across libraries.

  • Conclusion: 2716 qualified Stack Overflow posts and 500 GitHub bug-fix commits were analyzed to characterize deep learning-library usage bugs.The study also examined pipeline stages and buggy-code antipatterns.
  • Conclusion: Data bugs and logic bugs are the most severe bug types, appearing more than 50% of the time.This conclusion summarizes the paper’s cross-library bug-type finding.
  • Conclusion: Incorrect Model Parameter (IPS) and Structural Inefficiency (SI) are the major root causes of these bugs.The conclusion identifies these causes without assigning separate frequencies to either one.
  • Conclusion: Bugs in deep learning-library usage are strongly correlated across libraries.The study relates this commonality to recurring antipatterns in buggy code.
Loading 1906.01388v1…