PaperlayerView exact revision on arXiv

Source-linked AI summary

Repairing Deep Neural Networks: Fix Patterns and Challenges

Md Johirul Islam, Rangeet Pan, Giang Nguyen, Hridesh Rajan

arXiv:2005.00972v1cs.SE

TL;DR

DNN bug repair patterns and repair challenges are not yet fully understood, despite the need for software-engineering support. This study examines fixes across five deep-learning libraries and reports distinctive patterns, recurring risks, and developer challenges.

  • Problem

    The characteristics of manual DNN repairs and the challenges they pose for automated repair remain insufficiently understood.

    Source: §1 · p. 1

  • Method

    The authors study bug fixes drawn from Stack Overflow and GitHub for five DNN libraries, manually classifying repairs and contributing a benchmark of 667 bug-repair instances.

    Source: §1 · p. 2, §11 · p. 11

  • Results

    DNN repair patterns differ from traditional fixes, with data dimension and network connectivity common; 29% of sampled Stack Overflow fixes introduced a new bug.

    Source: §1 · p. 2, §6 · p. 8, §11 · p. 11

  • Takeaways & Limitations

    The authors identify shared repair patterns and API compatibility testing as potential focuses for DNN repair efforts.

    Source: §9 · p. 10

    The study relies on the DNN bug dataset prepared by Islam et al. [11].

    Source: §8 · p. 10

Abstract

from arXiv · show

Significant interest in applying Deep Neural Network (DNN) has fueled the need to support engineering of software that uses DNNs. Repairing software that uses DNNs is one such unmistakable SE need where automated tools could be beneficial; however, we do not fully understand challenges to repairing and patterns that are utilized when manually repairing DNNs. What challenges should automated repair tools address? What are the repair patterns whose automation could help developers? Which repair patterns should be assigned a higher priority for building automated bug repair tools? This work presents a comprehensive study of bug fix patterns to address these questions. We have studied 415 repairs from Stack overflow and 555 repairs from Github for five popular deep learning libraries Caffe, Keras, Tensorflow, Theano, and Torch to understand challenges in repairs and bug repair patterns. Our key findings reveal that DNN bug fix patterns are distinctive compared to traditional bug fix patterns; the most common bug fix patterns are fixing data dimension and neural network connectivity; DNN bug fixes have the potential to introduce adversarial vulnerabilities; DNN bug fixes frequently introduce new bugs; and DNN bug localization, reuse of trained model, and coping with frequent releases are major challenges faced by developers when fixing bugs. We also contribute a benchmark of 667 DNN (bug, repair) instances.

1 INTRODUCTION

The introduction frames DNN bug repair as an important software-engineering need, while noting that repair characteristics and challenges remain insufficiently understood. The study examines repairs across five libraries and reports distinctive patterns, recurring risks, and challenges for developers.

1 INTRODUCTION

  • The study analyzes manual repairs for Caffe, Keras, Tensorflow, Theano, and Torch, drawing on Stack Overflow and GitHub bug datasets.The researchers collected repair code and manually classified fixes; their criteria yielded 320 Stack Overflow fixes and 347 GitHub fixes.Source: §1 · pp. 1–2
  • DNN bug-fix patterns differ from traditional patterns, with data dimension and network connectivity among the most common fixes.The authors also report fixes that may introduce adversarial vulnerabilities [10] and that frequently introduce new bugs.Source: §1 · p. 2
  • The authors identify bug localization, reuse of trained models, and coping with frequent releases as major repair challenges.They note that DNN errors and faults can be non-trivially related, complicating bug localization.Source: §1 · p. 2
  • The study contributes a benchmark of 667 DNN bug-repair instances.The benchmark is reported as publicly accessible.Source: §1 · p. 2

2 METHODOLOGY

The methodology builds a repair dataset from Stack Overflow and GitHub, then develops and applies a classification scheme for DNN-specific fixes. The scheme distinguishes repair categories and describes changes to data, model structure, training, and API use.

2.1 Dataset

  • The study used 415 Stack Overflow and 555 GitHub bugs across five deep-learning libraries, identifying 320 Stack Overflow fixes and collecting 347 GitHub bug fixes.Stack Overflow fixes included accepted answers or, when no answer was accepted, answers scoring at least 5. Bugs not yet fixed or from repositories made private or deleted were omitted.Source: §2.1 · p. 2

2.2 Bug Fix Pattern Classification

  • The authors refined prior non-ML categories into 15 DNN-specific repair patterns, finding only 4 of the original 28 categories applicable.The classification scheme was developed through open coding.Source: §2.2 · pp. 2–3

2.2.1 Loss Function.

  • Repair categories include changing the loss function, modifying node connections, adding layers, and adjusting layer dimensions.Connection changes include altering weights, pruning edges, or adding backpropagation; added layers may include dense, convolution, or dropout layers.Source: §2.2.2 · p. 3, §2.2.3 · p. 3, §2.2.4 · p. 3

2.2.5 Data Dimension.

  • Data-dimension fixes align input data with the DNN, while accuracy-metric choices depend on whether the task is classification or regression.The described metric examples include classification accuracy, F1 score, confusion matrix, and logarithmic loss for regression.Source: §2.2.5 · p. 3, §2.2.6 · p. 3

2.2.9 Iterations.

  • Other categories change input types, activation functions, or training iterations, including batch size and epoch adjustments.The scheme also includes adapting code to new library versions, which may break backward compatibility.Source: §2.2.7 · p. 3, §2.2.8 · p. 3, §2.2.9 · p. 3

2.2.11 API Contract.

  • API-contract fixes add adapters between incompatible composed operations, while data-wrangling fixes reshape data for downstream operations without changing its intent.Examples of data wrangling include data shifting and shuffling.Source: §2.2.11 · p. 3, §2.2.12 · p. 3

2.2.13 Monitor.

  • The classification also includes diagnostic code that helps localize bugs without repairing them, and changes to optimization algorithms.Optimization algorithms determine the iterative process used to improve model accuracy.Source: §2.2.13 · p. 3, §2.2.14 · p. 3

2.2.15 Change Neural Architecture.

  • Some fixes redo the DNN model because the initial architecture was unsuitable.This category is classified as changing neural architecture.Source: §2.2.15 · p. 3

2.3 Labeling

Two Ph.D. students with DNN-library expertise labeled the repair patterns using a classification scheme, with disagreements reconciled through moderated discussion.

2.3 Labeling

  • Rater agreement was above 85% throughout the labeling process, and conflicts were reconciled through moderated discussion.The reported Cohen’s Kappa values were 82% for the first 100 labels and 85% for the second 100.Source: §2.3 · p. 3

3 BUG FIX PATTERNS

The study characterizes DNN bug-fix patterns through distributions and source-code diffs, finding both recurring repair types and substantial commonality across Stack Overflow and GitHub. Its subsections detail dimension, connectivity, version, layer, and loss-function changes, along with their practical trade-offs.

3 BUG FIX PATTERNS

  • The study analyzes fix-pattern distributions by categorizing fixes and examining source code and diffs to investigate the challenges underlying those patterns.The section presents the distribution of bug-fix patterns in Stack Overflow and GitHub.Source: §3 · p. 3

3.1 Data Dimension

  • Reordering and reshaping account for 79.7% of Stack Overflow data-dimension fixes and require understanding layer specifications and library requirements.Nine of 59 data-dimension fixes reorder inputs; the paper says reshaping and reordering theoretically have no side effects on model behavior.Source: §3.1 · p. 5

3.2 Layer Dimension

  • In GitHub, layer-dimension fixes make up 15.6% and address 75.9% of crash-related fixes; no tool analyzes both data- and layer-dimension changes to choose an optimum.The choice of changing data or layer dimensions can affect model performance.Source: §3.2 · p. 5

3.3 Version-related Fixes

  • Version-related fixes are the highest category in GitHub at 17.6%, and TensorFlow's invasive, backward-incompatible changes make repairs more complicated.Some fixes update API signatures to match library changes.Source: §3.3 · p. 5

3.4 Network Connection

  • Network-connection fixes occur in 17.8% of Stack Overflow and 14.1% of GitHub repairs, addressing crashes, incorrect functionality, and poor performance.The reported effects are crashes (57.14%), incorrect functionality (16.19%), and bad performance (12.38%).Source: §3.4 · p. 5

3.5 Add Layer

  • About 30% of Stack Overflow add-layer fixes add Dropout, which can make training take approximately 2–3 times longer.Dropout randomly removes connections, and the paper associates this with slower convergence.Source: §3.5 · p. 6

3.6 Loss Function

  • Loss-function changes fix 6.2% of Stack Overflow bugs and 3.7% of GitHub bugs, and are reported to enhance prediction accuracy and robustness against adversarial attacks.The paper also identifies a need for live trainable-parameter analysis to localize some convergence bugs; without such frameworks, developers rely on theoretical knowledge.Source: §3.6 · p. 6

3.7 Commonality of Fix Patterns in Stack Overflow and GitHub

  • The Stack Overflow and GitHub fix-pattern distributions are similar (p-value 0.83), although data-dimension, layer-dimension, and versioning categories differ significantly.The other categories have similar shares of occurrences across the two sources.Source: §3.7 · pp. 6–7

4 FIX PATTERNS ACROSS BUG TYPES

Repair patterns vary by bug type: data bugs center on dimension changes, logic bugs on network connectivity, and data-flow bugs on layer dimensions. The findings suggest different bug types may require distinct technical approaches.

4 FIX PATTERNS ACROSS BUG TYPES

  • Version upgrades can alter API names, signatures, and probabilistic behavior; localizing probabilistic changes is difficult without specialized analysis tools.The authors suggest DNN-specific probabilistic analysis techniques for these versioning bugs.Source: §4 · p. 7
  • Interlibrary specification mismatches require expertise in both libraries, as APIs may make similar assumptions but behave differently.For image preprocessing, the discussion contrasts avoiding APIs from other libraries with using specialized image-processing libraries for better performance.Source: §4 · p. 8
  • 41.77% of Stack Overflow data bugs were fixed by changing data dimensions; GitHub data fixes most often changed data type (38.64%) or data dimension (29.55%).The authors connect these changes to the importance of data dimensions for correct DNN functionality.Source: §4 · p. 8
  • Network connectivity changes were the most common logic-bug fix (∼27.03% in Stack Overflow; ∼33.33% in GitHub), while layer-dimension changes led for data-flow bugs (∼36.36%; ∼38.98%).Source: §4 · p. 8
  • Different bug types may call for distinct technical approaches to repair.Source: §4 · p. 8

5 FIX PATTERNS ACROSS LIBRARIES

The study compares bug-fix pattern distributions across five libraries using pairwise t-tests. It reports similar patterns for some library pairs and suggests that repair approaches may be reused across related libraries.

5 FIX PATTERNS ACROSS LIBRARIES

  • Pairwise t-tests at a 95% significance level compare fix-pattern distributions across the five libraries.Source: §5 · p. 8
  • The Caffe–Theano, Caffe–Torch, and Theano–Torch comparisons had p-values of .19, .30, and 0.8, respectively, so the study did not reject equal distributions for those pairs.Source: §5 · p. 8
  • Caffe, Theano, and Torch show similar bug-fix patterns; Keras and TensorFlow form a strongly related pair (p-value 0.84).The authors suggest similar automated repair approaches may be reused for these groups, with a common intermediate representation for Caffe, Theano, and Torch.Source: §5 · p. 8, Table 4 · p. 8

6 INTRODUCTION OF BUGS THROUGH FIXES

The study finds that bug fixes can introduce new bugs, often of different types from the original bug, with API changes prominent among the causes. These introduced bugs commonly crash, and API-change-related fixes can also lead to bad performance.

6 INTRODUCTION OF BUGS THROUGH FIXES

  • 29% of 100 randomly selected Stack Overflow fixes introduced at least one new bug of a different type from the original bug.Only 6.8% of introduced bugs matched the parent bug's type, 13.8% its root cause, and 24.1% its impact.Source: §6 · p. 8
  • 37.9% of introduced bugs were API bugs, 34.5% were due to API change, and 55.2% ended in a crash.Source: §6 · p. 8
  • API changes, mostly due to API versioning, were a major reason for introduced API bugs, which primarily led to crashes and bad performance.Source: §6 · pp. 8–9

7 CHALLENGES IN FIXING BUGS

The study groups repair challenges separately from bug-fix patterns. It identifies difficulties involving reused models, error localization, and rapidly changing DNN libraries.

7.1 DNN Reuse

  • Training DNN models can require substantial computation and labeled data, while reusing pretrained models can bring backdoor, bias, or developer-intent mismatch issues.One example involved overfitting in a reused ResNet50 model; adding dropout and dense layers was used to reduce it.Source: §7.1 · p. 9

7.2 Untraceable or Semi-Traceable Error

  • 11 of 50 crash posts lacked an error message, and 20 of the remaining 39 had fixes that did not match it; a missing compilation step could surface later at prediction.Source: §7.2 · p. 9

7.3 Fast and Furious Releases

  • Rapid versioning and invasive library changes are among the challenges discussed for DNN software maintenance.Source: §7.3 · p. 10

8 THREATS TO VALIDITY

The authors discuss threats involving dataset choice, classification of fixes, and unequal numbers of Stack Overflow posts across libraries. They describe mitigation steps for the classification and post-count concerns.

External Threat

  • The study uses a benchmark dataset of DNN bugs prepared by Islam et al. [11], which the authors identify as a possible external threat.Source: §8 · p. 10

Internal Threat

  • The authors identify coding-based classification and rater expertise as internal threats, and report using open coding, two trained expert raters, and kappa measurement to mitigate them.They report perfect agreement according to the kappa coefficient.Source: §8 · p. 10

Threat from Unequal Post Counts

  • Although Stack Overflow post counts differed by library, the ANOVA result was F (0.0002) < F-critical (2.50), which the authors say indicates the fix-pattern distribution was not unbalanced.Source: §8 · p. 10

9 DISCUSSION

The discussion outlines DNN-specific repair challenges and candidate research directions. It connects repair risks and recurring patterns to analysis, validation, and monitoring needs.

9 DISCUSSION

  • Data-related repairs require understanding implicit dependencies between data and models, while connectivity repairs may use a common network representation and reachability analysis.The authors also suggest mining layer and loss-function specifications from codebases to support automated fixes.Source: §9 · p. 10
  • The discussion reports that some fixes can reduce model robustness and that layer-dimension fixes may introduce data bugs; API compatibility tests are proposed for API-related repairs.Source: §9 · p. 10
  • The authors identify analysis representations for DNNs and runtime monitoring as directions for repair and trace collection, particularly for safety-critical data-science applications.Source: §9 · p. 10

10 RELATED WORKS

Prior studies examined traditional bug fixes, DNN bug characteristics, and bug detection, but did not address DNN repair patterns and challenges across multiple libraries. This study contrasts its focus and scope with that earlier work.

Study on traditional non-DNN Bugs

  • Traditional bug-fix studies found frequent IF-related and Method Call bugs, but their classification did not capture most DNN repair strategies, motivating an open-coded scheme.The study identifies Method Call and sequence addition or deletion patterns in DNN repairs, but no evidence of other strategies from the traditional scheme.Source: §10 · p. 11

Study on DNN Bugs

  • Zhang et al. studied TensorFlow bug patterns and detection challenges, but not repair patterns; this study covers five libraries and repair challenges.Source: §10 · p. 11
  • Islam et al. examined bug characteristics across five DNN libraries, while this study uses their dataset to investigate repair patterns and challenges [11].Source: §10 · p. 11

Related studies on ML and DNN libraries

  • Sun et al. studied bug-fix patterns in three ML libraries, whereas this study examines DNN models created using DNN libraries and reports no commonalities.Source: §10 · p. 11
  • Zhang et al. classified Stack Overflow bug-related posts and studied response times, while this work examines bug-fixing strategies across five DNN libraries.Source: §10 · p. 11
  • Pham et al. used cross-validation across three libraries to localize and detect bugs; this study uses GitHub commits and Stack Overflow posts to compare repair strategies across five.It also discusses fix patterns, repair challenges, and bugs introduced by fixes.Source: §10 · p. 11

11 CONCLUSION AND FUTURE WORK

The conclusion identifies distinctive DNN repair patterns, risks, and challenges, then outlines future work on repair tools, representations, localization, bug detection, and version upgrades.

11 CONCLUSION AND FUTURE WORK

  • DNN bug-fix patterns differ from traditional repairs; common fixes address data-DNN incompatibility and network structure, while fixes can introduce bugs or adversarial vulnerabilities.The conclusion notes that version upgrades are prevalent and can involve invasive, backward-incompatible changes and probabilistic behavior changes.Source: §11 · p. 11
  • Repair tools should represent DNN structure and code together, supporting fixes such as connecting disconnected layers or adding missing layers.Source: §11 · p. 11
  • The study identifies difficult bug localization, limited insight into model behavior that hinders model reuse, and difficulty keeping pace with rapid releases.Source: §11 · p. 11
  • Future work includes DNN-aware localization tools and detecting dimension-mismatch bugs and fixes that may introduce vulnerabilities.Source: §11 · p. 11
  • Future upgrade tools should encode version-change semantics and track changes to DNN-library signatures and semantics.Source: §11 · p. 11
Loading 2005.00972v1…