Source-linked AI summary
Toward Understanding Deep Learning Framework Bugs
Junjie Chen, Yihua Liang, Qingchao Shen, Jiajun Jiang, Shuochuan Li
TL;DR
DL framework bugs can propagate to dependent programs and models, but their characteristics and testing coverage remain insufficiently understood. The paper manually analyzes 1,000 bugs across four frameworks, derives findings and guidelines, and evaluates TenFuzz. TenFuzz detects 6 bugs, including 3 previously unknown bugs confirmed by maintainers, in a preliminary TensorFlow study.
Problem
DL framework bugs can cause unexpected behavior in dependent programs and models, while existing work lacks comprehensive understanding of their characteristics.
Method
The authors manually analyze 1,000 bugs from four diverse DL frameworks, relate causes and symptoms to framework levels, assess testing techniques, and develop TenFuzz from the findings.
Results
TenFuzz detects 6 bugs in a preliminary TensorFlow evaluation, including 3 previously unknown bugs confirmed by maintainers.
Takeaways & Limitations
The study provides actionable guidelines for future DL framework bug detection and debugging.
Takeaways & Limitations
Findings 11 and 12 may not represent DL framework testing techniques beyond the three techniques studied.
Abstract
from arXiv · showhide
DL frameworks are the basis of constructing all DL programs and models, and thus their bugs could lead to the unexpected behaviors of any DL program or model relying on them. Such a wide effect demonstrates the necessity and importance of guaranteeing DL frameworks' quality. Understanding the characteristics of DL framework bugs is a fundamental step for this quality assurance task, facilitating designing effective bug detection and debugging approaches. Hence, in this work we conduct the most large-scale study on 1,000 bugs from four popular and diverse DL frameworks (i.e., TensorFlow, PyTorch, MXNet, and DL4J). By analyzing the root causes and symptoms of DL framework bugs associated with 5 components decomposed from DL frameworks, as well as measuring test coverage achieved by three state-of-the-art testing techniques, we obtain 12 major findings for the comprehensive understanding of DL framework bugs and the current status of existing DL framework testing practice, and then provide a series of actionable guidelines for better DL framework bug detection and debugging. Finally, based on the guidelines, we design and implement a prototype DL-framework testing tool, called TenFuzz, which is evaluated to be effective and finds 3 unknown bugs on the latest TensorFlow framework in a preliminary study, indicating the significance of our guidelines.
1 INTRODUCTION
DL framework bugs threaten the quality of programs and models, yet existing research and testing provide limited comprehensive understanding of their characteristics. This study analyzes 1,000 bugs across four diverse frameworks, derives findings and guidelines, and evaluates TenFuzz.
- Motivation: DL framework bugs can affect every program or model relying on framework APIs, making framework quality essential for overall DL-system quality.Prior work has focused more heavily on model and program levels than on bugs inside framework code.
- Research gap: Existing DL framework testing techniques detect some new bugs but often treat frameworks as black boxes without comprehensive bug-characteristic knowledge.This gap may limit detection performance and hinder more effective testing and debugging approaches.
- Study scope: 1,000 real bugs from TensorFlow, PyTorch, MXNet, and DL4J form the study’s large-scale empirical basis.The frameworks differ in computational-graph types, implementation languages, and development organizations, supporting broader conclusions.
- Contributions: The study decomposes frameworks into five levels and identifies 13 root causes and 6 symptoms through systematic manual analysis.It also associates bug characteristics with framework levels and derives 10 major findings.
- Contributions: TenFuzz successfully detected 6 TensorFlow bugs in a preliminary evaluation, including 3 previously unknown bugs confirmed by maintainers.The tool was designed from the study’s findings and actionable testing guidelines.
2 DEEP LEARNING FRAMEWORKS
The paper models DL frameworks as a five-level architecture spanning user-facing APIs, graph processing, operations, utilities, and environment-dependent processing. This decomposition organizes how framework functionality and bugs can be analyzed.
- Architecture: A DL framework is decomposed into User-Level API, Graph-Level Implementation, Operation Implementation, General Utility, and Environment-Dependent Processing.The levels range from directly accessible user APIs to infrastructure-related processing.
- User-Level API: User-Level API provides high-level interfaces for data processing, model construction, training, and utility functions supporting DL workflows.Examples include image resizing, text tokenization, and model checkpointing.
- Graph-Level Implementation: Graph-Level Implementation constructs, transforms, and executes static or dynamic computational graphs.Its functions include graph partitioning, optimization, format conversion, data propagation, and gradient computation.
- Operation Implementation: Operation Implementation contains tensor-processing operators such as convolution, pooling, normalization, mathematical, and array-manipulation operations.Each operation consumes zero or more tensors and produces zero or more tensors.
- Lower levels: General Utility supplies shared data structures and functions, whereas Environment-Dependent Processing connects framework functionality to hardware and operating-system characteristics.Environment-dependent examples include device-specific memory-allocation strategies.
3 METHODOLOGY
The methodology samples and manually labels 1,000 bugs from four diverse DL frameworks using balanced data collection and a structured classification process. The design supports cross-framework analysis while acknowledging sampling and technique-selection limitations.
- Data collection: Researchers collected closed and merged bug-fixing pull requests because they contain accepted fixes and richer implementation and discussion evidence.Related issue reports, code changes, and developer discussions supported bug understanding.
- Data collection: Manual analysis covered bugs from June 2018 through October 2021, using reversed chronological selection for frameworks with more candidate pull requests.PyTorch supplied the smallest pool and was fully analyzed first to establish the 250-bug target.
- Validity: The study may not represent all DL framework bugs because manual inspection was limited and developers may make mistakes when verifying fixes.The authors identify these as data-related threats despite systematic collection and review.
- Data collection: The dataset contains 250 bugs from each of TensorFlow, PyTorch, MXNet, and DL4J, selected to balance framework representation.The frameworks were chosen using popularity measures and remain diverse in languages, organizations, and computational-graph types.
- Manual labeling: Each bug was labeled by root cause, symptom, pipeline-observation stage, and framework level using adapted taxonomies and source-file classification.Two authors independently labeled pull requests, resolving inconsistencies through discussion with a third author.
4.1 RQ1: Root Causes
DL framework bugs arise from both framework-specific and common software causes, with framework-specific causes accounting for 50.2% of bugs and Incorrect Algorithm Implementation most prevalent.
- Root Cause Distribution: 152 Incorrect Algorithm Implementation bugs were identified, including 121 bugs in DL-specific algorithms.The DL-specific subset accounts for 79.61% of these bugs.
- Root Cause Distribution: 50.2% of DL framework bugs stem from four framework-characteristic causes, led by Incorrect Algorithm Implementation.These causes are Incorrect Algorithm Implementation, Type Issue, Tensor Shape Misalignment, and Environment Incompatibility.
- Root Cause Distribution: Type Issue was the second most prevalent cause, accounting for 14.20% of bugs and mainly involving tensor types.Tensor-type problems account for 100 of 142 Type Issue bugs.
- Root Cause Distribution: Misconfiguration was the third most prevalent cause, with bugs distributed across compilation, installation, and compatibility configurations.The corresponding shares were 22.39%, 41.04%, and 36.57%, respectively.
- Root Cause Distribution: API Misuse differed from traditional software: 71.19% of its bugs involved wrong APIs, whereas API Missing/Redundancy predominated in MuBench [15].This distributional difference implies that DL frameworks require different API-misuse detection strategies.
4.2 RQ2: Symptoms
DL framework bugs manifest through six symptoms, with crashes most common and incorrect functionality second; over half are observed during training, complicating testing and debugging.
- 4.2.1 Symptom Classification Results.: The study classifies six symptoms: Crash, Incorrect Functionality, Build Failure, Poor Performance, Hang, and Unreported.Unreported bugs are those whose symptoms cannot be identified from pull requests, related issues, discussions, and code changes.
- 4.2.1 Symptom Classification Results.: 51.40% of DL framework bugs exhibit Crash, making it the most common symptom across the four studied frameworks.Crash counts are 122 for TensorFlow, 106 for PyTorch, 142 for MXNet, and 144 for DL4J, totaling 514 bugs.
- 4.2.1 Symptom Classification Results.: Incorrect Functionality is the second most common symptom, accounting for 24.30% of bugs and creating a difficult test-oracle problem.Among 243 such bugs, 115 produce incorrect intermediate states, 105 incorrect prediction results, and 23 incorrect model structures.
- 4.2.2 Symptom Distribution.: About 55.10% of DL framework bugs are observed during Training, where costly and non-deterministic execution can make testing and debugging lengthy.Incorrect Functionality accounts for 27.95% of bugs observed at this stage, and manifestations may appear only after hours or days.
4.3 RQ3: Relationship between Root Causes and Symptoms
Root causes relate systematically to symptoms: Build Failure is concentrated in configuration and environment issues, while Poor Performance is linked to algorithmic and API problems.
- 4.3 RQ3: Relationship between Root Causes and Symptoms.: Crash and Incorrect Functionality are the most common symptoms for most root causes, making their test oracles broadly useful across bug categories.Crash has an explicit oracle, whereas Incorrect Functionality remains difficult to detect because differential testing can produce false positives and false negatives under DL randomness.
- 4.3 RQ3: Relationship between Root Causes and Symptoms.: Type Issue and Tensor Shape Misalignment can produce many Incorrect Functionality bugs rather than only crashes.An unsafe pointer conversion in a TensorFlow Ethos-U kernel changed a value incorrectly and produced incorrect intermediate results.
- 4.3 RQ3: Relationship between Root Causes and Symptoms.: Developers can prioritize highly relevant root causes according to a bug’s symptom to speed debugging.This guideline follows the observed relationships between symptoms and root-cause categories.
- 4.3 RQ3: Relationship between Root Causes and Symptoms.: Build Failure is highly relevant to Misconfiguration and Environment Incompatibility, while Poor Performance is linked to Incorrect Algorithm Implementation and API Misuse.Among 188 Build Failure bugs, 65.43% are caused by Misconfiguration and 13.30% by Environment Incompatibility; among 21 Poor Performance bugs, 52.38% involve the two highlighted causes.
4.4 RQ4: Bug-Occurring Levels
Bug occurrence differs across the five framework levels: Operation Implementation contains the most bugs, while DL-specific and traditional causes concentrate at different levels.
- 4.4 RQ4: Bug-Occurring Levels.: 435 bugs occur in Graph-Level Implementation and Operation Implementation, the two core levels underlying training and inference computation graphs.The study identifies extensive testing of Operation Implementation as urgent for improving DL framework reliability.
- 4.4 RQ4: Bug-Occurring Levels.: Operation Implementation contains 30.77% of bugs, the largest share, while Environment-Dependent Processing contains the fewest.Operation Implementation includes many complex algorithms and substantial source code, whereas Environment-Dependent Processing involves comparatively little code.
- 4.4 RQ4: Bug-Occurring Levels.: DL-specific causes are chiefly distributed at Operation Implementation, whereas traditional bug categories are chiefly distributed at User-Level API.The five-level architecture comprises User-Level API, Graph-Level Implementation, Operation Implementation, General Utility, and Environment-Dependent Processing.
4.5 RQ5: Bug Commonality
The four DL frameworks share substantial root-cause and symptom patterns, although their component-level bug distributions differ with framework designs and implementations.
- 4.5 RQ5: Bug Commonality.: All four frameworks share the same top five root causes and commonly exhibit Crash and Incorrect Functionality symptoms.The top five root causes are Incorrect Algorithm Implementation, Type Issue, Misconfiguration, Tensor Shape Misalignment, and API Misuse.
- 4.5 RQ5: Bug Commonality.: Frameworks display diverse bug distributions across components because their designs and implementations differ.The observed commonality of root causes and symptoms should therefore be combined with component-level differences when designing detection approaches.
- 4.5 RQ5: Bug Commonality.: The four DL frameworks show significant commonality in both root causes and symptoms.The study compares framework distributions using pairwise Spearman correlations across root causes, symptoms, and components.
5 IMPLICATIONS AND THE APPLICATION
The section evaluates existing DL-framework testing techniques, derives actionable testing and debugging guidelines, and applies them in TenFuzz, which confirmed six unique TensorFlow bugs, including three still present in the latest version.
- 5.1 Status of Existing Testing Techniques: The studied techniques achieve low coverage, especially for Operation Implementation, General Utility, and Environment-Dependent Processing.Their combined line, branch, and function coverage is substantially below the equipped test suite on MXNet and PyTorch.
- 5.1 Status of Existing Testing Techniques: Coverage largely depends on pre-trained models because LEMON and Audee add only small increments beyond CRADLE.
- 5.2 Implications: The study recommends component-targeted testing, prioritizing Operation Implementation because it contains many bugs but remains poorly covered.The authors also propose mutation operators for tensor types, shapes, and environments, plus improved oracles for crashes and incorrect functionality.
- 5.2 Implications: Additional guidelines address unstable, time-consuming training-bug reproduction and automated fixing of build failures caused by misconfiguration or environment incompatibility.
- 5.3 TenFuzz: A Preliminary Application: TenFuzz mutates applicable TensorFlow tests and uses differential testing across versions with voting to identify inconsistent behavior.
- 5.3 TenFuzz: A Preliminary Application: TenFuzz reported 9 potential tests and detected 6 unique bugs; 3 remained in the latest TensorFlow version, and all reported bugs were reproduced and confirmed by developers.
- 5.3 TenFuzz: A Preliminary Application: TenFuzz is conceptually general but requires framework-specific mutation-operator changes because equipped tests use different data structures and formats.
6 THREATS TO VALIDITY
The validity discussion identifies threats from the sampled data, manual labeling, and selected testing techniques, while describing procedures used to mitigate labeling concerns.
- The study cannot account for bug importance because the source repositories do not provide that information.
- The study’s data comprise 1,000 bugs from four DL frameworks collected through systematic identification and manual investigation.
- Manual labeling is an internal validity threat, mitigated through independent labeling, senior-author adjudication, and a 95% Cohen’s Kappa agreement.
- Findings 11 and 12 may not generalize to other DL-framework testing techniques because the evaluation selected only three techniques.
7 RELATED WORK
Related work spans DL-framework and DL-program bug studies, traditional software-bug research, and graph-level or operator-level DL-framework testing; this study emphasizes broader framework coverage and actionable guidance.
- Unlike prior work focused on one framework, this study analyzes 1,000 bugs across four diverse DL frameworks to support broader conclusions.
- DL-program bug studies concern incorrect API usage, whereas DL-framework bug studies concern defects inside framework implementations.
- The paper distinguishes its contribution from traditional software-bug studies by targeting DL-framework bugs and providing testing guidance based on empirical findings.
- Existing DL-framework testing techniques mainly generate graph-level or operator-level tests, including differential-testing approaches such as CRADLE, LEMON, and Audee.
8 CONCLUSION
The conclusion presents a large empirical study of DL-framework bugs and testing practices, yielding 12 findings and actionable implications for future detection and debugging research.
- The authors manually analyze 1,000 bugs from four popular DL frameworks across root causes, symptoms, and framework-component correlations.
- The study summarizes 12 major findings and derives actionable implications for future DL-framework bug detection and debugging.