Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow

Motivation

• Silent framework bugs can preserve apparently successful model creation, training, or inference while producing inaccurate predictions, degraded computation, or incorrect information.Their lack of concrete errors makes it difficult for users to connect suspicious behavior to framework defects such as incorrect weight updates.Source: §1 · p. 2

Research gap

• The work addresses a gap left by prior DL-bug studies that did not precisely examine real silent bugs or their effects on users’ code.It formulates questions about program consequences, API sources and root causes, and relevance to DL users.Source: §1 · pp. 3–4

Validation

• A survey of 103 TensorFlow DL users assessed the relevance and completeness of the categories and impact levels, with participants acknowledging the findings.The survey specifically validated the study’s characterization of how silent bugs impact users’ programs.Source: §1 · p. 4

Study scope

• 1,168 closed TensorFlow issues yielded 77 reproducible silent bugs affecting users’ DL programs.The study classifies these bugs by their observed effects on programs that create, train, or infer DL models.Source: §1 · p. 4

Contributions

• The study identifies seven silent-bug categories, four impact levels, and 14 Keras API components associated with the bugs.It also examines possible root causes using an existing taxonomy and classifies impacts on users’ programs.Source: §1 · pp. 4–5

TensorFlow

• TensorFlow uses dataflow graphs to define ML computations, operations, and states, with differentiable programming and tensor operations across CPUs, GPUs, and TPUs.The platform also supports deployment by exporting graphs to external runtime environments.Source: §2.1 · p. 5

Keras

• Keras is a Python DL API running on TensorFlow that supplies high-level abstractions and building blocks for developing and shipping DL solutions.It was designed to enable fast experimentation and lower the expertise required for DL development.Source: §2.1 · p. 6

Issue tracking

• TensorFlow’s GitHub repository stores user issues containing diagnostic details such as platform, version, reproduction code, and expected behavior.Developers use these reports and associated commits to investigate and address library problems.Source: §2.2 · p. 6

Classification

• 77 bugs were classified into 7 impact scenarios and 14 affected API components, with each bug assigned one of four impact levels.The classification used issue descriptions, reproduction artifacts, and observed consequences for users’ programs.Source: §3.1.3 · pp. 9–11

Impact scenarios

• Wrong parameter setting affected 20.8% of bugs and could alter network behavior without warning, potentially causing convergence issues.One example changed an optimizer’s learning-rate behavior even when the learning rate was set to zero.Source: §3.2.1 · p. 13
• Wrong save/reload affected 13% of bugs and could completely change model results, including a reported accuracy collapse after loading saved weights.Without verification, users could obtain mostly incorrect predictions because no errors or warnings were raised.Source: §3.2.1 · pp. 14–15
• Wrong resulting shape affected 7.79% of bugs, while performance degradation affected 5.19%, including excessive memory use and substantially slower execution.The shape category could return incorrect model information without altering operation, whereas performance degradation excluded prediction-accuracy changes.Source: §3.2.1 · pp. 15–16

Impact levels

• Wrong calculation produced the highest-impact scenario, with 13 Level-4 samples, followed by wrong save/reload with 6 high-impact samples.These scenarios could drastically affect model results and often required inspecting the model to detect the problem.Source: §3.2.1 · pp. 18–19

Faulty components

• Engine, Layer, and Model were the most affected components, while all issues in Saving were high-impact and nearly half in Layer and Backend were high-impact.Engine contained 19 samples, Layer 14, and Model 13; Engine is a hidden Keras backbone component.Source: §3.2.2 · p. 21

• 97.1% of respondents found the silent-bug definition meaningful, supporting its relevance among surveyed DL users.Source: §4.2.1 · p. 26
• Respondents generally judged the categories difficult or dangerous to diagnose, assess, and fix, with limited evidence for Wrong Shape and Performance Degradation.Source: §4.2.2 · pp. 27–28
• The proposed categories may be too general, while documentation issues remained outside the paper’s silent-bug scope.Source: §4.2.2 · p. 28
• 75.8% of respondents agreed that the proposed impact scale was relevant, although some suggested merging levels 3 and 4.Source: §4.2.3 · p. 28
• 72.8% of respondents considered silent bugs more problematic than traditional bugs that cause errors or program hangs.Source: §4.2.4 · p. 29
• τ = 0.51 indicated a moderate-to-strong correlation between the study’s category impact levels and respondents’ perceived impacts.Source: §4.2.5 · p. 29

• Silent bugs can produce harmful program behavior without obvious symptoms, and respondents largely judged them more problematic than traditional bugs.Source: §5 · p. 31
• Root-cause analysis and repair remain difficult because silent bugs affect framework components differently from their observed user-program impacts.Source: §5 · p. 31
• Wrong save/reload bugs are amenable to tests that compare model structure, parameters, and benchmark predictions before and after reloading.Source: §5 · p. 31
• Existing differential-testing techniques have achieved some success but generally provide low coverage and repeatedly exercise the same code.Source: §5 · p. 32
• Testing should prioritize Engine, Model, and Layer components, especially Engine because dependent modules can mask incorrect behavior.Source: §5 · pp. 32–33

• Software-engineering research has studied silent failures, test-code “silent horrors,” miscompilation, and numerical correctness bugs as related but distinct problems.Source: §6 · p. 33
• Equivalent Modulo Input found silent compiler defects effectively, but the paper notes that translating this approach to DL frameworks is unclear.Source: §6 · p. 33
• Earlier DL studies examined reproducible bugs, taxonomies, symptoms, causes, and repair patterns, but focused mainly on DL programs or general TensorFlow bugs.Source: §6 · pp. 33–34
• Unlike prior studies, this paper specifically investigates silent bugs inside a DL framework and their impact on users’ programs and experience.Source: §6 · p. 34

Construct validity

• Closed issues may underrepresent silent bugs because open issues lack solutions, commits, or fixed versions needed to verify reproducibility and localization.The study therefore inspected verified reproducible bugs with fixed versions to reduce subjectivity and noise.Source: §7 · p. 34
• Independent raters used open coding, repeated consensus rounds, and a DL-user survey to mitigate bias in the newly developed scenario and impact categories.No prior taxonomy existed for these categories.Source: §7 · pp. 34–35

Internal validity

• The 77-bug sample is smaller than broader TensorFlow studies, but it targets the harder-to-diagnose silent-bug subcategory and yielded seven validated categories.The closest categories in one 250-bug study comprised at most 19% of cases, or 47 bugs.Source: §7 · p. 35

External validity

• Analyzing one framework limits generalization, even though the authors argue TensorFlow's prevalence, Keras's importance, and cross-framework similarities support broader relevance.The authors also state that the established impact remains important even if the phenomenon does not generalize.Source: §7 · p. 35

Reliability validity

• A detailed replication package is provided so other researchers can reproduce or extend the study.Source: §7 · p. 35

8 Conclusion

• 77 reproducible silent bugs were identified by inspecting 1,168 TensorFlow GitHub issues, then classified into 7 scenarios and 4 impact levels.The classification covers effects on designing models, training, and inference, as well as responsible framework components.Source: §8 · p. 35
• A survey of 103 TensorFlow users found general agreement that silent bugs significantly affect DL libraries and acknowledged the proposed categories and impact scale.Source: §8 · p. 35
• The study provides a basis for detection techniques by showing where silent bugs occur in Keras/TensorFlow and how they manifest in users' programs.Source: §8 · pp. 35–36
• Future work includes comparing bug types across frameworks and designing techniques to identify abnormal behavior and trace silent-bug root causes.The authors suggest similar observations may apply to other frameworks because they face similar issues.Source: §8 · p. 36