PaperlayerView exact revision on arXiv

Source-linked AI summary

Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source

Anjiang Wei, Yinlin Deng, Chenyuan Yang, Lingming Zhang

arXiv:2201.06589v4cs.SE

TL;DR

Testing DL libraries is limited despite their importance, partly because Python’s dynamic typing complicates automatic API test-input generation. FreeFuzz mines three open-source input sources, traces dynamic execution information, and uses it for API fuzzing. On PyTorch and TensorFlow, it traced 1158 APIs and detected 49 bugs, including 38 confirmed previously unknown.

  • Problem

    Testing research has largely targeted DL models, while limited work tests the underlying DL libraries whose Python APIs make valid parameter types difficult to infer automatically.

  • Method

    FreeFuzz runs documentation snippets, developer tests, and DL models with instrumentation to trace API argument types, values, and tensor shapes for mutation-based fuzzing.

  • Results

    FreeFuzz traced valid dynamic information for 1158 APIs, detected 49 bugs, and had 9X more API coverage with 3.5X lower overhead than LEMON.

  • Takeaways & Limitations

    FreeFuzz provides automated API-level fuzzing that broadens DL-library test coverage and detects bugs in PyTorch and TensorFlow.

  • Takeaways & Limitations

    The evaluation covers PyTorch and TensorFlow, and nondeterminism may affect FreeFuzz’s effectiveness across runs.

Abstract

from arXiv · show

Deep learning (DL) systems can make our life much easier, and thus are gaining more and more attention from both academia and industry. Meanwhile, bugs in DL systems can be disastrous, and can even threaten human lives in safety-critical applications. To date, a huge body of research efforts have been dedicated to testing DL models. However, interestingly, there is still limited work for testing the underlying DL libraries, which are the foundation for building, optimizing, and running DL models. One potential reason is that test generation for the underlying DL libraries can be rather challenging since their public APIs are mainly exposed in Python, making it even hard to automatically determine the API input parameter types due to dynamic typing. In this paper, we propose FreeFuzz, the first approach to fuzzing DL libraries via mining from open source. More specifically, FreeFuzz obtains code/models from three different sources: 1) code snippets from the library documentation, 2) library developer tests, and 3) DL models in the wild. Then, FreeFuzz automatically runs all the collected code/models with instrumentation to trace the dynamic information for each covered API, including the types and values of each parameter during invocation, and shapes of input/output tensors. Lastly, FreeFuzz will leverage the traced dynamic information to perform fuzz testing for each covered API. The extensive study of FreeFuzz on PyTorch and TensorFlow, two of the most popular DL libraries, shows that FreeFuzz is able to automatically trace valid dynamic information for fuzzing 1158 popular APIs, 9X more than state-of-the-art LEMON with 3.5X lower overhead than LEMON. To date, FreeFuzz has detected 49 bugs for PyTorch and TensorFlow (with 38 already confirmed by developers as previously unknown).

1 INTRODUCTION

Testing DL libraries remains limited despite their central role in building and running models. FreeFuzz addresses this gap through automated API-level fuzzing that mines diverse execution data from open-source code and models.

  • DL libraries are central infrastructure for building, training, optimizing, and deploying models, yet testing research has focused mainly on DL models.
  • Existing DL-library testing uses limited input sources, covering at most 59 TensorFlow APIs in the authors’ empirical results.CRADLE uses 30 pretrained models, while LEMON uses 12.
  • Model-level mutation restricts API arguments through shape constraints, limiting the diversity of generated test inputs.LEMON’s intact-layer mutation requires the added or deleted layer’s output shape to match its input shape.
  • API-level fuzzing offers finer-grained and more systematic testing without external-dataset iteration or complex bug localization.Testing one API at a time avoids accumulated floating-point precision loss.
  • FreeFuzz mines documentation snippets, developer tests, and DL models, traces dynamic API information, and mutates it for fuzz testing.Traced information includes argument types, values, and tensor shapes; mutation uses type, random-value, and database-value strategies.
  • FreeFuzz traces 1158 APIs and detects 49 bugs, including 38 confirmed previously unknown and 21 fixed by developers.

2 BACKGROUND

DL libraries support model training, inference, and hardware abstraction through extensive APIs. Their Python-facing dynamic typing makes API test-input generation difficult because valid parameter types may not be inferable from definitions.

  • 2.1 Preliminaries for Deep Learning Libraries: DL libraries provide APIs for constructing neural-network layers and support training and inference workflows.PyTorch examples invoke Conv2d and Maxpool2d to add layers, while training updates weights and inference produces predictions.
  • 2.1 Preliminaries for Deep Learning Libraries: DL libraries abstract over different hardware, allowing users to configure execution across platforms.
  • 2.2 Fuzzing Deep Learning libraries: Typical DL libraries contain hundreds or thousands of APIs, making manual test-input construction impractical.
  • 2.2 Fuzzing Deep Learning libraries: Python’s dynamic typing prevents static determination of many API parameter types from definitions alone.For PyTorch 2D-Convolution, parameters such as in_channels, out_channels, and kernel_size lack inferable types from the shown definition.
  • 2.2 Fuzzing Deep Learning libraries: A single API argument may accept multiple valid types, such as an integer or a tuple for stride and strings as well as integers for padding.
  • 2.2 Fuzzing Deep Learning libraries: CRADLE tests DL libraries by checking cross-backend inconsistencies while executing existing models with dataset inputs.
  • 2.2 Fuzzing Deep Learning libraries: LEMON extends this model-level approach with intact-layer and inner-layer mutation rules intended to invoke more library code.

3 APPROACH

FreeFuzz fuzzes deep-learning libraries at the API level by collecting executions from open-source documentation, developer tests, and models, then tracing and mutating valid dynamic inputs. Its pipeline uses customized type monitoring, multiple mutation strategies, and database-guided value reuse before executing tests with differential and metamorphic oracles.

  • 3.1 Code Collection: FreeFuzz automatically collects code and models from documentation snippets, library developer tests, and DL models in the wild.These sources provide diverse executions for API-level fuzzing.
  • 3.2 Instrumentation: Instrumentation hooks 630 PyTorch APIs and 1900 TensorFlow APIs, recording parameter types and values during executions.The traced information forms the spaces used for fuzzing.
  • 3.2 Instrumentation: FuzzType records finer-grained tensor information, including dimensions and data types, beyond Python’s built-in type system.For example, a tensor can be represented as Tensor<4,float32> rather than only torch.Tensor.
  • 3.3 Mutation: FreeFuzz mutates traced arguments through tensor-dimension and dtype changes, random values, and database values selected from APIs sharing argument names and types.The mutation design combines type mutation, random value mutation, and database value mutation.
  • 3.3 Mutation: The mutation algorithm selects an argument, obtains its customized type, and nondeterministically applies type, random-value, or database-value mutation.Arguments are mutated one by one using the traced type, name, and value information.
  • 3.3 Mutation: Database-guided mutation computes normalized text similarity between API definitions, samples a similar API, and reuses its recorded argument values.Higher-similarity APIs receive higher sampling probabilities through Softmax normalization.

4 EXPERIMENTAL SETUP

The evaluation examines FreeFuzz’s input sources, mutation settings, mutation strategies, comparison with existing work, and real-world bug detection on PyTorch and TensorFlow. It measures API coverage, traced value-space size, and C/C++ line coverage using collected documentation code and related inputs.

  • Five research questions assess input-source contributions, mutation counts, mutation strategies, comparisons with existing work, and real-world bug detection.
  • Experiments use stable releases PyTorch 1.8 and TensorFlow 2.4 on a machine with four 2.20GHz Intel Xeon cores, NVIDIA A100 GPUs, Ubuntu 16.04, and Python 3.9.
  • Documentation collection downloads 497 PyTorch and 512 TensorFlow code snippets to form the original seed test pool.
  • The evaluation measures covered APIs, deduplicated API value-space size, and line coverage.
  • Line coverage is collected only for C/C++ code, whose implementation provides fundamental operator support in the evaluated DL libraries.

5 RESULT ANALYSIS

FreeFuzz benefits from combining diverse open-source inputs, API-level mutation, and differential testing to broaden coverage and detect real DL-library bugs. Across PyTorch and TensorFlow, the approach improves coverage over prior work while maintaining acceptable tracing overhead and revealing confirmed defects.

  • Input sources: 59/62 APIs were covered by all three input sources for PyTorch/TensorFlow, showing that each source also contributes distinct API coverage.The majority of covered code is shared across sources, but API coverage has substantial source-specific contributions.
  • Overhead: Less than 20 minutes, 2.5/5.0 hours, and less than 1 hour were required to trace documentation, developer tests, and models respectively for PyTorch/TensorFlow.Tracing is a one-time effort reusable across subsequent versions of the same libraries.
  • Mutation effectiveness: FreeFuzz covered more code as mutation counts increased, with coverage becoming largely stable after 600 mutations.The result supports the effectiveness of the mutation strategies, while the default configuration uses 1000 mutants per covered API.
  • Mutation effectiveness: All three mutation strategies were necessary for the best code coverage, with type mutation more effective than random-value or database-value mutation.Random-value and database-value mutation performed similarly in code coverage.
  • Comparison with prior work: FreeFuzz covered ∼9X more APIs than LEMON, consumed ∼3.5X less time, and achieved ∼20% higher coverage.With models only, FreeFuzz also exceeded LEMON’s code coverage within 20 minutes, or 75X faster.
  • Bug detection: 49 bugs were detected across PyTorch and TensorFlow, including 38 confirmed previously unknown bugs and 21 fixed by developers.Only one of the 49 bugs was detectable by both LEMON and CRADLE.
  • Bug detection: FreeFuzz detected wrong-computation, performance, and crash bugs through differential and metamorphic testing.Examples include backend-dependent 2D-convolution results, a float16 performance regression, and an invalid Conv3d padding-mode input.

6 DISCUSSION AND FUTURE WORK

FreeFuzz can generalize beyond deep-learning library fuzzing, but its inputs remain imperfect and its API-level scope leaves sequence-triggered and infrastructure-dependent failures unresolved.

  • Validity of Test Inputs: FreeFuzz’s mining and mutation strategies improve input validity but still cannot satisfy all complicated input constraints.Invalid inputs nevertheless detected bugs, including unexpected crashes.
  • Generalizability and Specificity: FreeFuzz includes DL-specific components, so generalization beyond DL libraries is limited by tensor types, mutation rules, and DL-specific oracles.The documentation and developer-test mining idea can generalize to APIs in dynamically typed languages.
  • Future Work: FreeFuzz tests single-API correctness and may miss bugs triggered only by invoking API sequences.The authors identify sequence-based testing as future work.
  • Future Work: Implementation-dependent flaky tests can pass on one machine and fail on another despite identical scripts and library versions.The authors attribute this behavior probably to differences in underlying infrastructure and hardware.

7 RELATED WORK

Prior work has extensively tested DL models, while library-testing research has focused on differential testing and model mutation; FreeFuzz instead targets the underlying libraries directly.

  • DL Model Testing: DL model testing spans adversarial attacks, defense techniques, robustness metrics, application-specific testing, static analysis, gradient propagation, concolic testing, mutation testing, and test-input generation.These approaches primarily improve model quality or detect model-level and architecture-level issues.
  • DL Library Testing: FreeFuzz differs from prior work by targeting the underlying DL libraries that support training and deploying diverse DL models.The paper positions DL libraries as the infrastructure beneath model development and use.
  • DL Library Testing: CRADLE introduced differential testing for Keras, while LEMON extended DL-library testing with mutation rules for generating more models.FreeFuzz is presented as distinct from these model-focused library-testing approaches.

8 CONCLUSION

FreeFuzz mines open-source documentation, developer tests, and real-world DL models to fuzz library APIs using traced dynamic information. On PyTorch and TensorFlow, it covered 1158 APIs and found 49 bugs, including 38 confirmed as previously unknown.

  • Approach: FreeFuzz combines three open-source sources—library documentation, developer tests, and DL models in the wild—with instrumentation-based dynamic tracing for API fuzzing.The traced information includes dynamic invocation data used to fuzz each covered API.
  • Results: 1158 popular APIs received valid dynamic information for fuzzing, 9X more than state-of-the-art LEMON with 3.5X lower overhead.These results come from the extensive PyTorch and TensorFlow study.
  • Results: 49 bugs were detected in PyTorch and TensorFlow, with 38 confirmed by developers as previously unknown bugs.The confirmation count is reported as part of the paper’s conclusion.
Loading 2201.06589v4…