Fuzzing Deep-Learning Libraries via Automated Relational API Inference

1 INTRODUCTION

• DL libraries receive less testing attention than DL models, although they underpin model building, training, and execution.Prior DL testing largely focused on models, while library-testing techniques included model-level differential testing and API-level input mining.Source: §1 · p. 1
• Prior fuzzers leave APIs untested when no documentation example, developer test, or DL model directly invokes them.FreeFuzz covered up to 1,158 APIs for PyTorch and TensorFlow but could not test an API without an invoking snippet.Source: §1 · pp. 1–2
• DeepREL treats identical outputs as value equivalence and similar execution status as status equivalence, allowing relational APIs to supply inputs and serve as test oracles.The status relation also covers functionally similar APIs whose outputs differ, such as adaptive average and maximum pooling.Source: §1 · p. 2
• DeepREL infers candidate API relations from syntactic and semantic information, synthesizes test programs, validates pairs with representative inputs, and fuzzes verified pairs.The approach was designed to work without relying on documented API relations, which are rare in DL libraries.Source: §1 · p. 2
• 157% more APIs were covered by DeepREL than by FreeFuzz, with 162 bugs detected, including 106 confirmed as previously unknown.DeepREL also detected 13.5% of high-priority bugs in the PyTorch issue-tracking system in three months and 14 confirmed documentation bugs.Source: §1 · p. 2

2.1 Basics about DL Models and APIs

• A DL program defines a computation graph of API nodes and tensor-flow edges, with forward, loss-computation, and backward parts.Training runs inference, computes loss and gradients, then updates weights through an optimizer.Source: §2.1 · pp. 2–3

2.2 Fuzzing DL Libraries

• Model-level testing mutates or compares DL models across backends, but its API coverage is constrained; LEMON covered only 35 TensorFlow APIs.LEMON's intact-layer mutation requires matching input and output tensor shapes, limiting which APIs can be added or removed.Source: §2.2 · p. 3
• FreeFuzz mines and mutates traced API inputs from documentation, developer tests, and 202 DL models, but covers only 1,158 APIs and leaves 6,815 uncovered.Its CPU/GPU differential oracle may miss bugs when hardware backends share code logic or design.Source: §2.2 · p. 3
• DeepREL builds on API-level fuzzing by inferring relational APIs, while its approach can also be built on other API-level fuzzers such as DocTer.FreeFuzz was selected as the base because it was publicly available, recent, and fully automated.Source: §2.2 · p. 3

3 PRELIMINARIES

• Value equivalence modulo D holds when source and target APIs produce the same results for every input in D.The broadcast_shapes and broadcast_tensors example illustrates APIs with the same functionality despite different implementations.Source: §3 · p. 3
• Status equivalence modulo D holds when source and target APIs have the same invocation status for every input in D, even if their outputs differ.Adaptive average and maximum pooling can share status while producing different outputs; status equivalence is a relaxation of value equivalence, itself a relaxation of semantic equivalence.Source: §3 · p. 4
• The domain D constrains which inputs the relation covers, so representative inputs from the intersection of both APIs' valid input spaces matter.Source: §3 · p. 4

4 FUZZING RELATIONAL APIS

• DeepREL takes API documentation and a database of historically valid invocations as inputs, with each invocation record containing dynamically traced argument values.The database can be collected from documentation examples, library tests, and DL models.Source: §4 · p. 4

4.1 API Matcher

• The API Matcher ranks candidate pairs using the maximum of signature similarity and document similarity, combining API syntax with description semantics.It pairs each API with its K closest neighbors; the default K is 10.Source: §4.1 · pp. 4–5

4.2 Invocation Synthesizer

• The Invocation Synthesizer maps source arguments to target arguments using weighted bipartite matching, and uses defaults for unmatched optional arguments.It can also extract documentation code snippets as templates for target invocations; argument matching aborts when unmatched arguments are non-optional.Source: §4.2 · pp. 5–6

4.3 API Match Verifier

• The API Match Verifier accepts pairs when representative inputs produce equal values or, if values differ, matching execution statuses; otherwise, it rejects them.Unrepresentative verifying inputs can lead to false acceptance, while inputs that trigger real bugs can lead to rejection of true matches.Source: §4.3 · pp. 6–7

4.4 API Fuzzer

• The API Fuzzer mutates valid source-API inputs and compares the verified pair's outputs for value equivalence or execution statuses for status equivalence.These comparisons are used to detect potential consistency bugs.Source: §4.4 · p. 7

4.5 The Iterative Process

• DeepREL repeats its four phases until no newly covered API is found or it reaches the iteration limit, using successful target invocations as sources in later rounds.The default iteration limit is 10.Source: §4.5 · p. 7

5 EXPERIMENTAL SETUP

• Experiments use PyTorch 1.10 and TensorFlow 2.7, identified as their latest stable releases, on an 8-core Xeon machine with 16GB RAM, Ubuntu 20.04, and Python 3.9.The libraries are described as popular and widely studied in prior DL-library testing work.Source: §5 · p. 7

5.1 Implementation

• The implementation parses documentation for 7,973 APIs and uses FreeFuzz invocation traces to verify pairs and mutate inputs for fuzzing.It uses the first 100 valid source invocations for verification when available, and generates 1,000 fuzzing inputs per source API.Source: §5.1 · p. 7

5.2 Metrics

• The evaluation measures covered APIs, the false-positive rate among inconsistent API pairs, and the number of distinct bugs detected.An API counts as covered when the verifier successfully invokes it as a source or target API.Source: §5.2 · pp. 7–8

6.1 RQ1: Effectiveness in API Coverage

• 2,973 APIs were covered by DeepREL, 157% more than FreeFuzz’s 1,158 across PyTorch and TensorFlow.For PyTorch, DeepREL covered 1,071 APIs, 128% more than FreeFuzz.Source: Table 3 · p. 8, §6.1 · p. 8
• Of 13,098 verified API pairs, 11,419 involved newly covered APIs; PyTorch had 2,933 status-equivalent versus 1,357 value-equivalent pairs.A documentation study found 2,692 of 2,942 explicitly linked API pairs were value-equivalent, including 1,828 TensorFlow backward-compatibility pairs.Source: §6.1 · p. 8

6.2 RQ2: False Positives

• DeepREL’s false-positive rate was 30.72%; incomplete valid-input data could let the verifier accept incorrect API relations.The value-equivalence oracle had a lower false-positive rate than the status-equivalence oracle.Source: §6.2 · p. 9, Table 5 · p. 10

6.3 RQ3: Impacts of Configuration

• K=10 is the authors’ default because larger K brings marginal API-coverage gains while worsening false-positive rate and runtime.K=10, 15, and 20 had similar coverage, all significantly higher than K=5.Source: §6.3 · p. 9
• As iterations increase, API-coverage gains diminish and false-positive rate generally rises; runtime grows more at early iterations.Default runtime was 12.8 hours for PyTorch and 26.3 hours for TensorFlow; coverage converged within 10 iterations.Source: §6.3 · p. 9

6.4 RQ4: Bugs Detected

• Of 106 confirmed bugs, 87 were consistency bugs, and 52 of those involved newly covered APIs; DeepREL also found 23 high-priority PyTorch bugs.Those 23 represented 13.5% of the 170 high-priority issues in PyTorch’s issue tracker during the stated three-month period.Source: §6.4 · pp. 9–10
• 162 bugs were detected, including 106 confirmed as previously unknown and 29 already fixed.Only 9 of the 106 confirmed bugs were also found by FreeFuzz; 14 documentation bugs were found separately.Source: §6.4 · p. 9, Table 6

6.5 Threats to validity

• The evaluation’s external-validity threat is its benchmark scope: DeepREL was evaluated on TensorFlow and PyTorch.The authors identify implementation correctness as an internal-validity threat and metrics as a construct-validity threat.Source: §6.5 · p. 11

• API-mapping work learns cross-language API correspondences for code migration, while DeepREL targets fuzzing and dynamically verifies documentation-derived relations without requiring extensive API usages or training data.Source: §7 · p. 11
• Function-synonym approaches find functions with similar roles, but DeepREL handles arbitrary within-library relations and complex DL inputs beyond APIs with matching functions and argument lists.FPDiff uses elementary double/int inputs, which cannot serve as a universal input set for complex DL APIs.Source: §7 · p. 11
• MeMo derives metamorphic relations from natural-language specifications, but its reliance on well-documented relations and lack of invocation synthesis and verification limit its DL-library results.Replacing DeepREL’s first three phases with MeMo yielded 17/28 API relations for TensorFlow/PyTorch, versus 8808/4290 for DeepREL, and at most 8 of DeepREL’s 106 confirmed bugs.Source: §7 · p. 12

• 162 bugs were detected, with 106 confirmed by developers as previously unknown bugs.Source: §8 · p. 12
• 13.5% of high-priority bugs in the entire PyTorch issue-tracking system were detected in a three-month period.Source: §8 · p. 12
• 14 documentation bugs were detected, and all were confirmed.Source: §8 · p. 12