NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers

• NNSmith generates valid computation graphs, compiles them, runs compiled and reference models on generated inputs, and compares their outputs.Source: §1 · p. 2
• Operator specifications and incremental constraint solving let NNSmith generate diverse graphs while satisfying operator and tensor-type constraints.NNSmith uses an SMT solver to check type-matching constraints as it inserts candidate operators.Source: §1 · p. 2
• Gradient-guided search found viable inputs and weights for 98% of generated models with negligible overhead, avoiding floating-point exceptional values.Source: §1 · p. 2
• NNSmith is designed for extension to new operators and compilers through operator specifications and a library of common patterns.The authors report that new operator specifications can be written in a few lines of code.Source: §1 · p. 2
• NNSmith outperformed the state-of-the-art fuzzer by 1.8× for ONNXRuntime and 1.08× for TVM in total branch coverage, and 32.7× and 10.8×, respectively, in unique branches.Source: §1 · p. 2
• 72 new bugs were found; developers confirmed 58 and fixed 51 across the evaluated frameworks.Source: §1 · p. 2

2.1 The DNN Computation Graph

• A DNN computation graph connects tensor operators, and a graph is valid only when its operators and their input and output tensor types satisfy the required constraints.A tensor type comprises a tensor’s shape and element type.Source: §2.1 · p. 2

2.2 DL Compilers

• DL compilers convert input graphs to internal representations and apply transformation passes; bugs can arise in either phase.The later transformation phase is generally harder to identify and debug, motivating tests with diverse graph structures and operators.Source: §2.2 · p. 3

2.3 Challenges in Finding DL Compiler Bugs

• Prior fuzzers’ graph restrictions limit diversity, while shape-fixing by slicing and padding can bias generated graphs and silence bugs.NNSmith’s example shows a layout bug triggered by a graph that LEMON cannot generate and that GraphFuzzer’s slicing would alter.Source: §2.3 · p. 3
• Default or arbitrary operator attributes can miss bugs, and default weights produced NaN or Inf in 56.8% of generated 20-node models.Prior testing frameworks did not address exceptional values, and up to 41% of their bug reports could be false alarms.Source: §2.3 · p. 3
• Differential testing must exclude executions with exceptional values even when final outputs appear normal, because upstream NaN or Inf can make equivalent operators produce different results.Source: §2.3 · pp. 3–4

3.1 Modeling DNN Operators

• Operator specifications encode compiler validity requirements and output guarantees, which NNSmith combines across connected operators to generate valid models.Specifications describe input and output tensor types, constraints on inputs and attributes, and type-transfer rules.Source: §3.1 · p. 4

3.2 Model Generation

• NNSmith grows connected graphs through forward or backward insertion, using type matching and SMT constraint checks to find feasible insertion points.Forward insertion adds an operator to existing outputs; backward insertion replaces a placeholder and creates inputs of inferred types.Source: §3.2 · pp. 4–5
• Because SMT solvers can favor boundary-valued attributes, NNSmith adds randomly selected binning constraints and relaxes them when they make the graph unsatisfiable.Exponential-range bins encourage diversity, with constraint dropping and retries preserving satisfiability.Source: §3.2 · pp. 5–6

3.3 Improving Numeric Validity with Gradients

• With random weights and inputs, 56.8% of generated 20-operator graphs produce floating-point exceptions, which can propagate and prevent output comparisons.Vulnerable operators such as square root and power are valid only over subsets of their input domains.Source: §3.3 · p. 6
• NNSmith iteratively optimizes inputs and weights against losses derived from violated operator-domain inequalities, using proxy derivatives where gradients are zero or unavailable.It targets the first exceptional-value-producing operator and restarts with new initial values when gradients are unusable; the search can fail when a model has no valid inputs.Source: §3.3 · pp. 6–7

4 IMPLEMENTATION

• NNSmith uses Z3 to obtain valid symbolic graphs, materializes them in topological order with PyTorch functors, and exports models to ONNX.PyTorch also implements the search for inputs and weights producing numerically valid outputs.Source: §4 · p. 7
• NNSmith probes each compiler with single-operator models across data types, then uses the inferred support set to avoid generating unsupported operators.This avoids “Not-Implemented” errors.Source: §4 · p. 7
• When PyTorch and an optimized compiler disagree, NNSmith checks the compiler’s O0 mode; disagreement there supports fault-localizing the bug to compiler optimization.The extra recompilation distinguishes optimization errors from possible converter inconsistencies.Source: §4 · pp. 7–8
• Meta types for unary/binary, reduce, and broadcast operators helped implement 59 of 73 operator specifications in four lines or fewer.Specifications draw on framework documentation and source code, and can be shared across compilers accepting ONNX models.Source: §4 · p. 8
• Attribute binning adds zero and negative padding cases where needed and constrains Slice start/end indices to valid input-tensor ranges.The implementation adds a zero-only bin for Conv2d padding and zero and negative bins for ConstPad, ReplicatePad, and ReflectPad.Source: §4 · p. 8

5.1 Experimental Setup

• Coverage runs lasted four hours; NNSmith and GraphFuzzer used default graphs of 10 nodes, and the comparison included LEMON, GraphFuzzer, and TVM-specific Tzer.Coverage tracked total and unique branches; TensorRT was excluded from coverage evaluation because it is closed-source.Source: §5.1 · p. 8

5.2 End-to-end Coverage Efficiency

• 1.8× on ONNXRuntime and 1.08× on TVM: NNSmith exceeded GraphFuzzer in total branch coverage; pass-only gains were 1.85× and 1.09×, respectively.Coverage was measured over all files and pass-only files.Source: Figure 4 · p. 9, Figure 6 · p. 9, §5.2 · p. 9
• 32.7× on ONNXRuntime and 10.8× on TVM: NNSmith achieved more unique branches than LEMON; against Tzer on TVM, it achieved 1.4× total and 13× unique coverage.For pass-only coverage, NNSmith exceeded Tzer by 123× in unique coverage.Source: Figure 7 · p. 9, Figure 8 · p. 9, §5.2 · p. 9

5.3 Ablation Study

• 2.07× unique operator instances: attribute binning also raised unique branch coverage 2.2× for ONNXRuntime and 1.8× for TVM, while total-coverage gains were up to 2.3%.Operator instances were distinguished by input types and operator attributes.Source: Figure 9 · p. 10, Figure 10 · p. 10, §5.3 · p. 10
• 1.16–1.34× over Sampling: full gradient search improved numerical validity as model size grew, and proxy derivatives consistently improved success within less time.For 10-node models, search took 3.5 ms on average versus 83 ms for generation, reaching a 98% success rate.Source: Figure 11 · p. 10, §5.3 · p. 10

5.4 Bug Study

• 72 new bugs were found; 58 were confirmed and 51 fixed, including 10 PyTorch exporter conversion bugs; 21 of 27 fixed transformation bugs were optimization bugs.The findings included 17 semantic bugs and 55 crash bugs.Source: §5.4 · p. 10
• 49 of 72 bugs could not be triggered by LEMON or GraphFuzzer; NNSmith triggered 38 unique ONNXRuntime crashes and 13 TVM crashes, while GraphFuzzer triggered one in each.The prior approaches restrict how non-shape-preserving operators can be connected.Source: §5.4 · pp. 10–11
• Reported bug patterns include wrong broadcasting and layout transformations, integer type mismatches that stop TVM compilation, and floating-point comparisons that can produce false alarms.For example, a TVM layout rewrite followed by a channel-strided Slice can crash.Source: §5.4 · p. 11

6 RELATED WORK

• The section focuses on prior work most closely related to deep-learning system fuzzing and compiler fuzzing.Source: §6 · p. 11

6.1 DL System Fuzzing

• DL-system fuzzers use model mutation, low-level IR generation, or API testing, but prior approaches restrict model diversity or miss graph-level optimizations.LEMON and GraphFuzzer constrain operator or shape choices; IR-level and API-level techniques also have limits for graph-level optimization bugs.Source: §6.1 · p. 12

6.2 Compiler Fuzzing

• General compiler fuzzers use grammar-based generation, seed mutation, or learned generation, but are difficult to apply directly to graph-level deep-learning compilers.The paper reports NNSmith's superiority over a state-of-the-art IR-level deep-learning compiler fuzzer.Source: §6.2 · p. 12

• NNSmith found 72 new bugs in TVM, TensorRT, ONNXRuntime, and PyTorch, with 58 confirmed or fixed.Source: §7 · p. 12
• NNSmith uses abstract operator models, incremental graph generation, and attribute binning to generate diverse, valid tests.Source: §7 · p. 12
• Gradient search finds inputs that avoid NaN/Inf, while the tool's operator specifications support extension in a few lines and ONNX makes it applicable to systems that support that format.Source: §7 · p. 12

• The artifact includes bug-report links as evidence of real-world bug finding.Source: §A.1 · p. 12
• It provides scripts and code to regenerate the main results, packaged with user-friendly HTML documentation in a pre-built Docker image.Source: §A.1 · pp. 12–13
• Full artifact evaluation requires an X86 CPU with Docker, about 21 hours of machine time, and one hour of manual inspection.Source: §A.1 · p. 13

• The artifact runs on Linux with an X86 CPU and measures C++ source-level branch coverage.Source: §A.2 · p. 13
• It requires approximately 256GB of disk space; Docker installation and image import take a few minutes.Source: §A.2 · p. 13
• The artifact is publicly available under Apache-2 and archived as DOI 10.5281/zenodo.7222132.Source: §A.2 · p. 13

• The hardware dependencies subsection specifies an X86 CPU.Source: §A.3.2 · p. 13
• The hardware requirement is a computer with an X86 CPU and over 256 gigabytes of disk space.Source: §A.3.2 · p. 13
• Docker is listed as the software dependency.Source: §A.3.2 · p. 13

• The Docker image can be pulled directly from Docker Hub using the `docker pull` command.The image is `ganler/nnsmith-asplos23-ae`.Source: §A.4 · p. 13
• Alternatively, the image can be imported locally from the Zenodo repository.Source: §A.4 · p. 13
• The instructions for the local import extract the artifact archive, set an image name, and import the image tar file with Docker.Source: §A.4 · p. 13

A.5 Experiment workflow

• Detailed installation and evaluation steps are available in the artifact HTML documentation and online documentation.Source: §A.5 · p. 13
• The experiment workflow runs NNSmith and baselines on instrumented TVM and ONNXRuntime for four hours each, then visualizes coverage trends and a Venn diagram.Source: §A.5 · p. 13

A.6 Evaluation and expected results

• The artifact documentation includes steps to reproduce the main experiments in § 5.2 and § 5.4, and non-main ablation study results in § 5.3.Source: §A.6 · p. 13