WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models

1 Introduction

• Black- and grey-box fuzzers struggle to satisfy the precise conditions of intricate optimizations, while traditional white-box analysis faces compiler-scale complexity and path explosion.Modern examples include LLVM at 14M lines and TensorFlow at 3.5M lines of code.Source: §1 · p. 3
• The permute_linear_fusion example requires a specific permute-to-linear tensor pattern that ordinary fuzzers lack source-code guidance to construct.LLM-generated tests triggered the optimization and exposed a previously unknown, developer-confirmed high-priority bug.Source: §1 · pp. 3–4

Proposal

• WhiteFox combines an analysis LLM that summarizes optimization-triggering requirements with a generation LLM that synthesizes test programs.Optimization-triggering tests feed back as few-shot examples to guide future generation.Source: §1 · p. 4

Novelty

• WhiteFox uses LLMs to bridge low-level optimization implementations and high-level test programs, extending white-box fuzzing beyond DL compilers.The authors propose adapting the approach to other compilers and complex software systems.Source: §1 · p. 4

Study

• 101 bugs were found in DL compilers, including 92 previously unknown bugs and 70 fixed bugs.WhiteFox also received PyTorch team acknowledgment and is being integrated into its development pipeline.Source: §1 · p. 5
• WhiteFox practiced 2.5x more optimizations than state-of-the-art baselines across PyTorch Inductor, TensorFlow-XLA, and TensorFlow Lite.The implementation uses GPT4 for analysis and StarCoder for test generation.Source: §1 · pp. 4–5

2.1 Compiler Fuzzing

• Grammar- and mutation-based fuzzers generate valid or mutated compiler inputs, but DL-model fuzzing additionally requires valid compositions of tensor operations.Existing techniques use curated models, shape-preserving operators, or manually and automatically defined operator constraints.Source: §2.1 · p. 5
• Grey-box fuzzers use coverage feedback to explore deeper behaviors, including IR-level testing and joint mutation of IR files and optimization passes.Examples include PolyGlot, Tzer, and GrayC.Source: §2.1 · p. 5
• Competitive white-box compiler fuzzing remains rare because compiler-scale source analysis causes path explosion and binary-oriented hybrid fuzzing misses source-level semantics.These limitations hinder detailed modeling of compiler behavior.Source: §2.1 · pp. 5–6

2.2 LLM-based Testing

• LLM-based testing research includes unit-test generation systems such as TeCo, TestPilot, MuTAP, ChatUniTest, CodaMosa, and ChatGPT-SBST.These systems demonstrate growing use of LLMs for generating tests across software-testing settings.Source: §2.2 · p. 6
• LLM-based unit-test generators target individual modules, whereas WhiteFox targets compiler testing at the entire system level using optimization source code as guidance.WhiteFox also uses automated compiler-fuzzing oracles such as differential testing.Source: §2.2 · p. 6
• General LLM fuzzers such as TitanFuzz and Fuzz4All often neglect source-code implementation and therefore tend to focus on compiler front ends.WhiteFox instead uses low-level optimization implementations and is reported to outperform TitanFuzz.Source: §2.2 · p. 6

3 Design

• WhiteFox analyzes optimization implementations, generates high-level test programs, feeds successful triggers back into generation, and checks optimized versus minimally optimized executions for bugs.The framework combines requirement summarization, test generation, feedback-guided refinement, and test oracles.Source: §3 · pp. 6–7, §3.4 · pp. 12–13

3.1 Requirement Summarization

• Directly prompting on optimization source is difficult because compiler code is lengthy, low-level, and dominated by implementation details unrelated to high-level triggering inputs.These properties make source-code understanding and mapping to user-visible inputs challenging for LLMs.Source: §3.1 · p. 7
• WhiteFox summarizes low-level optimization logic into mixed natural-language and pseudo-code requirements that map internal compiler constructs to high-level APIs.For example, TensorFlow Lite’s TFL::ConcatenationOp is represented as the public tf.concat operation.Source: §3.1 · pp. 7–8

3.2 Test Generation

• The generation agent uses optimization requirements and few-shot examples to produce valid high-level tests whose code patterns correspond to the stated trigger conditions.Examples clarify both the required public API form and the semantic meaning of operations such as permute.Source: §3.2 · pp. 9–10

3.3 Feedback Loop

• Successful triggering tests become subsequent few-shot examples, while Thompson Sampling balances exploration and exploitation when selecting examples with different triggering rates.When no trigger has been found, WhiteFox continues using the initial prompt until one is discovered.Source: §3.3 · pp. 11–12

3.4 Test Oracle

• WhiteFox treats optimized-versus-unoptimized output discrepancies, compilation-status differences, and compile- or execution-time crashes as bug candidates.Differential testing compares outputs for the same inputs across optimization modes, while crash detection captures failures in compilation and execution.Source: §3.4 · pp. 12–13

4 Implementation

• WhiteFox identifies optimization functions and auxiliary helpers from compiler source directories, then instruments collected functions to log activated optimization passes.Keyword matching helps locate relevant code, while auxiliary calls can expose activation conditions.Source: §4 · p. 13
• GPT4 performs one-time requirement analysis, StarCoder generates batches of tests iteratively, and compiler-specific few-shot prompts require only one manually authored example per compiler.The model choices balance broad reasoning ability against affordable continuous generation.Source: §4 · pp. 13–14

5.1 Research Questions

• The study asks whether WhiteFox outperforms state-of-the-art fuzzers, whether its components are effective, and whether it detects real-world bugs.Source: §5.1 · p. 14

5.2 Experimental Setup

• The evaluation compares WhiteFox with TitanFuzz and NNSmith on PyTorch Inductor, TensorFlow Lite, and TensorFlow-XLA.TensorFlow-XLA optimizations longer than 400 lines are excluded because of the LLM context-window limit.Source: §5.2 · pp. 14–15

5.3 Metrics

• Optimization triggering, triggering tests, and detected bugs are primary metrics because bugs can manifest only when the relevant optimization is activated.Source: §5.3 · p. 15
• Code coverage is additionally measured in Python for PyTorch Inductor and C++ for TensorFlow Lite and TensorFlow-XLA.Source: §5.3 · p. 15

5.2 Experimental Setup

• WhiteFox is compared under default settings and a reduced-test WhiteFox-Mini configuration to account for NNSmith’s shorter execution time.Table 2 includes both requirement/test generation time and test-execution time.Source: Table 2 · p. 16

6.1 Comparison with Prior Work

• Up to 8.2x more optimizations are triggered overall; in PyTorch Inductor, WhiteFox triggers 41 of 61 optimizations versus at most 5 for baselines.WhiteFox also uses less time than every technique except NNSmith, while WhiteFox-Mini exceeds NNSmith with less time.Source: §6.1 · p. 16
• WhiteFox outperforms all baselines except TitanFuzz on TensorFlow-XLA, where it triggers two fewer optimizations but uniquely covers four optimizations TitanFuzz misses.The targeted TensorFlow-XLA optimizations are relatively simple after filtering, and many are common model patterns.Source: §6.1 · pp. 16–17
• WhiteFox covers every baseline-triggered optimization on TensorFlow Lite and adds three unique optimizations, while adding 34 unique optimizations beyond the seven baseline optimizations on PyTorch Inductor.For TensorFlow-XLA, baselines trigger 26 optimizations, including 20 covered by WhiteFox.Source: §6.1 · p. 17
• Over 24 hours, WhiteFox achieves the strongest optimization-triggering results and up to 19.9% higher code coverage than baselines on PyTorch Inductor and TensorFlow-XLA.TensorFlow Lite code coverage is 5.9% below TitanFuzz, but WhiteFox performs better on bug finding.Source: §6.1 · p. 17

6.2.1 Requirement Generation & Test Generation.

• Requirement descriptions outperform directly supplying implementation code, with WF-Mix generating 1.74x more triggering tests and seven additional triggered optimizations than WF-Impl.Implementation source code contains redundant, unrelated information in a low-level format.Source: §6.2.1 · p. 18
• Mixed natural-language and pseudo-code requirements trigger the most optimizations and tests among the evaluated requirement formats.Natural language preserves more triggering information, whereas pseudo-code is easier for the generation model to correlate with test programs.Source: §6.2.1 · p. 18
• Using GPT4 for analysis outperforms StarCoder, while StarCoder-based requirements still outperform direct implementation-code prompting.The results support separating requirement analysis from test generation in the dual-model design.Source: §6.2.1 · p. 18

6.2.2 Feedback Loop.

• The feedback ablation compares WhiteFox, WF-Naive, and WF-No-Feedback using triggering-test counts and code coverage.The feedback loop supplies optimization-triggering examples for subsequent generation, while Thompson Sampling selects among examples.Source: §6.2.2 · pp. 18–19
• WhiteFox’s feedback loop increases triggering tests by 2.6x over no feedback, while Thompson Sampling adds 1.3x over uniform example selection.WhiteFox outperforms the other variants on 32 of 41 triggered optimizations and achieves higher code coverage than WF-Naive.Source: Figure 6 · p. 18, §6.2.2 · p. 19, Table 5 · p. 19

6.3 Bug Finding

• 101 bugs were detected across PyTorch Inductor, TensorFlow-XLA, and TensorFlow Lite; 92 were previously unknown and 70 were fixed.Source: §6.3 · p. 19
• 68 of 79 PyTorch Inductor bugs were undetectable by the baselines, while baselines found only 1 of 22 TensorFlow bugs.Source: §6.3 · p. 19, §6.3.1 · p. 19

6.3.1 Bug Analysis.

• 47 of 68 fixed bugs were repaired in PyTorch Inductor optimization code, but only 3 were covered by TitanFuzz or NNSmith.Source: §6.3.1 · p. 20

6.3.2 Bug Characteristics.

• Detected defects included crashes, miscompilations, failed or incorrectly passed optimizations, and five out-of-bounds read vulnerabilities alongside six crash bugs exploitable for denial of service.Source: §6.3.2 · p. 20

6.3.4 Bug Examples.

• An attention-module optimization rearranged tensor layouts incorrectly, producing wrong outputs; the high-priority bug was subsequently fixed.Source: §6.3.4 · p. 20
• WhiteFox uniquely found binary-ReLU bugs producing negative outputs or bfloat16 crashes, and a TensorFlow Lite reshape bug caused by violating an inverse-operation assumption.Source: §6.3.4 · p. 21
• A TensorFlow-XLA optimization omitted embedding-index validation, creating an out-of-bounds read when index 64 exceeded the maximum valid index 63.Source: §6.3.4 · pp. 21–22

7.1 Real-world Impact

• PyTorch requested WhiteFox integration, and its extension to 38 new optimizations found 14 confirmed bugs in the latest Inductor version.Source: §7.1 · p. 22

7.2 Generality: Case Study on LLVM

• On LLVM, WhiteFox triggered 6.5x more optimizations than baselines with less time cost, demonstrating cross-compiler applicability.Source: §7.2 · p. 22
• WhiteFox detected six LLVM bugs, including a confirmed crash exploitable for denial of service in Just-in-Time-enabled systems.Source: §7.2 · p. 23

7.3 Limitations and Future Work

• For arbitrary compiler functions, the mapping from high-level inputs to low-level implementation may be unclear, limiting straightforward adaptation beyond optimization testing.Source: §7.3 · p. 23
• Traditional fuzzers could reduce the higher computational cost of invoking smaller LLMs by using LLM-produced generation or mutation rules.Source: §7.3 · p. 23

• WhiteFox combines an analysis LLM that summarizes optimization-triggering patterns with a generation LLM that synthesizes test programs, finding 101 DL-compiler bugs, including 92 previously unknown and 70 fixed.The approach also adapts to conventional C/C++ compilers.Source: §8 · p. 23