Source-linked AI summary
Fuzz4All: Universal Fuzzing with Large Language Models
Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, Lingming Zhang
TL;DR
Existing fuzzers are often tied to particular languages, systems, and feature sets, limiting their reuse as targets evolve. Fuzz4All uses LLM-based autoprompting and an iterative fuzzing loop to generate diverse inputs across languages and systems. Across six languages and nine SUTs, it improves coverage over existing fuzzers and finds 98 bugs, including 64 confirmed previously unknown.
Problem
Existing fuzzers are often tightly coupled to specific languages and systems and cover restricted feature sets, limiting support for evolving targets.
Method
Fuzz4All distills user-provided documentation, examples, or specifications into prompts and iteratively updates them with generated examples and strategies for LLM-based fuzzing.
Results
36.8% average coverage improvement over the previous state of the art was achieved across six languages and nine SUTs, with 98 bugs detected and 64 confirmed previously unknown.
Takeaways & Limitations
Universal fuzzing can target many programming languages, systems, and selected features while supporting general and targeted fuzzing.
Takeaways & Limitations
The distillation LLM may hallucinate made-up or inaccurate information when generating initial inputs.
Abstract
from arXiv · showhide
Fuzzing has achieved tremendous success in discovering bugs and vulnerabilities in various software systems. Systems under test (SUTs) that take in programming or formal language as inputs, e.g., compilers, runtime engines, constraint solvers, and software libraries with accessible APIs, are especially important as they are fundamental building blocks of software development. However, existing fuzzers for such systems often target a specific language, and thus cannot be easily applied to other languages or even other versions of the same language. Moreover, the inputs generated by existing fuzzers are often limited to specific features of the input language, and thus can hardly reveal bugs related to other or new features. This paper presents Fuzz4All, the first fuzzer that is universal in the sense that it can target many different input languages and many different features of these languages. The key idea behind Fuzz4All is to leverage large language models (LLMs) as an input generation and mutation engine, which enables the approach to produce diverse and realistic inputs for any practically relevant language. To realize this potential, we present a novel autoprompting technique, which creates LLM prompts that are wellsuited for fuzzing, and a novel LLM-powered fuzzing loop, which iteratively updates the prompt to create new fuzzing inputs. We evaluate Fuzz4All on nine systems under test that take in six different languages (C, C++, Go, SMT2, Java and Python) as inputs. The evaluation shows, across all six languages, that universal fuzzing achieves higher coverage than existing, language-specific fuzzers. Furthermore, Fuzz4All has identified 98 bugs in widely used systems, such as GCC, Clang, Z3, CVC5, OpenJDK, and the Qiskit quantum computing platform, with 64 bugs already confirmed by developers as previously unknown.
1 INTRODUCTION
Traditional fuzzers are often tied to particular languages, systems, and feature sets, limiting reuse and support for evolving targets. Fuzz4All uses LLM-driven autoprompting and iterative generation to fuzz diverse systems and languages without instrumentation.
- Programming-language SUTs such as compilers and constraint solvers are important because bugs in them can affect downstream applications.
- Traditional fuzzers are tightly coupled to target languages and systems, making their implementation effort difficult to reuse across targets.Csmith exceeds 80k lines of code, while Syzkaller contains tens of thousands of handcrafted rules.
- Traditional fuzzers may lose effectiveness as languages and systems evolve, especially when new features are added.Csmith supports only a limited set of features up to C++11, while C++ has evolved substantially since then.
- Generation- and mutation-based fuzzers cover restricted portions of the input space because of limited grammars, semantic rules, mutation operators, or seed requirements.
- Fuzz4All distills user-provided documentation, examples, or specifications into prompts, then iteratively updates them with examples and generation strategies.The generated inputs are passed to the SUT and checked with a user-provided oracle, such as crash detection.
- 36.8% average coverage improvement over the previous state of the art was achieved across six languages and nine SUTs, alongside 98 detected bugs.64 bugs were confirmed by developers as previously unknown, and the approach requires no SUT instrumentation.
2 BACKGROUND AND RELATED WORK
Prior fuzzing research includes language-specific generators, general-purpose mutation systems, learning-based models, and LLM-based tools. Fuzz4All differs by applying black-box LLM prompting across many languages and systems rather than relying on language-specific models or handcrafted patterns.
- Large Language Models: LLMs can be prompted rather than fine-tuned, avoiding model-weight updates while using task descriptions and optional examples.
- Fuzz4All: Fuzz4All uses GPT4-generated prompts scored by a fuzzing-specific objective and applies black-box prompting across many programming languages.
- Fuzzing: Traditional generation-based fuzzers synthesize inputs from grammars and semantic knowledge, while mutation-based systems transform high-quality seeds.
- General-purpose fuzzing: General-purpose fuzzers such as AFL and libFuzzer prioritize byte-level mutations using coverage-oriented genetic algorithms, without understanding the target language.
- Learning-based fuzzing: Learning-based fuzzers train models to synthesize inputs for domains including PDF parsers, OpenCL, and C.
- LLM-based fuzzing: Prior LLM-based fuzzers target specific libraries, such as deep-learning libraries, using handcrafted prompts and mutation patterns.
3 FUZZ4ALL APPROACH
Fuzz4All accepts arbitrary descriptions of a target and distills them into an initial prompt before repeatedly generating and revising inputs. This design supports general and targeted fuzzing across programming-language SUTs.
- Fuzz4All accepts documentation, code snippets, specifications, or combinations of modalities describing the desired fuzzing inputs.
- An autoprompting stage distills potentially long user input into a concise, informative prompt for the generation LLM.
- Fuzz4All is implemented with GPT4 and StarCoder as distillation and generation LLMs, respectively.
- The fuzzing loop uses the selected prompt to sample inputs and updates it with previously generated examples to avoid repetitive outputs.
3.1 Autoprompting
Autoprompting converts arbitrary user material into a fuzzing prompt, generates candidate prompts at different temperatures, and selects the candidate whose outputs contain the most valid code.
- Autoprompting accepts technical documentation, examples, specifications, or mixed inputs and distills them because raw inputs may be redundant or irrelevant.
- The distillation LLM is instructed to summarize the target’s usage and functionality concisely, producing candidate prompts from the user input.
- Fuzz4All first obtains a high-confidence candidate with temperature 0, then samples higher-temperature candidates for diversity.
- Each candidate prompt is evaluated through a small fuzzing experiment that generates code snippets and counts unique snippets accepted by the SUT.
- The highest-scoring candidate becomes the initial prompt for the fuzzing campaign.
- For std::expected, autoprompting reduced the source material from 498 words to a 214-word targeted description.The corresponding character count decreased from 3,262 to 1,410.
- Fuzz4All’s black-box autoprompting evaluates prompts on downstream valid-code generation rather than an approximate proxy score.
3.2 Fuzzing Loop
Fuzz4All iteratively augments an initial prompt with valid generated examples and generation strategies to produce diverse inputs. It tests each input against a user-defined oracle until the fuzzing budget is exhausted.
- Fuzzing Loop: The fuzzing loop aims to avoid repeated LLM outputs by continuously updating the original prompt with examples and generation strategies.The strategies direct the model to synthesize, mutate, or semantically transform fuzzing inputs.
- Initialization: The first generation uses the generate-new strategy because no prior fuzzing examples are available.The initial prompt is augmented with the strategy before the generation LLM produces the first batch.
- Iterative Generation: Each subsequent iteration randomly selects a valid prior input and one of three strategies to create new fuzzing inputs.The selected strategy instructs the model how to use the example, including mutation or semantic transformation.
- Oracle-Based Testing: The loop passes every generated input to the SUT and records a bug when the user-defined oracle identifies unexpected behavior.The loop continues until the fuzzing budget is exhausted.
4 EXPERIMENTAL DESIGN
The evaluation tests Fuzz4All across six input languages and nine SUTs against language-specific generation- and mutation-based baselines. Experiments measure coverage and generation behavior under repeated fuzzing campaigns, with user-provided documentation defining target inputs and features.
- Research Questions: The study evaluates Fuzz4All on six languages and nine SUTs while addressing comparisons, targeted fuzzing, component effects, and real-world bug discovery.The research questions cover baseline comparison, targeted fuzzing, ablations, and bugs found in practice.
- Implementation: Fuzz4All uses an 872-LoC Python implementation with GPT-4 for autoprompting and StarCoder for fuzzing-input generation.Autoprompting distills user inputs into prompts, while StarCoder generates inputs using a batch size of 30 and maximum output length of 1,024.
- Targets and Baselines: Table 1 organizes the evaluated languages and SUTs with their corresponding baseline tools and coverage-measurement versions.Coverage comparisons use one SUT per language; other fuzzing experiments target nightly releases.
- Targets and Baselines: The baselines span specialized fuzzers for C/C++, SMT2, Go, Java, and related target systems, including Csmith, GrayC, YARPGen, TypeFuzz, go-fuzz, and Hephaestus.The evaluation also targets GCC, Clang, Z3, CVC5, javac, and Qiskit with documentation or examples supplied as Fuzz4All inputs.
- Campaign Protocol: RQ1 uses 24-hour campaigns repeated five times, while later RQs use 10,000 generated inputs and four repetitions for ablation studies.Runs use a 64-core workstation with 256 GB RAM and four RTX A6000 GPUs, with one GPU per fuzzing run.
5 RESULTS
Across six languages and nine SUTs, Fuzz4All achieves the highest final coverage against the evaluated baselines and continues discovering inputs covering new code late in 24-hour campaigns. This result holds despite fewer and less-valid generated inputs, while targeted documentation enables strong coverage on Qiskit.
- Coverage over Time: 36.8% average coverage improvement over the top-performing baselines makes Fuzz4All the highest-coverage approach across all evaluated targets.Figure 4 summarizes average coverage across five runs, with shaded regions showing minimum and maximum coverage.
- Coverage over Time: Fuzz4All continues finding inputs that cover new code near the end of 24-hour campaigns, unlike baselines that reach a coverage plateau.The paper attributes this pattern to iterative prompt updates with new examples and generation strategies, while presenting the mechanism as a hypothesis.
- Generation Validity, Number, and Coverage: Fuzz4All generates 56.0% fewer valid inputs and 43.0% fewer total inputs than traditional fuzzers, yet obtains 36.8% higher average coverage.The reported trade-off is associated with greater program diversity and GPU-inference bottlenecks.
- Generation Validity, Number, and Coverage: Qiskit fuzzing achieves a +75.6% coverage improvement over the state-of-the-art fuzzer despite low validity rates for quantum-program inputs.User-provided documentation helps Fuzz4All generate inputs using quantum library APIs.
5.2 RQ2: Effectiveness of Targeted Fuzzing
Fuzz4All supports targeted fuzzing by using feature-specific documentation as input, producing inputs that directly exercise the selected features. Targeting achieves high feature hit rates, whereas general fuzzing is inefficient for rare or newly added features.
- 83.0% average hit rate shows that targeted fuzzing inputs directly use the selected features.
- Related features can also receive moderately high cross-feature hit rates, while unrelated features are hit less often.The paper illustrates this with the related C keywords typedef and union compared with goto.
- 96.0% average hit-rate reduction makes general fuzzing extremely inefficient for targeting specific features.In Qiskit, general fuzzing achieved a 0% hit rate for the three target features.
- Feature-specific documentation enables Fuzz4All to generate inputs using newly added features that general fuzzing may rarely produce.The paper identifies this capability as valuable for testing novel features or components of a SUT.
5.3 RQ3: Ablation Study
The ablation study evaluates how autoprompting and the fuzzing loop affect coverage and validity. Results favor distilled prompts, selected examples, and the full set of generation strategies.
- Autoprompting: No-input prompting achieves the lowest coverage across all studied languages.Without feature information, the LLM generates simple, highly valid snippets but is less effective at covering the SUT.
- Autoprompting: Autoprompting improves the initial prompt by distilling user inputs into a form designed for LLM generation.Raw documentation can contain irrelevant information and lower both validity and coverage.
- Fuzzing loop: 8.0% repeated inputs without examples falls to 4.7% with the full Fuzz4All approach.Selecting examples avoids repeatedly sampling similar inputs from the same initial-prompt distribution.
- Fuzzing loop: The full Fuzz4All loop achieves the highest coverage across all SUTs.Its additional semantic-equiv and mutate-existing strategies provide useful instructions to the generation LLM beyond generate-new.
5.4 RQ4: Bug Finding
Fuzz4All detects bugs across its nine studied systems under test, including compiler crashes, runtime failures, and invalid-output behavior. Several examples were confirmed and fixed by developers.
- 98 bugs were detected across nine studied systems, including 64 confirmed by developers as previously unknown.The results demonstrate bug-finding effectiveness across languages and SUTs.
- A GCC bug involving noexcept(x) and std::optional caused an internal compiler error on valid code.Developers confirmed and fixed the bug and added a modified submitted snippet to GCC’s official test suite.
- A Clang bug caused a segmentation fault when throw inside decltype was evaluated before the return type.The generated code used unusual trailing-return-type syntax and was invalid, but still exposed a compiler bug.
- Targeted fuzzing found a Go standard-library bug where nil input caused a segmentation fault instead of a useful failure message.Developers confirmed and fixed the bug, which go-fuzz could not find because it requires manually written inputs.
- Qiskit’s QASM exporter silently produced invalid output when a register name conflicted with a QASM operation name.Reimporting the ambiguous output led to a crash.
6 THREATS TO VALIDITY
The evaluation’s main external threat is the choice of targets, while LLM data shift and hallucinated distilled prompts threaten reproducibility and input quality.
- Reapplying Fuzz4All with the exact StarCoder checkpoint may degrade effectiveness as training-data distribution shifts.Autoprompting with up-to-date documentation or example code can mitigate this threat by providing current information.
- The distillation LLM may hallucinate made-up or inaccurate information when generating initial prompts.The authors identify this as a common limitation of LLM-based pipelines.
- The evaluation uses nine SUTs across six languages and repeats 24-hour campaigns five times to address target selection and run variance.
7 CONCLUSION
Fuzz4All is a universal LLM-based fuzzer for arbitrary programming-language SUTs, improving coverage across nine SUTs and six languages while finding previously unknown bugs.
- Fuzz4All leverages LLMs to support general and targeted fuzzing of arbitrary SUTs across multiple programming languages.
- Fuzz4All uses autoprompting to summarize user-provided inputs and iteratively updates prompts with code examples and generation strategies.
- 98 bugs were detected, including 64 confirmed by developers as previously unknown.