Source-linked AI summary
UNIFUZZ: A Holistic and Pragmatic Metrics-Driven Platform for Evaluating Fuzzers
Yuwei Li, Shouling Ji, Yuan Chen, Sizhuang Liang, Wei-Han Lee, Yueyao Chen, Chenyang Lyu, Chunming Wu, Raheem Beyah, Peng Cheng, Kangjie Lu, Ting Wang
TL;DR
Comparing fuzzers is difficult because evaluations use inconsistent benchmarks, metrics, and environments. The paper develops UNIFUZZ, an open-source platform integrating usable fuzzers, real-world benchmarks, and complementary metrics, and finds that no fuzzer consistently dominates while metric choice and evaluation factors affect conclusions. These results support more comprehensive and rigorous fuzzing evaluation.
Problem
Inconsistent benchmarks, performance metrics, and evaluation environments make it difficult to compare fuzzers reliably and identify promising fuzzing techniques.
Method
UNIFUZZ combines 35 fuzzers, 20 real-world benchmark programs, six metric categories, usability testing, and crash-analysis support in an open-source evaluation platform.
Results
No fuzzer consistently outperforms the others across tested programs, and performance conclusions vary with metrics, instrumentation methods, and crash analysis tools.
Takeaways & Limitations
Reliable fuzzer assessment requires pragmatic benchmarks, comprehensive metrics, and rigorous attention to evaluation factors.
Takeaways & Limitations
The paper notes that the six metric categories and their concrete metrics require further research, with individual metrics such as CVSS having limitations.
Abstract
from arXiv · showhide
A flurry of fuzzing tools (fuzzers) have been proposed in the literature, aiming at detecting software vulnerabilities effectively and efficiently. To date, it is however still challenging to compare fuzzers due to the inconsistency of the benchmarks, performance metrics, and/or environments for evaluation, which buries the useful insights and thus impedes the discovery of promising fuzzing primitives. In this paper, we design and develop UNIFUZZ, an open-source and metrics-driven platform for assessing fuzzers in a comprehensive and quantitative manner. Specifically, UNIFUZZ to date has incorporated 35 usable fuzzers, a benchmark of 20 real-world programs, and six categories of performance metrics. We first systematically study the usability of existing fuzzers, find and fix a number of flaws, and integrate them into UNIFUZZ. Based on the study, we propose a collection of pragmatic performance metrics to evaluate fuzzers from six complementary perspectives. Using UNIFUZZ, we conduct in-depth evaluations of several prominent fuzzers including AFL [1], AFLFast [2], Angora [3], Honggfuzz [4], MOPT [5], QSYM [6], T-Fuzz [7] and VUzzer64 [8]. We find that none of them outperforms the others across all the target programs, and that using a single metric to assess the performance of a fuzzer may lead to unilateral conclusions, which demonstrates the significance of comprehensive metrics. Moreover, we identify and investigate previously overlooked factors that may significantly affect a fuzzer's performance, including instrumentation methods and crash analysis tools. Our empirical results show that they are critical to the evaluation of a fuzzer. We hope that our findings can shed light on reliable fuzzing evaluation, so that we can discover promising fuzzing primitives to effectively facilitate fuzzer designs in the future.
1 Introduction
UNIFUZZ addresses unreliable and incomparable fuzzer evaluations with an open-source platform combining usable fuzzers, real-world benchmarks, and comprehensive metrics. Its evaluations show that no fuzzer consistently dominates and that instrumentation, crash analysis, and metric choice materially affect conclusions.
- Evaluation gap: Fuzzing research faces unreliable comparisons because experiments, benchmarks, environments, and performance metrics vary across studies.Prior evaluations may use insufficient repetitions, different environments, and self-chosen target programs.
- Platform: UNIFUZZ integrates 35 fuzzers, 20 real-world benchmark programs, and six categories of performance metrics into an open-source evaluation platform.The platform tests fuzzer usability, fixes more than 15 flaws, and provides deployment support and benchmark side information.
- Metrics: Its metrics cover unique-bug quantity, bug quality, discovery speed, discovery stability, coverage, and overhead.These categories are intended to assess fuzzer performance from complementary perspectives.
- Findings: Evaluations of eight prominent coverage-based fuzzers find that no fuzzer outperforms the others on every tested benchmark program.This differs from conclusions reported in individual fuzzing papers and suggests that evaluation subjectivity and bias may matter.
- Findings: Using a single metric can produce unilateral conclusions, while instrumentation methods and crash analysis tools can significantly affect fuzzer assessment.The paper therefore argues for comprehensive metrics and more rigorous, precise experiments.
2 Motivation of UNIFUZZ
UNIFUZZ is motivated by practical weaknesses in existing fuzzer evaluations: difficult-to-use implementations, inadequate benchmarks, and incomplete metrics. These weaknesses can introduce bias, obscure meaningful comparisons, and make reliable assessment costly.
- Usability: Existing fuzzers often have usability problems, including difficult setup, implementation flaws, incorrect crash judgments, and abnormal fuzzing behavior.These issues can affect practical performance and reproducibility.
- Benchmarks: Pragmatic benchmarks should resemble real-world software, cover diverse functionalities and vulnerabilities, remain feasible to evaluate, and provide usable setup information.The benchmark discussion emphasizes representativeness, diversity, manageable difficulty, and ease of use.
- Benchmarks: Synthetic benchmarks can reward understanding of simple artificial bug-injection patterns, so strong performance on them may not transfer to real-world programs.This creates a risk that benchmark results overstate practical effectiveness.
- Benchmarks: Existing real-world benchmarks are also limited by insufficient standard coverage, self-selected programs, and inconvenient bug validation.These limitations can make fair, comprehensive comparisons difficult.
- Metrics: The common metrics of unique crashes, unique bugs, and coverage do not fully capture fuzzer performance, and unique crashes may misrepresent unique bugs.Bug quality and other dimensions therefore require additional measurement.
3 Design of UNIFUZZ
UNIFUZZ combines usable fuzzers, pragmatic real-world benchmarks, crash and CVE analysis, and six complementary metric categories into a comprehensive evaluation platform.
- Platform overview: UNIFUZZ integrates usable fuzzers, pragmatic benchmarks, and performance metrics as its three main components.The platform is designed as an open-source evaluation system for fuzzers.
- Usable fuzzers: The platform incorporates 35 fuzzers, tests their usability, and reports more than 15 implementation flaws to developers.Dockerfiles support installation and deployment, while some reported flaws were fixed and released.
- Pragmatic benchmarks: UNIFUZZ constructs a benchmark of 20 real-world programs spanning six functionality types and varied vulnerability types.Programs are selected from fuzzing-related evaluations and intended to be realistic, comprehensive, practical, and convenient to use.
- Crash analysis: Crash triage uses output analysis, stack traces, and stack hashing, while ASan is prioritized and GDB supplements detection.The stack-hash approach is practical but depends on selecting a deduplication depth that can affect false positives and negatives.
- CVE matching: A CVE keywords database structures vulnerability information and supports initial crash-to-CVE matching followed by manual verification.The database records CVE IDs, vulnerability types, vulnerable functions and files, stack traces, and generating tools.
- Performance metrics: UNIFUZZ proposes six metric categories: unique-bug quantity, bug quality, finding speed, finding stability, coverage, and overhead.Bug quality considers severity and difficulty of discovery, while quantity metrics account for repeated randomized experiments.
4 Evaluations of the State-of-the-art Fuzzers
The evaluation compares eight prominent coverage-based fuzzers on UNIFUZZ’s real-world programs under controlled, resource-bounded experiments, while documenting exclusions caused by tool constraints.
- Fuzzers: Experiments evaluate eight prominent coverage-based fuzzers selected from UNIFUZZ.The selection includes AFL, AFLFast, Angora, Honggfuzz, MOPT, QSYM, T-Fuzz, and VUzzer64.
- Programs: The study uses 20 real-world programs, additionally uses LAVA-M, and validates bugs with ASan and GDB for all fuzzers.Programs are compiled according to each fuzzer’s instrumentation or compilation requirements.
- Initial seeds: Each benchmark program uses the same initial seeds across fuzzers, with up to 100 randomly selected seeds after format and tool requirements are checked.Seeds are collected from the Internet before filtering and random selection.
- Environment: Experiments run on identical servers with one CPU core, 2 GB RAM, and 1 GB swap space allocated per fuzzer.Memory limits increase to 8 GB when a fuzzer cannot run successfully with 2 GB.
- Evaluation scope: T-Fuzz on ffmpeg and VUzzer64 on sqlite3 are excluded because of excessive memory use and unsupported stdin input, respectively.These exclusions limit the included result cases for the affected fuzzer-program pairs.
4.2 Quantity of Unique Bugs
UNIFUZZ evaluates the quantity of unique bugs using deduplication, repeated trials, and statistical comparisons. Results show that no fuzzer dominates across all programs, while outcomes can depend strongly on the benchmark and analysis method.
- Unique bugs are deduplicated using the top three ASan stack-trace functions plus vulnerability type, with GDB and other tools supplementing cases ASan cannot detect.
- No fuzzer outperforms the others on all programs; across 20 real-world programs, QSYM leads on five, while Angora, Honggfuzz, and MOPT each lead on three.AFL leads on tiffsplit, whereas AFLFast, T-Fuzz, and VUzzer64 lead on no target program.
- 30 repetitions are used to quantify unique-bug results, with p values and ˆA12 scores reported for statistical significance and effect size.Mann-Whitney U testing uses AFL as the baseline; p < 0.05 indicates significance and ˆA12 ≥ 0.71 indicates a large effect size.
- CrashAnalyzer does not produce new crashes for jhead, flvmeta, or wav2swf, and T-Fuzz does not generate transformed binaries for 17 of 20 tested programs.
- Statistical comparisons do not establish universal superiority: no remaining fuzzer significantly outperforms AFL on all real-world programs.Some fuzzers significantly outperform AFL on all four LAVA-M programs, while MOPT does so on 17 real-world programs.
4.3 The Quality of Bugs
UNIFUZZ assesses bug quality through severity and rareness, recognizing that equal bug counts can conceal differences in danger or difficulty. Its results show distinct strengths among fuzzers under these quality measures.
- Bug quality is defined by severity and rareness, providing dimensions beyond the number of unique bugs.
- Severity of Bugs: CVSS assigns numerical severity scores to CVEs, with scores of 7.0 or higher classified as highly severe.The evaluation manually checks initial CVE matches before associating CVEs with CVSS scores.
- Severity of Bugs: MOPT detects EXPLOITABLE bugs on nine programs, while Angora, Honggfuzz, and QSYM lead on three, five, and three programs, respectively.VUzzer64 detects EXPLOITABLE bugs on only two programs.
- Rareness of Bugs: QSYM discovers 262 unique rare bugs across all real-world programs, followed by MOPT with 90 and Angora with 56.The rare-bug metric is relative to the set of compared fuzzers rather than an absolute quantity.
4.4 Speed of Finding Bugs
Speed is measured by tracking the average number of unique bugs found over time across 30 repetitions. The results show that rankings can change over time and that similar final bug counts may hide substantial speed differences.
- Figure 3 reports the average number of unique bugs found over time in 30 repetitions to measure bug-finding speed.
- No fuzzer wins on speed across all programs, and performance comparisons can reverse as fuzzing time increases.On sqlite3, MOPT finds fewer bugs than QSYM early but more after 10 hours.
- Angora, MOPT, and QSYM find 2.5, 2.6, and 2.4 average unique bugs on mp42aac within 24 hours, yet MOPT finds them faster.
4.5 Stability of Finding Bugs
Stability is measured with the relative standard deviation of unique-bug counts across 30 repetitions. The results show that fuzzing outcomes are variable and that stability differs by fuzzer and program.
- Figure 4 uses relative standard deviation across 30 repetitions, where lower RSD indicates better stability.
- All fuzzers are not consistently stable in finding bugs, reflecting the randomness of fuzzing and the importance of repeated experiments.
- Angora and T-Fuzz achieve lower RSD, whereas AFL and Honggfuzz achieve higher RSD among the seven fuzzers.The stability of a fuzzer also varies across programs.
4.6 Coverage
UNIFUZZ measures coverage uniformly across fuzzers by replaying coverage-improving test cases on the same instrumented binaries. Line coverage is only weakly correlated with unique-bug counts.
- Coverage: UNIFUZZ standardizes coverage comparison by calculating line coverage with the same instrumentation and granularity for different fuzzers.It saves test cases that increase coverage and uses afl-cov to calculate line coverage.
- Coverage: The coverage analysis reports line coverage on the real-world programs rather than relying on each fuzzer’s native coverage tracking.The approach addresses differences between compile-time, binary, edge, and basic-block instrumentation.
- Coverage: Most fuzzers have Spearman correlation coefficients below 0.60 between unique bugs and line coverage.This indicates that the relationship between the two measures is not strong.
4.7 Overhead
UNIFUZZ evaluates overhead through memory consumption and finds substantial variation among fuzzers, including much higher usage for T-Fuzz and Angora on specific programs.
- Overhead: 24.6 MB, 22.2 MB, and 51.8 MB are the average memory consumptions of AFL, AFLFast, and MOPT, respectively.These three fuzzers consume less memory holistically than the other evaluated fuzzers.
- Overhead: 1,082 MB is T-Fuzz’s average memory consumption, almost 50 times that of AFLFast and the highest among the fuzzers.This comparison highlights the scale of overhead differences across fuzzers.
- Overhead: More than 7 GB of memory is used by Angora when testing pdftotext, while its consumption on other programs is below 2 GB.Memory consumption therefore also varies substantially across target programs for the same fuzzer.
5 Further Analysis
UNIFUZZ examines instrumentation and crash-analysis tools as evaluation factors. The results show that compilation choices and analysis tools can change which bugs are observed and how fuzzers compare.
- Instrumentation: Different instrumentation methods can affect fuzzing evaluation because fuzzers produce binaries with different characteristics.AFL and Angora use compiler wrappers, whereas VUzzer uses Intel PIN for binary instrumentation.
- Instrumentation: Compile-time optimization can skip an erroneous assignment in a heap-buffer-overflow example, changing whether the vulnerability is exposed.The example compares GCC and Clang across optimization levels O0–O3.
- Crash Analysis Tools: 61.1% of 329,857 crash samples are validated by both ASan and GDB, while 14.5% are validated only by GDB and 12.2% only by ASan.Neither tool validates the remaining 12.2% of samples.
- Crash Analysis Tools: Using ASan versus GDB changes the observed fuzzing results on ffmpeg: only Honggfuzz finds bugs with ASan, whereas AFL and AFLFast also find bugs with GDB.UNIFUZZ therefore uses ASan as the main tool and GDB as a supplement.
6 Discussion
The discussion identifies usability, experimental practice, benchmark design, and metrics as important evaluation concerns. It also states scope boundaries for measuring usability, selecting benchmarks, and defining performance metrics.
- The Usability of Fuzzers: Usability problems among fuzzers include implementation flaws and failures to reproduce, and UNIFUZZ tests 35 fuzzers while evaluating eight extensively.The authors intend the platform to support further work on fuzzer usability and performance.
- The Usability of Fuzzers: Usability depends on user expertise and factors including documentation, dependencies, implementation, and fuzzing-process robustness.The paper proposes checking documentation, installation, author tests, and runtime robustness.
- Fuzzing Experiments: Correct evaluation requires monitoring experiment status, mitigating disk I/O, and following practices such as repeated runs and varied seed sets.Low CPU utilization can indicate abnormal execution, while disk I/O can become a bottleneck.
- The Benchmarks for Evaluating Fuzzers: UNIFUZZ’s 20 real-world programs vary in functionality, size, and vulnerability type, while preserving original program code and supporting offline crash analysis.The benchmark includes crash triage, CVE matching, and bug-severity analysis.
- The Benchmarks for Evaluating Fuzzers: The benchmark is currently selected mainly from top fuzzing papers and focuses on general program-level fuzzers, leaving compiler and kernel fuzzers for future extension.The authors also note that benchmarks need continued updating as fuzzers develop.
- Performance Metrics: The six metric categories remain limited because their classification and concrete measures require further research and better standards.The paper specifically notes limitations of CVSS scores and Exploitable’s rule-based severity assessment.
7 Conclusion
UNIFUZZ enables comprehensive, fair comparison of fuzzers using diverse benchmarks and metrics. Its evaluations show that results depend on the fuzzer, benchmark type, metric, instrumentation method, and crash analysis tool.
- UNIFUZZ incorporates 35 fuzzers, 20 real-world benchmark programs, and six categories of performance metrics for systematic comparison.
- No fuzzer consistently outperforms the others, revealing potential subjectivity and bias in existing fuzzing evaluations.
- Performance on synthetic benchmark programs may not match performance on real-world programs, underscoring the importance of pragmatic benchmarks.
- Fuzzer performance varies across metrics, so reliable assessment requires comprehensive rather than single-metric evaluation.
- Instrumentation methods and crash analysis tools can significantly affect fuzzer evaluation.