Source-linked AI summary
Magma: A Ground-Truth Fuzzing Benchmark
Ahmad Hazimeh, Adrian Herrera, Mathias Payer
TL;DR
Fuzzer evaluation is hindered by inadequate metrics, inconsistent targets, and stochastic behavior. Magma addresses these issues with real programs, reinserted real bugs, ground-truth instrumentation, and bug-centric metrics. Its evaluation demonstrates systematic comparison and provides insight into fuzzer performance, crash-count misleadingness, randomness, and fault-detection shortcomings.
Problem
Existing fuzzer evaluations lack sufficiently accurate metrics and unified targets, while crash counts can misrepresent bug findings.
Method
Magma forward-ports real bugs into seven real targets and instruments them to measure bug reachability, triggering, and detection.
Results
The evaluation of seven widely used fuzzers shows that ground truth enables systematic performance comparison and reveals shortcomings in crash counts, randomness effects, and fault detection.
Takeaways & Limitations
Ground truth supports more accurate and meaningful fuzzer evaluation by providing accessible bug-centric performance information.
Takeaways & Limitations
Only 45 % of Magma’s 118 bugs have proof-of-vulnerability inputs that confidently support performance measurement.
Abstract
from arXiv · showhide
High scalability and low running costs have made fuzz testing the de facto standard for discovering software bugs. Fuzzing techniques are constantly being improved in a race to build the ultimate bug-finding tool. However, while fuzzing excels at finding bugs in the wild, evaluating and comparing fuzzer performance is challenging due to the lack of metrics and benchmarks. For example, crash count, perhaps the most commonly-used performance metric, is inaccurate due to imperfections in deduplication techniques. Additionally, the lack of a unified set of targets results in ad hoc evaluations that hinder fair comparison. We tackle these problems by developing Magma, a ground-truth fuzzing benchmark that enables uniform fuzzer evaluation and comparison. By introducing real bugs into real software, Magma allows for the realistic evaluation of fuzzers against a broad set of targets. By instrumenting these bugs, Magma also enables the collection of bug-centric performance metrics independent of the fuzzer. Magma is an open benchmark consisting of seven targets that perform a variety of input manipulations and complex computations, presenting a challenge to state-of-the-art fuzzers. We evaluate seven widely-used mutation-based fuzzers (AFL, AFLFast, AFL++, FairFuzz, MOpt-AFL, honggfuzz, and SymCC-AFL) against Magma over 200,000 CPU-hours. Based on the number of bugs reached, triggered, and detected, we draw conclusions about the fuzzers' exploration and detection capabilities. This provides insight into fuzzer performance evaluation, highlighting the importance of ground truth in performing more accurate and meaningful evaluations.
1 INTRODUCTION
Existing fuzzer evaluations are difficult to compare because commonly used metrics can misrepresent bugs and target sets vary across studies. Magma addresses this gap with real programs, real bugs, ground-truth metrics, and an evaluation of seven fuzzers.
- Benchmark gap: Evaluations also use widely varying target sets, making cross-fuzzer comparisons impossible.
- Metric deficiencies: Crash counts can inflate the apparent number of bugs because deduplication may fail to identify crashes’ root causes.Coverage profiles and stack hashes do not always accurately identify the underlying bug.
- Metric deficiencies: Accurate ground-truth bug counts usually require extensive manual triage by domain experts.
- Metric deficiencies: Higher code coverage does not necessarily indicate better fuzzer effectiveness because coverage-deduplicated crashes correlate weakly with ground-truth bugs.
- Magma: Magma introduces real bugs into seven real open-source targets and instruments them to measure whether bugs are reached, triggered, and detected.The benchmark analyzes and reinserts 118 bugs across targets totaling 2 MLOC.
- Contributions: The paper contributes bug-centric metrics, a comparison of existing benchmarks, Magma’s design and implementation, and an evaluation against seven widely used fuzzers.
2 BACKGROUND AND MOTIVATION
Fuzzing is stochastic and uses diverse fuzzer designs, making reliable evaluation difficult. Existing benchmarks and crash-based metrics have limitations in realism, bug diversity, ground-truth access, and comparison validity.
- Fuzzing background: Fuzzing’s stochastic behavior makes fuzzer evaluation and comparison difficult, regardless of the fuzzer’s input-generation strategy or program knowledge.
- Evaluation design: Meaningful comparison requires consistent seeds, trial durations, and many repeated trials because individual fuzzing trials vary substantially.
- Existing benchmarks: Existing benchmarks provide uneven support for realistic exploration and detection: CGC workloads are small and simple, while LAVA-M uses one simple bug type.
- Existing benchmarks: BugBench and Google FTS contain real programs with real bugs, but sparse bugs and limited automatic crash triage hinder fuzzer evaluation.
- Evaluation metrics: Crash counts can inflate results because detection methods may incompletely detect bugs and misidentify root causes, producing duplicated crashes.
3 DESIRED BENCHMARK PROPERTIES
A useful fuzzing benchmark must be diverse, verifiable, and usable while accounting for stochastic trial behavior. Ground truth should make bug-centric findings accessible and verifiable across realistic targets.
- Desired properties: Reproducibility is difficult because individual fuzzing trials vary widely, requiring many repeated trials in the same test environment.
- Desired properties: The benchmark properties are diversity, verifiability, and usability.These require realistic programs and bugs, accurate metrics, and no significant adoption barriers.
- Diversity (P1): Diversity requires real programs containing varied real bugs, while known bug distributions are needed for quantitative fuzzer comparisons.
- Diversity (P1): Bug diversity spans class, distribution, and complexity, including execution depth, path spread, input size, trigger ranges, and input transformations.
- Diversity (P1): Target diversity should cover application domains, operations, and input structures such as binary and text formats.
- Verifiability (P2): Ground-truth metrics should verify bugs a fuzzer can reach, trigger, and detect, enabling accurate evaluation and meaningful comparisons.
- Usability (P3): A usable benchmark needs few targets, many discoverable bugs, and a framework that reports fuzzer progress and performance.
4 MAGMA: APPROACH
Magma is a ground-truth benchmark built from seven diverse real-world targets containing 118 real bugs, with instrumentation that measures bug reachability, triggering, and detection. Its target selection and bug-centric metrics support more uniform and informative fuzzer evaluation.
- Benchmark construction: The benchmark includes 25 drivers that exercise different functionality within its seven target codebases.The drivers are sourced from the original target projects, where domain experts developed them.
- Target selection: Magma selects targets for diverse functionality and uses PCA over instruction-category traces to quantify workload diversity.The analysis covers 284 subjects and 94 instruction categories; the first four components account for 60% of variance.
- Target selection: The first four principal components tightly cluster LAVA-M workloads, whereas CGC and Magma span a wider variety of workloads.OpenSSL appears distinct because of its cryptographic and networking code, while CGC’s TAINTEDLOVE is distinct because of its floating-point operations.
- Bug selection and insertion: Magma contains 118 bugs spanning 11 CWEs and uses real bugs forward-ported to recent target versions rather than old, unpatched codebases.It has the second-largest CWE variety and second-largest bug density among compared benchmarks, though the cited passage truncates the density comparison.
- Bug-centric evaluation: Magma distinguishes bugs that are reached, triggered, and detected, using canaries and runtime monitoring to measure each stage.Fatal canaries can force a crash when a trigger condition is satisfied, enabling ideal-sanitization evaluation but sacrificing detection-capability measurement.
5 DESIGN AND IMPLEMENTATION DECISIONS
Magma’s design decisions prioritize verifiable, current-version bug instances and careful handling of forward-porting complexity. The benchmark also limits oracle data after the first triggered bug because execution may enter an unreliable weird state.
- Forward-porting: Forward-porting reintroduces known bugs into newer target versions while preserving ground-truth oracles, but future changes can make some bugs obsolete or untriggerable.Magma uses fuzzing-generated proofs of vulnerability to reduce manual verification costs and identify triggerable bugs.
- Manual bug insertion: All 118 bugs are manually introduced through report analysis, fix-commit inspection, trigger-condition modeling, and runtime canary injection.The process was chosen because automated alternatives would require complex program analysis or natural-language inference.
- Manual bug insertion: Manual forward-porting relies heavily on prose in bug reports to infer developer intent and precise trigger conditions.Automating this would require either potentially explosive function analysis or natural-language inference beyond Magma’s development scope.
- Weird-state handling: After an undetected bug triggers, Magma records oracle data only through the first triggered bug because later execution may enter an unreliable weird state.The execution trace continues, but post-transition information is excluded from collected bug-oracle data.
- Weird-state handling: In the example, tmp.len == 0 triggers a division-by-zero bug, whereas tmp.len > 16 triggers an out-of-bounds write and leaves the division-by-zero bug untriggered.The example illustrates how one bug’s state changes can prevent another bug from triggering in the same execution.
5.3 A Static Benchmark
Magma is a static benchmark with realistic workloads, but static benchmarks can encourage overfitting. Its forward-porting process supports updates as real-world targets evolve while preserving realistic code.
- Static Benchmark: Static benchmarks use realistic workloads but are susceptible to overfitting when systems are tuned specifically for benchmark performance.This risk arises because benchmark optimization may replace attention to real workloads.
- Static Benchmark: Dynamically synthesizing benchmarks could reduce overfitting, but synthesized workloads may differ from real-world scenarios and large faithful programs remain difficult to generate.Program synthesis may therefore produce biased or incomplete evaluations.
- Static Benchmark: Forward-porting lets Magma update targets as they evolve by adding minimal instrumentation and faulty code, while supporting new bugs and features.Two undergraduate students added over 60 bugs in three new targets in one semester without software security experience.
5.4 Leaky Oracles
Magma’s instrumentation must avoid revealing oracle conditions through feedback that could alter fuzzer exploration. The benchmark uses always-evaluated memory writes and tests for overfitting to instrumentation-specific access patterns.
- Leaky Oracles: Branch-based oracles can leak trigger-condition information to coverage-guided fuzzers, potentially causing inputs to overfit the oracle.A fuzzer maximizing branch coverage could discover and satisfy an oracle branch.
- Leaky Oracles: Magma evaluates trigger conditions through always-evaluated memory writes whose results are reported to a runtime monitor rather than exposed as ordinary branches.Compilers typically translate the comparisons into cmp and set instructions embedded in the execution path.
- Leaky Oracles: The canary’s faulty flag disables future canaries after the first bug is encountered, addressing interactions caused by weird states.This behavior is part of the canary implementation.
- Leaky Oracles: Magma avoids short-circuit-induced coverage leaks for compound conditions on x86-64 using custom assembly blocks that evaluate logical operators in one basic block.For other compilation targets, it uses C bitwise operators, which are more brittle and vulnerable to safety-agnostic compiler passes.
- Leaky Oracles: Instrumentation may still expose memory-access patterns to taint or data-flow analyses, so repeated runs with uninstrumented binaries can test whether results vary significantly.Statistical tests can help infer whether the fuzzer overfits those patterns.
5.5 Proofs of Vulnerability
Magma requires a proof-of-vulnerability input for every injected bug, but obtaining such inputs manually is laborious and domain-specific. The benchmark therefore reuses public PoVs or generates them through fuzzing.
- Proofs of Vulnerability: Every injected bug requires a proof-of-vulnerability input to verify that the bug can be triggered.Manually crafting PoVs requires knowledge of both the input format and target program.
- Proofs of Vulnerability: When public PoVs are unavailable, Magma runs multiple fuzzing campaigns and manually inspects untriggered bugs for path reachability and trigger-condition satisfiability.Inputs that trigger bugs are saved as PoVs.
5.6 Unknown Bugs
Because Magma uses real-world programs, some bugs may lack ground-truth oracles. Such uninstrumented crashes are excluded from benchmark metrics but can be used to extend Magma.
- Unknown Bugs: Crashes from bugs without available ground-truth oracles are excluded from Magma’s metrics, then investigated manually for possible benchmark inclusion.Execution-trace analysis can identify the root cause before adding the bug to Magma.
5.7 Fuzzer Compatibility
Magma targets fuzzers that concretely execute programs as OS processes, including symbolic-guided fuzzers when execution remains on the host system. This restriction enables low-complexity runtime monitoring through operating-system services.
- 5.7 Fuzzer Compatibility: Symbolic-execution fuzzers may create instrumentation incompatibilities and runtime-environment inconsistencies when their engines model the environment differently.These issues motivate Magma’s execution-context restriction.
- 5.7 Fuzzer Compatibility: Magma includes whitebox fuzzers using symbolic execution when they still execute the target concretely on the host system.SymCC is given as an included example.
- 5.7 Fuzzer Compatibility: Magma requires evaluated fuzzers to execute targets in OS processes with unrestricted access to OS facilities.The restriction covers system calls, libraries, and file-system access.
6 EVALUATION
The evaluation shows substantial differences among fuzzers across targets and campaign lengths, while many short-term differences are statistically inconclusive. Ground-truth metrics expose both exploration difficulty and detection gaps that crash counts alone cannot capture.
- 6 EVALUATION: AFL, AFLFast, AFL++, and SymCC-AFL performed similarly against most targets, with most mean bug-count differences statistically insignificant.AFL++ was an exception against libtiff, although the supplied passage truncates the specific result.
- 6 EVALUATION: FairFuzz showed significant regression against libxml2, openssl, and php, and its previously reported xmllint coverage improvement was not reproduced.The comparison concerns the results reported in this evaluation and the original FairFuzz evaluation.
- 6 EVALUATION: honggfuzz and MOpt-AFL significantly outperformed the other fuzzers on three of seven targets, while honggfuzz was also best on libpng.The authors attribute honggfuzz’s performance partly to memory-comparison wrapping that supplies comparison-progress information.
- 6 EVALUATION: 74 of 118 Magma bugs (62 %) were reached, while 43 of 54 verified bugs (79 %) were triggered during 24 h campaigns.No single fuzzer triggered more than 37 bugs, or 68 % of verified bugs.
- 6 EVALUATION: 19 of 43 triggered bugs (44 %) took more than 20 h on average to trigger, indicating persistent exploration difficulty.A set of 17 simple bugs was found consistently by all fuzzers within 24 h, providing a regression-detection baseline.
- 6 EVALUATION: Time-to-reach and time-to-trigger capture different difficulty dimensions: MAE115 took 10 s to reach but up to 20 h on average to trigger.The passage associates this gap with complex, stateful data-flow constraints.
- 6 EVALUATION: Campaign duration changed the ranking: MOpt-AFL triggered 11 additional bugs after 24 h and became most successful over seven days, whereas honggfuzz triggered none of those bugs.The result supports using long campaigns and survival-time analysis when comparing fuzzer performance.
- 6 EVALUATION: Magma’s bug-centric ground truth supports evaluating bugs reached, triggered, and detected, distinguishing path exploration from constraint-solving and detection capabilities.Ground truth is available for all 118 bugs, but only 45 % have a proof-of-triggerability (PoV), limiting confident performance measurement to those bugs.
7 CONCLUSIONS
Magma provides an open, ground-truth benchmark for accurate and consistent fuzzer comparison, and evaluation against it demonstrates the value of ground truth.
- Magma forward-ports 118 bugs across seven diverse real-world targets, providing an extensible benchmark for fuzzer evaluation.Its simple design supports collaborative updates and extensions.
- Evaluation of seven popular mutation-based fuzzers shows that ground truth enables systematic performance comparison.The evaluated fuzzers were AFL, AFLFast, AFL++, FairFuzz, MOpt-AFL, honggfuzz, and SymCC-AFL.
- The evaluation provides insight into misleading crash counts, randomness in fuzzer performance, and shortcomings in existing fault-detection methods.
- Ground-truth benchmarks such as Magma can make fuzzing evaluation reproducible and help researchers demonstrate the contributions of new approaches.Magma is open-source and available for adoption.
A BUGS AND REPORTS
The appendix catalogs injected bugs and reports their mean survival times, including longer-run measurements and bugs not triggered during evaluation.
- Table A1 lists the 118 injected bugs and their original bug reports; 78 bugs (66%) have a scope measure of one.PoVs for (*)-marked bugs come from bug reports.
- Table A2 reports mean reached and triggered bug survival times over 24 hours, sorted by difficulty and highlighting the best-performing fuzzer.Times are expressed in seconds, minutes, and hours.
- None of the bugs in the continuation of Table A2 were triggered by the seven evaluated fuzzers.
- Table A3 reports mean bug survival times over a 7-day period, with a continuation table for the remaining entries.