PaperlayerView exact revision on arXiv

Source-linked AI summary

Magma: A Ground-Truth Fuzzing Benchmark

Ahmad Hazimeh, Adrian Herrera, Mathias Payer

arXiv:2009.01120v2cs.CR

TL;DR

Fuzzer evaluation is difficult because existing metrics are insufficient and target sets vary. Magma introduces a ground-truth benchmark, and its evaluation shows that ground truth enables systematic fuzzer comparison.

  • Problem

    Fuzzer comparisons lack sufficient metrics and consistent target sets, making performance evaluation difficult, especially given fuzzing’s stochastic nature.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    Magma forward-ports 118 real bugs into seven open-source libraries and applications, instrumenting them with canaries and a runtime monitor to measure reaching and triggering.

    Source: §1 · p. 2, §4 · p. 7

  • Results

    The evaluation shows ground truth enables systematic fuzzer comparison and reveals misleading crash counts, randomness effects, and shortcomings in existing fault-detection methods.

    Source: §7 · p. 20

  • Takeaways & Limitations

    Magma’s extensible design supports open-source collaborative development of the benchmark.

    Source: §7 · p. 20

    Only 45 % of Magma’s bugs have a PoV, so only PoV-backed bugs can confidently measure fuzzer performance.

    Source: §6.4.1 · p. 19

Abstract

from arXiv · show

High scalability and low running costs have made fuzz testing the de facto standard for discovering software bugs. Fuzzing techniques are constantly being improved in a race to build the ultimate bug-finding tool. However, while fuzzing excels at finding bugs in the wild, evaluating and comparing fuzzer performance is challenging due to the lack of metrics and benchmarks. For example, crash count, perhaps the most commonly-used performance metric, is inaccurate due to imperfections in deduplication techniques. Additionally, the lack of a unified set of targets results in ad hoc evaluations that hinder fair comparison. We tackle these problems by developing Magma, a ground-truth fuzzing benchmark that enables uniform fuzzer evaluation and comparison. By introducing real bugs into real software, Magma allows for the realistic evaluation of fuzzers against a broad set of targets. By instrumenting these bugs, Magma also enables the collection of bug-centric performance metrics independent of the fuzzer. Magma is an open benchmark consisting of seven targets that perform a variety of input manipulations and complex computations, presenting a challenge to state-of-the-art fuzzers. We evaluate seven widely-used mutation-based fuzzers (AFL, AFLFast, AFL++, FairFuzz, MOpt-AFL, honggfuzz, and SymCC-AFL) against Magma over 200,000 CPU-hours. Based on the number of bugs reached, triggered, and detected, we draw conclusions about the fuzzers' exploration and detection capabilities. This provides insight into fuzzer performance evaluation, highlighting the importance of ground truth in performing more accurate and meaningful evaluations.

1 INTRODUCTION

The introduction argues that common fuzzing metrics and inconsistent target sets make fair fuzzer comparisons difficult. It presents Magma, a ground-truth benchmark built from real programs and bugs, with bug-centric metrics for evaluating fuzzers.

1 INTRODUCTION

  • Existing evaluations use crash counts, ground-truth bug counts, and coverage profiles, but inconsistent target sets across papers make cross-fuzzer comparisons impossible.The introduction argues that these metrics do not sufficiently measure a fuzzer’s ability to find bugs.Source: §1 · pp. 1–2
  • Crash counts can inflate bug totals [30], and coverage profiles or stack hashes may fail to identify crash root causes [30].Accurate ground-truth bug counts instead require extensive manual triage and domain expertise.Source: §1 · pp. 1–2
  • Higher code coverage does not necessarily mean a fuzzer is more effective, because coverage-deduplicated crashes correlate weakly with ground-truth bugs [30].Source: §1 · p. 2
  • Magma covers seven open-source libraries and applications totaling 2 MLOC, with 118 analyzed and reinserted bugs, each paired with a lightweight oracle for tracking whether it is reached or triggered.The benchmark also distinguishes triggering a bug from detecting a triggered bug, enabling evaluation across multiple dimensions.Source: §1 · p. 2
  • The paper’s stated contributions include bug-centric metrics, a quantitative comparison of existing benchmarks, Magma’s design and implementation, and evaluation against seven widely used fuzzers.Source: §1 · p. 2

2 BACKGROUND AND MOTIVATION

Because fuzzing is stochastic, evaluating and comparing fuzzers is difficult; existing metrics and benchmarks motivate the need for more consistent, ground-truth evaluation practices.

2.1 Fuzz testing (fuzzing).

  • Fuzzers generate inputs to trigger crashes, using target-format knowledge or mutations and varying levels of program-structure knowledge; this process is stochastic regardless of approach.Grammar-based fuzzers use a specified input format, mutational fuzzers iteratively modify inputs, and whitebox, blackbox, and greybox fuzzers differ in how they use program structure or runtime information.Source: §2.1 · pp. 2–3

2.2 The Current State of Fuzzer Evaluation.

  • Klees et al. found that none of 32 surveyed papers adequately supported claims of fuzzer improvement [30], motivating consistent evaluation criteria.These include suitable metrics and diverse, realistic targets, consistent seeds and trial durations, and enough repeated trials for statistically sound comparisons; the benchmark should provide ground truth.Source: §2.2 · p. 3

2.2.1 Existing Fuzzer Benchmarks.

  • Existing benchmarks have complementary limitations: LAVA-M uses a single simple bug type, while CGC has small, simple workloads and other benchmarks lack bug density or ground truth.BugBench and FTS have sparse bugs and lack automatic crash triage; FuzzBench uses coverage alone, UniFuzz [33] lacks ground truth, and OSS evaluations face version divergence and unknown crash causes.Source: §2.2.1 · pp. 3–4

2.2.2 Crashes as a Performance Metric.

  • Crash counts can be inflated despite deduplication [30], while bug counts are unreliable because bug definitions and detection techniques can miss bugs or misidentify root causes.The paper argues that comparing the number of bugs found is preferable, but formal program models are difficult to derive and blacklist-based detectors can produce incomplete or duplicated results.Source: §2.2.2 · p. 5

3 DESIRED BENCHMARK PROPERTIES

Because fuzzing is highly stochastic and may alter targets or inputs, the authors argue that fuzzing benchmarks need diversity, verifiability, and usability.

3.1 Diversity (P1)

  • A diverse benchmark should use real programs with real bugs and known, verifiable ground truth, avoiding bias from unknown bug sets and distributions.FuzzBench does not use bugs as an evaluation metric, while LAVA-M includes only out-of-bounds memory-access bugs.Source: §3.1 · pp. 5–6
  • Bug diversity spans CWE classes, bug depth, fan-in and spread, and triggering-input complexity, including bytes involved, value ranges, and input transformations.Source: §3.1 · p. 5
  • Target diversity spans application domains, operations, and input structures, including parsing, transformation, state management, and binary, text, grammar-based, and differently sized inputs.Source: §3.1 · pp. 5–6

3.2 Verifiability (P2)

  • Ground truth should make bug findings verifiable and expose accessible metrics for bugs reached, triggered, and detected, avoiding unreliable crash counts.Crash counts vary, can double-count, and are inconsistently deduplicated across trials.Source: §3.2 · p. 6
  • CGC and Google FTS provide crashing inputs but not crash-root-cause determination; LAVA-M reports triggered bugs without runtime access, while FuzzBench relies on coverage profiles.Google FTS’s triage approach can miss complex bugs, and coverage does not necessarily indicate bug-finding effectiveness.Source: §3.2 · p. 6

3.3 Usability (P3)

  • Usability requires support for varied fuzzer types, manageable repeated trials, and a small target set with many discoverable bugs plus a framework that reports progress and performance.Without easily accessible ground truth, users must triage crashes because crash counts cannot establish how many bugs were triggered.Source: §3.3 · p. 7

4 MAGMA: APPROACH

Magma combines seven real-world software targets with 118 verifiable bugs and instrumentation to evaluate fuzzers using bug-centric measurements. Its approach includes diverse target selection, forward-ported bugs, runtime monitoring, and an optional fatal-canaries mode.

4 MAGMA: APPROACH

  • Magma combines seven widely used real-world targets and 118 verifiable bugs with canary instrumentation and a runtime monitor for evaluating fuzzers.Bugs are injected into the latest target versions, and the benchmark is designed to impose minimal requirements on users.Source: §4 · p. 7
  • Fatal-canaries mode terminates the program when a canary condition is satisfied, enabling ideal-sanitization evaluations without sanitizer overhead.It increases executions during evaluation but sacrifices measurement of the fuzzer’s detection capability.Source: §4 · pp. 7–8

4.1 Target Selection

  • Figure 1 shows Magma workloads span diverse functionality, unlike tightly clustered LAVA-M subjects; Magma includes 25 drivers, and PCA’s first four components explain 60% of benchmark variance.The PCA compared 284 subjects using 94 instruction categories; openssl and CGC’s TAINTEDLOVE are examples of workloads distinct from the main clusters.Source: §4.1 · pp. 8–9, Figure 1 · p. 9

4.2 Bug Selection and Insertion

  • Magma contains 118 real bugs spanning 11 CWEs, forward-ported from bug reports into current target versions.Magma has the second-largest CWE variety after CGC and the second-largest bug density after LAVA-M; the paper contrasts Magma’s real-world targets with CGC’s workloads and LAVA-M’s single synthetic bug type. Forward-porting fixes the limitation of relying on old, unpatched codebases and makes targets extensible.Source: §4.2 · p. 9

4.3 Performance Metrics

  • Fuzzer performance is measured by bugs reached, triggered, and detected, distinguishing execution of a bug’s context from satisfying its trigger and recognizing its fault.Canaries provide ground-truth feedback on reached and triggered bugs; triggering alone does not mean the fuzzer detected the fault.Source: §4.3 · p. 10

4.4 Runtime Monitoring

  • Magma’s runtime monitor collects real-time reached- and triggered-bug statistics, while replaying fuzzer crashes against canaries helps assess detection.Crashes that do not correspond to an injected bug can be triaged and added to the benchmark.Source: §4.4 · p. 11

5 DESIGN AND IMPLEMENTATION DECISIONS · 5.1 Forward-Porting · 5.2 Weird States

Magma’s fuzzing focus shapes its design: bugs are manually forward-ported and verified as triggerable, while oracle collection stops at the first bug to avoid unreliable data from later weird states.

5 DESIGN AND IMPLEMENTATION DECISIONS

  • Magma’s design choices reflect its focus on fuzzing rather than serving as a general bug-detection benchmark.Source: §5 · p. 11

5.1.1 Forward-Porting vs. Back-Porting.

  • Forward-porting preserves known fixes and ground-truth oracles, while fuzz-generated PoVs verify that reintroduced bugs remain triggerable and can update the verified-bug list.Future code changes can make a forward-ported bug obsolete or its trigger unsatisfiable; this verification approach may favor fuzzer-discoverable bugs, but newly discovered PoVs can update the benchmark.Source: §5.1.1 · p. 11

5.1.2 Manual Forward-Porting.

  • 34% of Magma’s 118 bugs have a scope measure greater than one, requiring more advanced porting techniques that can handle interprocedural constraints.The authors measure porting complexity by the code blocks, functions, or modules spanned by each fix; a simple technique works for fixes with a scope of one.Source: §5.1.2 · pp. 11–12
  • Manual porting relies on bug-report prose and function names to infer developer intent and precise trigger conditions, while automation would require complex analysis or NLP.In-depth function analysis risks path explosion, and the authors consider either automated approach too complex for Magma’s development scope and likely to require years of research.Source: §5.1.2 · p. 12

5.2 Weird States

  • After the first bug triggers, Magma continues the execution trace but collects oracle data only up to that point because later information from the resulting weird state is unreliable.Oracles indicate whether a bug’s execution conditions are satisfied; they do not indicate that the bug was executed.Source: §5.2 · p. 12
  • The example shows that triggering an out-of-bounds write can alter tmp.len and thereby trigger a division-by-zero bug even when the input did not specify a zero-length string.A length of 16 triggers both bugs because the write replaces tmp.len with a NULL terminator, whereas a length greater than 16 triggers only the write and a length of zero triggers only the division-by-zero bug.Source: §5.2 · p. 12

5.3 A Static Benchmark

Magma is a static benchmark built around realistic workloads, which can make performance informative but also leaves it vulnerable to overfitting. It mitigates that risk by updating targets as real software evolves, while avoiding the potential fidelity problems of synthesized workloads.

5.3 A Static Benchmark

  • Static benchmarks such as SPEC CPU and DaCapo use realistic workloads, but developers may overfit systems to benchmark tasks instead of real-world workloads.Their premise is that strong performance on benchmark workloads predicts similar performance on real workloads.Source: §5.3 · p. 12
  • Synthesized benchmarks could reduce overfitting, but may create unrealistic workloads, and generating large programs faithful to real development patterns remains difficult.Such workloads could make evaluation biased or incomplete.Source: §5.3 · p. 13
  • Magma’s forward-porting process updates targets as software evolves, requiring only instrumentation and faulty code; two undergraduates added over 60 bugs across three targets in one semester.This process also allows new bugs and features to be introduced, helping keep targets representative of real, complex software for fuzzer evaluation.Source: §5.3 · p. 13

5.4 Leaky Oracles

Magma avoids exposing oracle trigger conditions through coverage by evaluating them without adding detectable branches. This simplifies instrumentation compared with paired binaries, though data-flow signals may still permit overfitting.

Leaky Oracles

  • Always-evaluate memory writes let bug oracles evaluate trigger conditions and share their results with the runtime monitor without exposing an oracle branch to coverage feedback.A paired instrumented and uninstrumented binary approach could isolate feedback, but duplicating executions would complicate synchronization and add runtime overhead.Source: §5.4 · p. 13
  • The canary’s faulty flag disables later canaries after the first bug is encountered, addressing the problem of weird states.The always-evaluated memory accesses implement the canary.Source: §5.4 · p. 13
  • To prevent compound trigger conditions from creating coverage-visible implicit branches, Magma uses single-basic-block x86-64 assembly, falling back to C bitwise operators on other targets.The fallback is more brittle and susceptible to safety-agnostic compiler passes.Source: §5.4 · pp. 13–14
  • Memory-access patterns may still reveal oracle outcomes to taint tracking or other data-flow analyses, so repeating fuzzing with an uninstrumented binary can test for significant result differences.Statistical tests can be used to infer whether the fuzzer overfits to those patterns.Source: §5.4 · p. 14

5.5 Proofs of Vulnerability

Magma requires a proof-of-vulnerability (PoV) input for every injected bug, but manually crafting one is arduous and requires domain-specific knowledge. The authors obtain PoVs from public reports when available and use fuzzing campaigns otherwise.

5.5 Proofs of Vulnerability

  • Every injected bug requires a PoV input to verify it can be triggered, but manually crafting these inputs is arduous and demands domain-specific knowledge.The required knowledge concerns both the input format and the target program.Source: §5.5 · p. 14
  • When public bug reports lack a PoV, multiple fuzzing campaigns seek inputs that trigger each bug; untriggered bugs are manually inspected for reachability and satisfiable trigger conditions.Available PoVs are extracted from public bug reports, and triggering inputs found in campaigns are saved as PoVs.Source: §5.5 · p. 14

5.6 Unknown Bugs

Magma excludes crashes from bugs lacking ground-truth oracles from its metrics, but uses them to improve the benchmark through manual root-cause analysis.

5.6 Unknown Bugs

  • Crashes from bugs without ground-truth oracles are excluded from Magma’s metrics but can improve the benchmark after manual root-cause analysis of execution traces.The identified bug can then be added to the benchmark.Source: §5.6 · p. 14

5.7 Fuzzer Compatibility

Magma evaluates fuzzers that execute targets as OS processes on the host, including whitebox fuzzers that use symbolic execution to guide inputs. This restriction allows its runtime monitor to collect canary statistics through OS services.

5.7 Fuzzer Compatibility

  • Symbolic-execution fuzzers such as Driller and T-Fuzz may conflict with Magma instrumentation or create runtime-environment inconsistencies.The inconsistencies can depend on how the symbolic execution engine models the environment.Source: §5.7 · p. 14
  • Magma focuses on fuzzing rather than all bug-finding tools, and permits whitebox fuzzers using symbolic execution if they execute targets on the host, as SymCC does.This contrasts with benchmarks such as the CGC and BugBench, which aim to evaluate all bug-finding tools.Source: §5.7 · p. 14
  • Magma restricts evaluated fuzzers to executing targets as OS processes with unrestricted OS access, enabling low-overhead canary monitoring through OS services.The required access includes system calls, libraries, and the file system.Source: §5.7 · p. 14

6 EVALUATION

Across 24-hour campaigns, fuzzers reached many Magma bugs but triggered fewer verified bugs, with substantial differences in performance and discovery time. The evaluation also shows how ground truth supports comparisons beyond crashes and reveals limits of current fuzzers and detectors.

6.1 Methodology; 6.2 Time to Bug.

  • Seven fuzzers ran ten matched 24-hour and 7-day campaigns per target (200,000 CPU-hours); survival analysis modeled discovery times and accounted for failed discoveries and trial variation.Campaigns used identical seeds and limits; fatal canaries measured reaching and triggering, while ASan replay assessed detected bugs.Source: §6.1 · p. 15, §6.2 · p. 15

6.3.1 Bug Count and Statistical Significance.

  • AFL, AFLFast, AFL++, and SymCC-AFL performed similarly on most targets; AFL++ significantly led libtiff, while FairFuzz regressed on libxml2, openssl, and php.Small mean bug-count differences were generally not statistically significant.Source: §6.3.1 · p. 16, Figure 4 · p. 16
  • Honggfuzz and MOpt-AFL significantly outperformed other fuzzers on three of seven targets, and honggfuzz was best on libpng, aided by comparison-function wrappers.Honggfuzz also found 11 bugs no other fuzzer triggered and ran at least three times faster than AFL-based fuzzers using persistent drivers.Source: §6.3.1 · p. 16, §6.3.2 · p. 18

6.3.2 Time to Bug.

  • During 24 h campaigns, fuzzers reached 74 of 118 Magma bugs (62%) and triggered 43 of 54 verified bugs with PoVs (79%); no single fuzzer triggered more than 37 (68% of verified bugs).Source: §6.3.2 · p. 16
  • Nineteen of 43 triggered bugs took over 20 hours on average, while 17 were consistently found within 24 hours; MAE115 took 10 seconds to reach but up to 20 hours to trigger.The reach-trigger gap indicates that bugs can require complex, stateful data-flow constraints; a purpose-built harness triggered AAH020 faster than a general-purpose driver.Source: §6.3.2 · pp. 16–18
  • In 7-day campaigns, MOpt-AFL triggered 11 additional bugs after 24 hours and was most successful, although honggfuzz had triggered bugs faster in 24-hour campaigns.Honggfuzz did not trigger any of those 11 additional bugs.Source: §6.3.2 · p. 17

6.3.3 Achilles’ Heel of Mutational Fuzzing; 6.3.4 Magic Value Identification.

  • For libpng, only honggfuzz, AFL, and SymCC-AFL triggered AAH001’s divide-by-zero bug requiring width 0x55555555, while honggfuzz and SymCC-AFL consistently triggered AAH007’s dangling-pointer bug.SymCC-AFL was fastest on AAH001, likely due to constraint solving; honggfuzz’s incremental string-comparison instrumentation helped with AAH007.Source: §6.3.3 · p. 18, §6.3.4 · p. 18

6.3.5 Semantic Bug Detection; 6.3.6 Comparison to LAVA-M.

  • AAH003 is a semantic data-inconsistency bug that standard sanitizers cannot detect without an expected-behavior specification; LAVA-M’s who target concentrates 94.3% of its bugs in 5.3% functional code.Fuzzers spent an average of 42.9% of their time executing who’s oracle code, and LAVA-M’s single simple bug type favored fuzzers with magic-value detection.Source: §6.3.5 · p. 18, §6.3.6 · p. 19

7 CONCLUSIONS

Magma is an open ground-truth benchmark built by forward-porting 118 real bugs across seven diverse targets to support accurate, consistent fuzzer evaluation. Its evaluation shows how ground truth enables systematic comparison and provides insight into misleading crash counts, randomness, and fault-detection shortcomings.

7 CONCLUSIONS

  • Magma forward-ports 118 bugs across seven diverse targets, and its simple design supports improvement, updates, and extension through open-source collaboration.The authors encourage fuzzer developers to incorporate their fuzzers into Magma.Source: §7 · p. 20
  • Evaluation of seven popular open-source mutation-based fuzzers shows that ground truth enables systematic comparison and reveals misleading crash counts, randomness effects, and shortcomings in some fault-detection methods.The evaluated fuzzers are AFL, AFLFast, AFL++, FairFuzz, MOpt-AFL, honggfuzz, and SymCC-AFL.Source: §7 · p. 20
  • Ground-truth benchmarks such as Magma can make fuzzer evaluation reproducible, helping researchers demonstrate the true contributions of new fuzzing approaches.Magma is open-source and available at https://hexhive.epfl.ch/magma/.Source: §7 · p. 20

A BUGS AND REPORTS

The appendix documents Magma’s injected bugs and reports mean bug survival times, including a set of bugs none of the seven evaluated fuzzers triggered.

A BUGS AND REPORTS

  • Of 118 injected bugs, 78 (66%) have a scope measure of one; relying on automatic porting would yield fewer and lower-quality canaries.PoVs for bugs marked with an asterisk are sourced from bug reports.Source: Table A1 · p. 24
  • Table A2 reports mean Reached and Triggered bug survival times over 24 hours, while Table A3 reports mean survival times over seven days.Source: Table A2 · p. 25, Table A3 · pp. 27–28
  • None of the bugs listed in the Table A2 continuation were triggered by the seven evaluated fuzzers.Source: A BUGS AND REPORTS · p. 26

Related papers

Loading 2009.01120v2…