Source-linked AI summary
The Impact of Magma: A Ground-Truth Fuzzing Benchmark
Ahmad Hazimeh, Adrian Herrera, Srividya Subramanian, Thaqiya Aman, Sara Vaccino, Qiang Liu, Mathias Payer
TL;DR
Fuzzer comparisons need metrics and targets that reflect real bug finding rather than relying on imperfect crash counts or coverage. Magma provides an open, ground-truth benchmark built from diverse real programs and instrumented real bugs, and its successive updates keep the benchmark current and repeatable. The benchmark has been adopted broadly by the research community while requiring ongoing maintenance as target code changes and new vulnerabilities appear.
Problem
Crash counts and code coverage are insufficient for reliable fuzzer comparison, motivating realistic targets and metrics that accurately measure bug finding.
Method
Magma combines diverse real-world targets, forward-ported ground-truth bugs, and canary-based metrics for reaching, triggering, and detecting bugs.
Results
Magma provides an up-to-date, realistic, and repeatable ground-truth setting for testing fuzzers, supported by continued community use and benchmark updates.
Takeaways & Limitations
Magma offers a maintained path for consistent fuzzer evaluation and performance comparison on real bugs in modern software.
Takeaways & Limitations
Maintaining relevance requires updates because code changes made 57 of 138 older bugs unappliable and 246 new CVEs affected Magma’s targets.
Abstract
from arXiv · showhide
Magma is an open-source and ground-truth fuzzing benchmark that enables uniform fuzzer evaluation and comparison. Magma was originally released with a research paper published at ACM SIGMETRICS 2021. This short paper explains the motivation, the design, and the impact of Magma, with a description of extensions to the original benchmark.
1. Introduction
Common fuzzing metrics such as crash counts and code coverage are insufficient for reliable fuzzer comparison because they do not accurately measure distinct ground-truth bugs. Magma addresses this gap with realistic, diverse targets and real bugs.
- Crash counts can inflate the number of actual bugs, while deduplication methods may fail to identify crash root causes accurately.
- Code coverage has only a weak correlation with ground-truth bugs, so higher coverage does not necessarily indicate better fuzzer effectiveness.
- Reliable evaluation should measure a fuzzer’s ability to find bugs using realistic targets that exercise diverse behavior.
- Magma is an open-source benchmark built from seven widely used open-source programs containing 118 manually analyzed and reinserted real bugs.Each bug has an oracle that detects whether it is reached or triggered.
2. The Magma Artifact
Magma combines diverse real-world targets, forward-ported real bugs, and bug-centric runtime metrics for ground-truth fuzzer evaluation. Its design distinguishes exploration, fault triggering, and detection while supporting extensible benchmark maintenance.
- 2.1. Target Selection: Magma’s seven targets span diverse functionality, enabling evaluation of fuzzer code-exploration capabilities.PCA shows the four LAVA-M targets clustering tightly, while CGC and Magma provide wider target variety.
- 2.2. Bug Selection and Insertion: Magma contains 118 bugs spanning 11 CWEs, combining broad bug-class coverage with realistic target codebases.Its bug density is 16.86 bugs per target, whereas LAVA-M has 566.25 bugs per target but only one synthetic bug type.
- 2.2. Bug Selection and Insertion: Forward-porting real bugs from reports and patches to current target versions avoids the limitations of old codebases and makes targets extensible.The process identifies and reverts bug-fix changes while disambiguating fixes that affect multiple bugs or commits.
- 2.3. Performance Metrics: Magma evaluates reaching, triggering, and detecting as distinct bug-centric performance dimensions.A canary records when buggy code is reached and when the trigger condition is satisfied; triggering does not necessarily mean the fuzzer detected the fault.
- 2.3. Performance Metrics: Runtime monitoring collects reached and triggered bugs, while replaying crashing inputs helps assess which bugs were reached, triggered, or detected.
3. Impact
Magma has become a broadly adopted fuzzing benchmark, used across many research directions and publication venues since its release.
- At least 45 high-quality papers have used Magma since its 2021 release.These papers appeared through 2025, with 14 papers in both 2024 and 2025.
- Magma’s users span security, software engineering, systems, and programming-language conferences.The paper presents this venue breadth as evidence of broad impact.
- Magma has been adopted for input generation, seed selection and mutation, scheduling, coverage feedback, sanitization, and directed fuzzing.
- Researchers have also used Magma for post-fuzzing analysis, acceleration, practical deployment, program analysis, program transformation, and backdoor detection.
4. Magma v1.3
Magma v1.3 updates targets, patches, infrastructure, and tooling to keep the benchmark relevant to modern fuzzing. Evaluation campaigns indicate that the updated benchmark remains reachable and useful for comparing fuzzers.
- 57 of 138 bugs in the old Magma version could no longer be applied after target code changes, motivating continued updates.The authors also found 246 newer CVEs affecting Magma’s targets.
- Magma v1.3 updates targets and fuzzers, repairs compatible bug patches, improves debugging and validation infrastructure, and adds automated versioning and patching.Eleven broken bug patches were removed, and PoC mode collects triggering inputs and crash logs.
- AFL++ triggered 40 bugs, Honggfuzz 28, and libFuzzer 10 during 24-hour campaigns on nine targets.
- Of 127 benchmark bugs, 77 were reached and 43 were triggered during the fuzzing campaigns.
- Among 57 bugs ported from older versions, 91% were reached and 34% were triggered.
- The update is intended to make Magma a sustainable benchmark that evolves with the fuzzing ecosystem.
5. Conclusion and Outlook
Magma provides an open, realistic, and repeatable ground-truth setting for evaluating fuzzers, while its maintenance approach keeps comparisons relevant as software and vulnerabilities change.
- Magma supports accurate and consistent fuzzer evaluation and performance comparison using real bugs across diverse targets.Its versions expand from seven targets and 118 bugs to nine updated targets and 127 bugs in v1.3.
- Magma v1.3 offers researchers an up-to-date, realistic, and repeatable benchmark for ground-truth fuzzing experiments.
- Updating software, refining infrastructure, and simplifying bug testing provides a path for maintaining benchmark relevance over time.