Source-linked AI summary
FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs
Ze Sheng, Aleksandar Kezic, Zhicheng Chen, Jeff Huang
TL;DR
Existing LLM bug-discovery benchmarks often require reproducing a predefined vulnerability, potentially overlooking other valid crashes. FuzzingBrain-Bench evaluates open-ended discovery by counting distinct crash signatures triggered through harnesses across diverse challenges, with Opus 4.8 achieving the strongest result while 13 challenges remained untriggered.
Problem
Existing dynamic benchmarks typically evaluate whether a model reproduces one predefined vulnerability, potentially overlooking valid crashes outside that target.
Method
FuzzingBrain-Bench gives models source code and sanitizer-instrumented harnesses in Docker images and scores distinct crash signatures triggered through each harness.
Results
Opus 4.8 triggered crashes in 60 of 77 challenges and scored 196 out of 579, outperforming Sonnet 4.6 and Haiku 4.5.
Takeaways & Limitations
The benchmark measures broader bug-discovery capability rather than reproduction of a particular known vulnerability across C, C++, and Java/JVM projects.
Takeaways & Limitations
Version 1 contains 77 challenges from 43 projects, and the authors plan a larger corpus and broader crash-signature coverage.
Abstract
from arXiv · showhide
Evaluating the ability of large language models (LLMs) to discover software bugs is increasingly important. Existing benchmarks typically evaluate this capability by asking the model to generate a proof-of-concept input that triggers a predefined target vulnerability. However, this setup may overlook valid crashes discovered by the model when they do not match the predefined target. As a result, the evaluation may not reflect the model's real capability. We present FuzzingBrain-Bench, a benchmark for assessing AI models' ability to discover bugs in open-source software. Models are given an open-source project and a sanitizer-instrumented harness in a self-contained Docker image. Their goal is to generate inputs that trigger as many distinct crashes as possible through the harness. A model's performance on each challenge is scored based on the number of distinct crash signatures it produces, capped at a predefined maximum and weighted by a difficulty coefficient. FuzzingBrain-Bench V1 consists of 77 challenges drawn from 43 open-source projects, with 36 C, 32 C++, and 9 Java/JVM challenges. We evaluate Claude Haiku 4.5, Claude Sonnet 4.6, and Claude Opus 4.8 on the full benchmark. Claude Opus 4.8 performs best, triggering crashes in 60 of 77 challenges and achieving a score of 196 out of 579. None of the three models triggers a crash in 13 challenges. The FuzzingBrain-Bench corpus and harnesses are publicly available at https://github.com/fuzzingbrain/FuzzingBrain-Bench.
1 INTRODUCTION
FuzzingBrain-Bench addresses limitations in static and predefined-target evaluations by dynamically measuring open-ended crash discovery across diverse software projects. Its benchmark credits distinct crashes triggered through sanitizer-instrumented harnesses, and Opus 4.8 performs best on the 77-challenge evaluation.
- Motivation: 48,185 CVEs were reported in 2025, motivating scalable techniques for identifying software defects and security vulnerabilities.This was a 20.6% increase over 39,962 reports in 2024.
- Prior evaluation limitations: Existing static evaluations use simple code snippets and vulnerability labels, which cannot adequately assess defect understanding or real-world complexity.Richer second-generation outputs improve the metric but matching recorded answers neither guarantees correctness nor demonstrates triggerability.
- Prior evaluation limitations: Dynamic benchmarks require inputs to trigger vulnerabilities through sanitizer-instrumented harnesses, but typically judge success against one predefined memory-safety target in C/C++.This can overlook valid crashes outside the target vulnerability.
- FuzzingBrain-Bench: FB-Bench counts any distinct crash signature satisfying its criteria, allowing models to discover defects beyond the vulnerability originally associated with a challenge.Models inspect source and harnesses, submit candidate inputs, and observe raw execution output inside self-contained Docker images.
- FuzzingBrain-Bench: FB-Bench covers memory leaks, out-of-memory and timeout targets, and Java/JVM vulnerabilities in addition to C/C++ memory-safety bugs.The benchmark uses AddressSanitizer, UndefinedBehaviorSanitizer, LeakSanitizer, libFuzzer, and Jazzer.
- Results: Opus 4.8 triggered crashes in 60 of 77 challenges and scored 196 out of 579, while 13 confirmed-vulnerability challenges remained untriggered by all three models.Sonnet 4.6 and Haiku 4.5 scored 156 and 58, respectively.
2 FUZZINGBRAIN-BENCH V1
FuzzingBrain-Bench V1 packages open-source projects and sanitizer-instrumented harnesses in isolated Docker challenges, then evaluates models by valid, distinct crash signatures rather than a predefined vulnerability. Its design controls execution access, defines broad crash validity, normalizes signatures for deduplication, and weights capped counts by empirically assigned difficulty.
- Data selection and challenge construction: 77 challenges are constructed from published bug reports and package vulnerable source, harnesses, configurations, and instrumented binaries in Docker images.Reference PoCs, git history, and reports are removed from the published images.
- System design: The agent receives challenge details through setup, edits inputs with exec, and submits candidates through run poc on harness, which executes each candidate three times.The agent loop and challenge container communicate over MCP’s stdio transport, while shell commands have no network access.
- Crash definition: A run is a crash when it meets any listed abnormal termination or diagnostic condition, but it is invalid if the harness itself faults or both output streams are empty.The criteria include fatal signals with prior output, sanitizer reports, non-zero exits with diagnostic output, timeouts for slowdown defects, and uncaught JVM exceptions.
- Crash definition: Valid crashes include non-sanitizer faults and faults in dependencies, provided they are triggered through the harness and do not occur in the harness itself.For JVM targets, an uncaught exception is the fault even without a sanitizer report.
- Crash signatures and deduplication: Each valid crash is deduplicated as a normalized fault class followed by up to three distinct relevant function names from the stack trace.Sanitizer, allocator, fuzzing-driver, runtime, system-library, and harness-wrapper frames are removed before names are normalized and repeated names collapsed.
- Difficulty and scoring: Difficulty coefficients D ∈{1, . . . , 5} depend on how many reference-panel models crash and the largest distinct-signature count, while scoring credits at most three signatures per challenge.Uncapped counts assign difficulty; capped counts prevent a prolific challenge or defect from dominating the final score.
3 EXPERIMENTAL SETUP
FuzzingBrain-Bench V1 evaluates 77 verified vulnerabilities across 43 open-source projects in C, C++, and Java/JVM using deterministic, offline harness grading.
- Corpus and challenge construction: 77 verified vulnerabilities span 43 open-source projects: 36 C, 32 C++, and 9 Java/JVM challenges.The corpus uses AddressSanitizer, Jazzer, libFuzzer, UndefinedBehaviorSanitizer, and LeakSanitizer configurations.
- Corpus and challenge construction: Each challenge is packaged with vulnerable-project source, harness code, sanitizer configuration, and related challenge metadata.The corpus was assembled from published bug reports selected from multiple open-source vulnerability sources.
- Evaluation procedure: Grading is deterministic and offline inside the challenge image, with each input executed three times to assess reproducibility and reduce bug-flakiness.The environment does not provide an answer key or a predefined correct input to the model.
- Evaluation procedure: The experiments are reproducible through an appendix tutorial covering single-challenge and full-corpus runs under the stated budgets.
4 RESULTS
Across the 77-challenge corpus, Opus achieves the highest score and crash coverage, while results vary with difficulty, capping, model effort, cost, tokens, and runtime.
- Overall performance: 196/579 (33.85%) is Opus’s highest benchmark score, followed by Sonnet’s 156/579 (26.94%) and Haiku’s 58/579 (10.02%).Opus, Sonnet, and Haiku generate crashing inputs for 60/77, 50/77, and 35/77 challenges, respectively.
- Capping effects: 33.16% is Opus’s score increase without per-challenge capping, compared with 30.77% for Sonnet and 12.07% for Haiku.Removing the cap preserves the same model hierarchy and widens the gap between Opus and Sonnet.
- Difficulty effects: All models crash every D1 challenge, whereas none crashes a D5 challenge.D4 and D5 together carry 327 of the 579 available points, so higher tiers strongly affect total scores.
- Difficulty effects: Sonnet outperforms Opus in lower-difficulty D1 and D2 challenges, while Opus scores better in D3 and D4 challenges.The reported explanation is that Opus often terminates voluntarily before using its full turn budget; its median usage is 51 turns.
- Cost, tokens, and runtime: 84:1, 143:1, and 188:1 are the input-to-output token ratios for Opus, Sonnet, and Haiku, respectively.Average single-call input counts are 33.0k, 71.4k, and 38.0k tokens for Opus, Sonnet, and Haiku.
- Cost, tokens, and runtime: 10.5, 22.3, and 5.5 hours are the summed agentloop times for Opus, Sonnet, and Haiku, respectively.At eight parallel jobs, projected full-corpus runtimes are roughly 1.3, 2.8, and 0.7 hours per model.
5 CONCLUSION AND FUTURE WORK
FB-Bench evaluates open-ended crash discovery rather than reproduction of a predefined vulnerability, and current models still miss some confirmed vulnerabilities.
- Conclusion: FB-Bench credits any distinct crash signature satisfying its crash criteria instead of requiring reproduction of a predefined target vulnerability.This broadens evaluation from reproducing one known defect to discovering multiple defects through the harness.
- Conclusion: 60/77 challenges are crashed by the best-scoring model, Claude Opus 4.8, while 13 confirmed-vulnerability challenges remain untriggered by all evaluated models.
- Future work: 77 challenges from 43 projects are identified as a relatively small V1 corpus, motivating planned expansion to 1000 challenges.
- Future work: Future versions plan broader crash-signature coverage through additional sanitizers and difficulty coefficients derived from a wider model panel.The current coefficients are measured over three Anthropic models, and UndefinedBehaviorSanitizer covers four V1 challenges versus 53 under AddressSanitizer.
A MCP TOOLS
The MCP tools support setup, shell-based workspace operations, and repeated harness execution for generating and evaluating candidate inputs.
- Available tools: The setup tool returns the workspace, source paths, target project and language, and harness configuration.
- Available tools: The exec tool runs shell commands in the challenge source root without network access and returns truncated standard output, errors, exit code, and duration.It is the only filesystem tool for reading, writing, and listing workspace files.
- Candidate evaluation: The run_poc_on_harness tool executes a candidate input through the configured sanitizer-instrumented harness and returns raw output rather than a pass/fail verdict.Its output includes stdout, stderr, exit code, signal, and duration.
- Candidate evaluation: Repeated harness runs classify candidate outcomes as new, duplicate, flaky_rounds, or flaky_location relative to the session’s prior submissions.Only reproducible faults count as distinct crashes.
B MODEL-FACING PROMPTS
The model-facing prompts direct agents to discover and validate as many distinct harness-reachable crashes as possible, rather than stopping after one finding. Agents inspect the harness and reachable code, test inputs iteratively, and prioritize unexplored crash locations or types.
- Agent objective: The agent must maximize the number of unique vulnerabilities by generating crashing inputs for the sanitizer-instrumented harness.Valid crashes include memory-safety errors, execution errors, resource issues, and uncaught JVM exceptions.
- Validity criteria: Malformed inputs rejected before reaching the target and clean inputs that trigger no fault are not vulnerabilities.The prompt distinguishes non-crashes from valid harness-triggered sanitizer errors.
- Workflow: The agent analyzes the exact harness input format and reachable source code before hypothesizing and testing faults.The harness defines which project code can be exercised.
- Workflow: The agent repeatedly writes candidates, executes them on the harness, reads raw output, and refines hypotheses using the execution result as ground truth.Inputs should be tested early and often rather than analyzed endlessly.
- Exploration strategy: After a crash, the agent is prompted to search for a different crash at another location or of another type instead of terminating.The prompt explicitly reinforces continued exploration after the first sanitizer-triggering input.
- Budget management: The agent uses remaining turns to run untested inputs and seeks new faults rather than refining previously successful inputs.A low-turn reminder redirects effort toward candidates not yet executed.
C TUTORIAL - HOW TO RUN THE BENCHMARK
The tutorial explains how to install FuzzingBrain-Bench, run individual or full-corpus evaluations, and inspect reproducible episode outputs. The benchmark uses Dockerized challenges and supports parallel, repeated model–challenge runs.
- Installation: FuzzingBrain-Bench requires Docker, Python 3.10 or newer, and an API key for the model under test.Each challenge is a public self-contained Docker image, while the agent loop runs on the host.
- Single challenge: A single challenge is the recommended installation check, invoked with `fb-bench run <challenge-id> --model <model-name>`.Available challenge and model identifiers can be listed before running.
- Run monitoring: Progress appears in the terminal during an episode, followed by a completion summary when the episode terminates.Figures 11 and 12 illustrate the running and completed terminal output.
- Outputs: Each episode preserves its score, dialogue, tool records, transcript, trajectory, and report in a dedicated output directory.Candidate inputs are retained by default, allowing re-grading without invoking the model again.
- Full benchmark: The full corpus is run by passing `all`, with options for model selection, output naming, maximum turns, and parallel jobs.Four cells can be run in parallel with `--jobs 4`.
- Repeated evaluation: The same model–challenge pair can be repeated three times with `--samples 3` for repeated evaluation.Results are organized into separate seed directories for each run cell.
D FLAKY CRASH - CASE STUDY
The case study shows why repeated verification is required for crash signatures. Stack exhaustion can fault at different points in one recursive cycle, so unstable signatures are classified as flaky location and not counted.
- Verification rule: FB-Bench accepts a crash signature as new only when all three verification rounds produce the same signature.A stack-exhaustion input was accepted when its three rounds agreed.
- Case study: A second submission of the same 150 032-byte input produced two signatures across three rounds and was returned as flaky location.The first submission of that input produced one signature in every round and was accepted as new.
- Failure analysis: The differing diagnostics shared the same recursive cycle, but their frame windows began at different functions because stack exhaustion occurred at different turns.The summary line also varied between the fuzzing engine’s allocation hook and sanitizer bookkeeping.
- Scoring consequence: Neither flaky-location signature reserves a finding, so the unstable second submission contributes no counted crash.Only the first submission is counted because its three verification rounds agree.