Source-linked AI summary
SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
Hwiwon Lee, Ziqi Zhang, Hanxiao Lu, Lingming Zhang
TL;DR
Existing benchmarks provide limited evidence about LLM agents’ performance on authentic security-engineering tasks. SEC-bench automates construction and verification of real-world vulnerability benchmarks through a multi-agent scaffold, then evaluates PoC generation and vulnerability patching. On the complete dataset, agents achieve at most 18.0% success in PoC generation and 34.0% in vulnerability patching, revealing substantial performance gaps.
Problem
Existing security benchmarks rely heavily on synthetic or simplified datasets and lack sufficient evidence about LLM agents’ performance on real-world security tasks.
Method
SEC-bench uses a multi-agent scaffold to construct repositories and harnesses, reproduce vulnerabilities in isolated environments, generate validated patches, and package security tasks.
Results
At most 18.0% of evaluated agents succeed in PoC generation and at most 34.0% succeed in vulnerability patching on the complete dataset.
Takeaways & Limitations
SEC-bench provides consistent, automated evaluation of LLM agents on authentic security-engineering tasks and exposes substantial capability gaps.
Takeaways & Limitations
SEC-bench focuses on C/C++ projects and a subset of vulnerability types detectable by memory-safety sanitizers.
Abstract
from arXiv · showhide
Rigorous security-focused evaluation of large language model (LLM) agents is imperative for establishing trust in their safe deployment throughout the software development lifecycle. However, existing benchmarks largely rely on synthetic challenges or simplified vulnerability datasets that fail to capture the complexity and ambiguity encountered by security engineers in practice. We introduce SEC-bench, the first fully automated benchmarking framework for evaluating LLM agents on authentic security engineering tasks. SEC-bench employs a novel multi-agent scaffold that automatically constructs code repositories with harnesses, reproduces vulnerabilities in isolated environments, and generates gold patches for reliable evaluation. Our framework automatically creates high-quality software vulnerability datasets with reproducible artifacts at a cost of only $0.87 per instance. Using SEC-bench, we implement two critical software security tasks to rigorously evaluate LLM agents' capabilities: proof-of-concept (PoC) generation and vulnerability patching. A comprehensive evaluation of state-of-the-art LLM code agents reveals significant performance gaps, achieving at most 18.0% success in PoC generation and 34.0% in vulnerability patching on our complete dataset. These results highlight the crucial steps needed toward developing LLM agents that are more practical, intelligent, and autonomous for security engineering.
1 Introduction
SEC-bench addresses the limited realism, scalability, and reliability of existing security benchmarks by automating construction and verification of real-world vulnerability instances. It evaluates LLM agents on PoC generation and vulnerability patching, where current systems achieve low success rates.
- Limitations of Existing Security Benchmarks: Existing security benchmarks inadequately represent real-world challenges because they lack scalable methods for constructing verifiable PoC inputs.Existing datasets can contain inaccuracies in up to 71% of samples, while some alternatives manually create only small, domain-specific challenge sets.
- SEC-bench: SEC-bench aims to automatically collect and verify real-world CVEs with reproducible PoC artifacts and validated patches.Its design targets high-quality vulnerabilities, minimal manual intervention, and realistic security-engineering scenarios.
- Challenges: Real-world vulnerability benchmarking is difficult because bug reports are unstructured, environments are sensitive, and public PoCs are often missing or unreliable.Thirty-three percent of analyzed GitHub reports ignore templates, reproduction often fails without exact environments, and nearly 40% of disclosures lack working PoCs or require manual repair.
- Evaluation: At most 18.0% of evaluated agents succeed in PoC generation and at most 34.0% succeed in vulnerability patching.The results indicate substantial performance gaps on the benchmark’s real-world security challenges.
- Contributions: The framework introduces a multi-agent scaffold for constructing practical, scalable benchmarks that reproduce vulnerabilities from real-world repositories.Specialized modules coordinate preprocessing, verification, and evaluation of vulnerability instances.
- Contributions: SEC-bench evaluates two security-engineering tasks: proof-of-concept generation and vulnerability patching.These tasks are intended to reflect security-engineering workflows rather than simplified benchmark challenges.
2 SEC-bench
SEC-bench combines preprocessing, multi-agent verification, and containerized evaluation to turn real-world CVEs into reproducible security tasks. Its verified instances expose substantial difficulty variation across agent stages and projects, while its statistics show larger issue descriptions and codebases than SWE-bench on several dimensions.
- Preprocessor: SEC-bench collects CVE metadata, bug reports, and project configurations before constructing customized Docker environments for verification.The pipeline begins from public CVE data and adapts environments to project-specific dependencies and settings.
- Evaluator: The evaluator packages verified instances into clean Docker images and formulates PoC-generation and vulnerability-patching tasks.The resulting tasks are designed to mirror professional security-engineering workflows.
- Verification: Sanitizer verdicts serve as the verification oracle, accepting PoCs when expected reports appear and patches when those reports disappear.This provides deterministic validation without subjective judgment.
- Verifier: SECVERIFIER uses coordinated builder, exploiter, and fixer agents to compile projects, validate PoCs, and produce vulnerability-resolving patches.A manager agent assigns subtasks, evaluates outputs, and requests targeted revisions when objectives are not met.
- Verification Performance: 81.7% of builder agents, 39.4% of exploiter agents, and 69.2% of fixer agents succeed, with exploitation the most difficult stage.Sequential execution means later agents are not run when an earlier agent fails; inaccurate or absent PoCs contribute to exploiter difficulty.
- Project Variation: Project success rates range from 4.2% for php and 12.0% for upx to above 40% for faad2, mruby, and njs.The text links lower rates to complex architectures, specialized vulnerabilities, and poorly documented reproduction steps.
- Dataset Statistics: SEC-bench averages 563.6 files, 482K lines of code, and 921.1-word issues, compared with 3,010 files, 438K lines, and 195.1 words for SWE-bench.Its longer issues reflect sanitizer reports that commonly include detailed crash information and call stacks.
3 Evaluation
The evaluation tests three code agents on PoC generation and vulnerability patching, finding low success rates and distinct failure patterns across tasks and scaffolds.
- Evaluation setup: The evaluation compares SWE-agent, OpenHands, and Aider with Claude 3.7 Sonnet, GPT-4o, and o3-mini.
- Evaluation setup: The benchmark evaluates PoC generation and vulnerability patching as security tasks reflecting real-world engineering workflows.
- Overall performance: 18.0% is the highest PoC-generation success rate, while 34.0% is the highest vulnerability-patching success rate.
- Overall performance: SWE-agent and OpenHands achieve comparable performance, while Aider performs consistently worse across models and tasks.
- Failure analysis: Patching failures comprise No Patch, Improper Format, Compilation Error, and Still Vulnerable categories.
- Failure analysis: SWE-agent mainly encounters compilation errors and unresolved vulnerabilities, OpenHands has 62.18% of failures from improper formatting, and Aider has more no-patch failures.
- Failure analysis: Large contexts can exceed token budgets, oversized patches increase formatting errors, and incorrect root-cause identification leaves vulnerabilities unresolved.
- Task challenges: PoC generation remains difficult because effective payloads require byte-level memory interactions, extensive code-flow analysis, dedicated harness use, and debugger support.
4 Related work
Related benchmarks span CTF-based, vulnerability-based, software-engineering, and code-agent settings, but existing approaches often trade off realism, reproducibility, scale, or label quality.
- Cybersecurity benchmarks: CTF-based benchmarks test LLMs with challenge-based tasks but may not reflect real-world vulnerabilities and require manual construction.
- Cybersecurity benchmarks: Vulnerability-based benchmarks cover broad weakness categories or selected CVEs, but commonly lack reproducibility, scale, or accurate labels.
- Software engineering benchmarks: SWE-BENCH and its variants use real-world GitHub bug-fixing issues, with extensions increasing language diversity and task difficulty.
- Code agents: Code-agent research includes environment interfaces, pair-programming tools, staged frameworks, reinforcement learning, and interactive training environments.
5 Limitations and Future Work
SEC-bench currently prioritizes deterministic validation in C/C++ and memory-safety vulnerabilities, while identifying broader language and vulnerability coverage as future work.
- Current scope: SEC-bench focuses on C/C++ projects because memory-safety sanitizers provide reliable, objective verification.
- Current scope: Its current implementation covers a subset of vulnerability types detectable by memory-safety sanitizers.
- Future work: Future extensions could add language-specific tools and verification methods for languages and vulnerability classes beyond the current sanitizer-supported scope.
6 Conclusion
SEC-bench provides a comprehensive framework for evaluating LLM agents on security engineering tasks and exposes significant performance gaps in current code agents.
- SEC-bench evaluates LLM agents on security engineering tasks through the multi-agent SECVERIFIER framework.
- The framework processes, reproduces, and verifies software vulnerabilities to create high-quality benchmarks from unstructured bug reports.
- The evaluation reveals significant performance gaps in state-of-the-art code agents.
A Statistics on CVE Dataset
SEC-bench’s CVE dataset is characterized by many high- and critical-severity vulnerabilities, with memory-safety CWE classes especially prominent.
- CVSS scores concentrate in the high and critical ranges, especially 7.0–10.0, with notable concentrations around 7.75 and 9.75.
- Memory-safety vulnerabilities are predominant among the CWE types represented in the dataset.
- CWE-125 Out-of-bounds Read and CWE-787 Out-of-bounds Write are highly frequent vulnerability classes.
- Figure 3 presents CVSS-score distributions in its upper figure and CWE-type distributions in its lower figure.
B Evaluation Procedure
The evaluation procedure uses selected code-agent frameworks, sanitizer-based verdicts, standardized environments, and task-specific prompts for PoC generation and vulnerability patching.
- Model and agent selection: The evaluation selects SWE-agent, OpenHands, and Aider as representative state-of-the-art code-agent frameworks.
- Verification methodology: Memory-safety sanitizers provide deterministic runtime verdicts for PoC verification and patch validation.
- Evaluation environment: Identical Docker containers standardize dependencies, vulnerable codebases, compilation tools, and sanitizers across evaluations.
- PoC generation: PoC generation prompts require agents to create artifacts that trigger the sanitizer error specified in the issue description.
- Vulnerability patching: Patching prompts require minimal repository changes that resolve crash points described in the issue.
- SECVERIFIER prompts: SECVERIFIER uses specialized Builder, Exploiter, and Fixer prompts, with an integrated single-agent prompt for ablation comparison.
E.1 PoC Adaptation vs. From-Scratch Generation
Successful PoC construction overwhelmingly adapts information from existing bug reports rather than generating artifacts from scratch. The case study shows that iterative code analysis and sanitizer feedback can nevertheless produce a valid from-scratch PoC, while iteration limits constrain this capability.
- Adaptation versus generation: 289 instances produced successful PoC artifacts, but only 3 represented genuine from-scratch generation using GPT-4o.
- Adaptation versus generation: Most successful PoC cases adapted existing PoC information from bug reports rather than generating new artifacts from scratch.
- Limitations: The low rate of from-scratch generation was constrained by 75-iteration caps and limited opportunities for extended reasoning and trial-and-error.
- From-scratch generation process: From-scratch generation required analyzing sanitizer reports, examining multiple files, crafting precise binary inputs, and refining them through sanitizer feedback.
- Case study: In the libplist case, the agent analyzed the sanitizer log and vulnerable parsing code before identifying the binary index-calculation context.
- Case study: The initial malformed binary plist PoC failed to trigger a sanitizer error.
- Case study: After refinement, a new binary input triggered the expected heap-buffer-overflow with matching sanitizer output.
- Limitations: Remaining challenges include binary input crafting, larger iteration budgets, and reasoning about low-level memory operations.
F Agent Trajectory Analysis
Trajectory analysis examines SWE-agent tool usage across PoC generation and vulnerability patching tasks, using probability-density distributions to characterize agent behavior throughout execution.
- The analysis plots probability-density distributions of SWE-agent tool usage across trajectories for PoC generation and vulnerability patching.Figures 10 and 11 present the respective task statistics.
- Open-tool usage remains high at 24–30% throughout PoC-generation trajectories, while bash usage rises to 40–46% in later turns.The normalized proportions indicate sustained file reading alongside increasing execution attempts.
F.1 Key Observations and Insights
Agents repeatedly analyze codebases during security tasks, delay editing during PoC generation, show limited task-specific tool specialization, and lack dynamic debugging capabilities. These patterns correspond with substantially lower PoC-generation success than patching success.
- Sustained Codebase Analysis: Open-tool usage exceeds 20% throughout patching trajectories and remains 24–30% during PoC generation, unlike the distinct phases observed in general software engineering tasks.Security tasks require sustained codebase examination rather than sharply separated reproduction, editing, and submission phases.
- Delayed Action in PoC Generation: At turn 10, change usage is 0.3% for PoC generation versus 4.5% for patching; by turn 20, it reaches 2.1% versus 11.7%.PoC generation requires more exploration before editing because agents must infer triggering inputs rather than receive explicit vulnerable locations.
- Limited Tool Specialization: Bash usage rises to 40–46% in later PoC-generation turns versus 18–28% for patching, indicating greater reliance on trial-and-error execution during PoC crafting.Both tasks show sustained open and bash usage, while declining goto and increasing search_dir usage indicate broader, less targeted searches.
- Absence of Debugging Capabilities: Agents lack dynamic debugging tools for inspecting program state, memory layouts, and PoC payloads through stepwise execution.They therefore rely on static analysis and trial-and-error when precise, byte-level inputs are required.
- Absence of Debugging Capabilities: PoC generation achieves just over 10% success, whereas vulnerability patching achieves around 30% because static analysis and compilation feedback provide more actionable signals.The absence of runtime inspection particularly affects PoC validation.
F.2 Implications for Future Agent Design
The trajectory findings motivate security-focused agents with better context management, specialized program analysis, and task-specific tool integration. These needs reflect that security tasks require specialized architectures and tool ecosystems beyond general software-engineering workflows.
- Enhanced Context Management: Future agents need context summarization and caching to reduce redundant file reads and lengthy sanitizer or compilation outputs.Sustained file-reading activity indicates that agents spend substantial tokens on repeated context consumption.
- Specialized Program Analysis Capabilities: Specialized dataflow analysis, taint tracking, and call-graph navigation could identify vulnerability-relevant paths without exhaustive sequential file examination.These capabilities address the continuous codebase examination observed in security tasks.
- Task-Specific Tool Integration: PoC generation needs dynamic debugging, binary manipulation, and runtime inspection, while patching needs stronger static-analysis and semantic-code-understanding integration.The proposed tools are task-specific responses to differing requirements for exploit crafting and fix validation.
- Security tasks require specialized agent architectures and tool ecosystems to achieve human-level performance.The implication follows from their fundamentally different challenges compared with general software engineering.