Source-linked AI summary
RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories
Yanlin Wang, Ziyao Zhang, Chong Wang, Xinyi Xu, Mingwei Liu, Yong Wang, Jiachi Chen, Zibin Zheng
TL;DR
Existing secure-code benchmarks often miss repository-scale interactions between functionality and security. This paper introduces RealSec-bench, a real-world Java benchmark built through SAST, LLM filtering, and human validation, and finds that current LLMs struggle to produce code that is simultaneously functional and secure while RAG and security-guideline prompting offer inconsistent benefits.
Problem
Existing benchmarks often rely on synthetic examples or isolated snippets, limiting evaluation of vulnerabilities arising from inter-procedural dataflows in real repositories.
Method
The paper constructs RealSec-bench from high-risk Java repositories using SAST scanning, reproducibility checks, LLM-based false-positive elimination, expert validation, and security-neutral docstring standardization.
Results
SecurePass@1 remains below 6% across all subjects, while RAG provides marginal and inconsistent improvements and security-guideline prompting can compromise functional correctness.
Takeaways & Limitations
Current models can handle localized code-quality issues but show a substantial gap between generating functionally correct code and generating code that is verifiably secure.
Takeaways & Limitations
The benchmark is restricted to open-source Maven-based Java repositories, and its SAST-based security assessment remains constrained by false positives and current LLM adjudication accuracy.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area. Existing benchmarks often fall short by relying on synthetic vulnerabilities or evaluating functional correctness in isolation, failing to capture the complex interplay between functionality and security found in real-world software. To address this gap, we introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories. Our methodology employs a multi-stage pipeline that combines systematic SAST scanning with CodeQL, LLM-based false positive elimination, and rigorous human expert validation. The resulting benchmark contains 105 instances grounded in real-word repository contexts, spanning 19 Common Weakness Enumeration (CWE) types and exhibiting a wide diversity of data flow complexities, including vulnerabilities with up to 34-hop inter-procedural dependencies. Using RealSec-bench, we conduct an extensive empirical study on 5 popular LLMs. We introduce a novel composite metric, SecurePass@K, to assess both functional correctness and security simultaneously. We find that while Retrieval-Augmented Generation (RAG) techniques can improve functional correctness, they provide negligible benefits to security. Furthermore, explicitly prompting models with general security guidelines often leads to compilation failures, harming functional correctness without reliably preventing vulnerabilities. Our work highlights the gap between functional and secure code generation in current LLMs.
1 Introduction
RealSec-bench addresses the limited real-world applicability of secure-code benchmarks by evaluating repository-level code generation with realistic inter-procedural vulnerabilities. Its study finds persistent security limitations: models may handle localized issues but struggle to generate code that is both functional and secure, while RAG and prompt-based security strategies lack universal effectiveness.
- Existing benchmarks often use synthetic examples or isolated snippets, missing repository-scale inter-procedural dataflows and cross-module security risks.
- RealSec-bench is constructed from high-risk Java repositories using SAST-based candidate extraction, reproducibility checks, LLM filtering, and human validation.
- SecurePass@1 remains below 6% across all subjects, showing that current models rarely achieve functional correctness and security simultaneously.
- 0.00% success rate occurs in complex domains such as cryptography, despite models addressing localized code-quality issues.
- RAG and prompt engineering do not provide universal solutions and can introduce trade-offs that degrade functional correctness.
2 RealSec-bench Construction
RealSec-bench is built through a multi-phase process that selects influential, high-risk Java repositories and transforms broad SAST findings into executable, standardized, human-verified benchmark instances.
- The construction process has two primary phases: high-risk repository selection and vulnerability instance construction and refinement.
- The initial collection starts from the top 4,000 most-starred Java repositories, then filters for Maven projects and topic diversity.
- 532 high-risk repositories are selected using CodeQL vulnerability counts and manual CWE balancing.
- More than 20,000 candidate findings are extracted with a high-recall CodeQL configuration before subsequent verification.
- Build checks, unit-test coverage, LLM false-positive filtering, expert review, and security-neutral docstring rewriting produce reproducible standardized instances.
3 Benchmark Characteristics
The benchmark contains 105 task instances from 30 Java repositories, covering 19 CWE types and a broad range of vulnerability prevalence and inter-procedural dataflow complexity.
- 105 task instances from 30 Java repositories are organized with repository metadata, source code, vulnerability reports, and validation tests.
- The benchmark spans 19 distinct CWE vulnerability types.
- Log Injection is the most frequent category, accounting for 56.2% of the dataset.
- Broken cryptographic algorithms and missing CSRF protection each account for 6.7% of instances.
- Rare but severe categories include XXE injection, deserialization, path traversal, and command injection, each occurring at 0.9%.
- 21.0% of instances have dependencies exceeding three hops, with extreme cases reaching 34 hops.
4 Evaluation Setup
The evaluation tests five LLMs with baseline, retrieval-augmented, and security-guideline prompting, using functional, security, and composite metrics. Secure@k uses a two-stage SAST and LLM-adjudication process, while SecurePass@k requires both functional correctness and security.
- Five popular LLMs are evaluated under standardized generation settings and repository-level prompts.
- The study compares origin generation, BM25 retrieval, dense RLCoder retrieval, SAST-based dataflow retrieval, and security-guideline prompting.
- Pass@k measures whether at least one of k generated samples passes the unit tests.
- Secure@k classifies a sample as secure after CodeQL scanning and secondary multi-LLM adjudication of detected vulnerabilities.
- SecurePass@k requires a generated sample to be both functionally correct and secure.
- The adjudication validation raises precision from 44.9% to 81.7% and recall to 98.0%, producing an F1-Score of 89.1%.
5 Evaluation Results & Analysis
RealSec-bench exposes substantial difficulty in simultaneously generating functional and secure repository-level code. Performance varies by vulnerability type, dependency complexity, retrieval strategy, and security prompting, with improvements often inconsistent or accompanied by functional trade-offs.
- Overall Performance: The top-performing model reaches only 16.19% Pass@1, while no model exceeds 8% SecurePass@1.These results reflect the benchmark’s combined functional and security requirements.
- Vulnerability Tasks: GPT-4.1 achieves 80% SecurePass@1 on Code Quality & Security, whereas cryptographic tasks produce near-zero secure success despite 38.89% Pass@1 for Claude-3.7-Sonnet.The results show large variation across vulnerability categories.
- Inter-procedural Complexity: GPT-4.1 reaches its highest SecurePass@1 of 5.7% on 0-hop tasks, while performance drops for many 1-hop cases and declines again at three or more hops.Two-hop performance can recover for some models, with Qwen3-235B reaching 1.9% SecurePass@1.
- Retrieval-Augmented Generation: RAG significantly improves functional correctness but provides no substantial security improvement overall, with model-specific SecurePass@1 peaks of 7.6%.Dense retrieval reaches 7.6% for Claude-3.7-Sonnet, while BM25 reaches 7.6% for Qwen3-235B.
- Retrieval-Augmented Generation: The dataflow retriever can omit utility functions and class hierarchies, making broader text-based retrieval more effective for valid, compilable solutions.Its narrow vulnerability-path focus does not always provide the broader repository context needed for generation.
- Security Prompting: Security-oriented prompting lacks uniform benefits: it improves some models but reduces Claude-3.7-Sonnet’s Pass@1 from 16.2% to 9.5%.The reported trade-off is reduced functional correctness under added security constraints.
6 Related Work
Prior code-generation benchmarks have progressed toward repository-level evaluation, but security benchmarks often remain synthetic or simplified. Retrieval and program-analysis methods improve contextual modeling while largely prioritizing functional correctness over security.
- Code Generation Benchmarks: Existing security-focused benchmarks frequently use synthetic data or simplified scenarios that omit complex cross-file dependencies.This limits their representation of repository-level security problems.
- Repository-level Code Generation: Repository-level generation increasingly uses RAG, agent tools, and program analysis to capture dataflow and control-flow structures.These approaches include dependency graphs and iterative refinement.
- Repository-level Code Generation: These methods generally prioritize functional correctness and can still produce vulnerable code, motivating evaluation of security-aware retrieval and prompting.The paper positions RealSec-bench as an empirical test of these security enhancements.
7 Conclusion
RealSec-bench is a security-focused benchmark built from Java repositories using SAST analysis and multi-stage human validation. Its evaluation shows that models often produce functionally passing code that retains vulnerabilities, while complex multi-function security reasoning remains difficult.
- Conclusion: RealSec-bench is constructed from Java repositories through SAST analysis and multi-stage human validation.The benchmark is designed for security-focused code-generation evaluation.
- Conclusion: Current models often generate solutions that pass functional tests while retaining critical vulnerabilities.The paper identifies this as a central gap between functional and secure code generation.
- Conclusion: Models struggle with vulnerability types requiring data-flow understanding across multiple functions and security contexts.The conclusion links these difficulties to complex, non-local reasoning requirements.
- Conclusion: The paper explores retrieval-augmented generation and advanced security-guideline prompting as directions for improving model performance.These strategies are presented as investigated approaches rather than universally effective solutions.
Limitations
The benchmark’s validity and generalizability are constrained by its reliance on SAST-based assessment and its focus on open-source, Maven-based Java repositories. The benchmark covers diverse CWE categories and security principles, but its scope does not extend directly to other languages or proprietary software.
- Limitations: SAST-based assessment introduces false-positive risk, and multi-LLM adjudication remains less accurate than manual expert annotation.The authors identify dynamic analysis and improved judging frameworks as future verification directions.
- Limitations: The benchmark is restricted to open-source, Maven-based Java repositories, limiting generalizability to other programming languages and proprietary software.This is the paper’s stated external-validity limitation.
- Security Guidelines: The security-prompting method maps benchmark CWE types to OWASP principles and embeds universal, context-independent guidance into generation.The directives address input/output integrity, access control, cryptography, operational security, and system configuration.
- Benchmark Scope: RealSec-bench contains 19 vulnerability patterns organized into four domains, including injection, file-system manipulation, cryptography and access control, and numeric or concurrency errors.The categorization provides granular coverage of security challenges.
B.2 Detail of Multi-hop Dependency Tasks
RealSec-bench spans mostly localized vulnerabilities but also includes a substantial long tail of multi-hop inter-procedural dependencies, reaching 34 hops. This distribution tests both basic vulnerability repair and deeper repository-level dataflow tracing.
- The most complex vulnerabilities require tracing paths of up to 34 hops from source to sink.
- 37 tasks (35.2%) are zero-hop vulnerabilities with direct source-to-sink dataflows.
- 25 tasks (23.8%) require one hop, 14 (13.3%) require two hops, and 7 (6.7%) require three hops.
- Nearly 80% of tasks involve zero to three hops, supporting evaluation of common and localized vulnerability fixes.
- 22 tasks (21.0%) exceed three hops, with dependencies extending through multiple calls, class boundaries, and control flows.
C.1 Performance in Different Hop Tasks.
Model performance varies non-linearly with inter-procedural dependency length. Models perform best on zero-hop tasks, show irregular recovery at two hops, and generally decline on dependencies of three or more hops.
- Claude-3.7-Sonnet reaches its highest SecurePass@1 score, 16.22%, on zero-hop tasks.
- Several strong models, including Claude-3.7-Sonnet and GPT-4.1, score 0.00% SecurePass@1 on one-hop tasks.
- Qwen3-235B and Claude-3.7-Sonnet reach 21.43% SecurePass@1 on two-hop tasks.
- Performance generally declines on tasks with three or more hops, reflecting limited long-range dependency tracing and security-context maintenance.
C.2 Performance in RAG Method
RAG substantially improves functional correctness but provides no significant security gain, with retrieval effectiveness varying across models and methods. Security-guideline prompting likewise produces model-dependent results rather than a universal improvement.
- RAG significantly improves functional correctness but offers no significant gain in security.
- No retrieval method is universally superior across models.
- Dense retrieval yields Claude-3.7-Sonnet’s highest SecurePass@1 score of 10.48%, while bm25 yields GPT-4.1-mini’s highest score of 8.57%.
- The inter-procedural dataflow retriever may omit utility functions, inheritance structures, or idiomatic patterns needed for compilable and functionally valid code.
- Security-guideline prompting raises SecurePass@1 from 4.76% to 8.57% for Deepseek-V3 and from 6.67% to 9.52% for GPT-4.1, but overall gains are not significant across models.