PaperlayerView exact revision on arXiv

Source-linked AI summary

Secure Code Generation via Online Reinforcement Learning with Vulnerability Reward Model

Tianyi Wu, Mingzhe Du, Yue Liu, Chengran Yang, Terry Yue Zhuo, Jiaheng Zhang, See-Kiong Ng

arXiv:2602.07422v1cs.CRcs.AIcs.CL

TL;DR

LLM-generated code often contains serious security vulnerabilities, while existing alignment methods can trade functionality for security. SecCoderX repurposes vulnerability-detection data into realistic RL prompts and a reasoning-based reward model, integrating both in online reinforcement learning. The framework reports improved security and Effective Safety Rate while preserving functional correctness.

  • Problem

    LLM-generated code frequently contains serious security vulnerabilities, and existing security-alignment methods can substantially reduce functionality.

  • Method

    SecCoderX synthesizes vulnerability-inducing coding tasks, trains a reasoning-based vulnerability reward model, and combines them in an online reinforcement-learning loop.

  • Results

    SecCoderX improves security and Effective Safety Rate without compromising functional correctness.

  • Takeaways & Limitations

    Repurposing vulnerability-detection resources provides the framework’s security supervision and realistic prompts for functionality-preserving secure code alignment.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used in software development, yet their tendency to generate insecure code remains a major barrier to real-world deployment. Existing secure code alignment methods often suffer from a functionality--security paradox, improving security at the cost of substantial utility degradation. We propose SecCoderX, an online reinforcement learning framework for functionality-preserving secure code generation. SecCoderX first bridges vulnerability detection and secure code generation by repurposing mature detection resources in two ways: (i) synthesizing diverse, reality-grounded vulnerability-inducing coding tasks for online RL rollouts, and (ii) training a reasoning-based vulnerability reward model that provides scalable and reliable security supervision. Together, these components are unified in an online RL loop to align code LLMs to generate secure and functional code. Extensive experiments demonstrate that SecCoderX achieves state-of-the-art performance, improving Effective Safety Rate (ESR) by approximately 10% over unaligned models, whereas prior methods often degrade ESR by 14-54%. We release our code, dataset and model checkpoints at https://github.com/AndrewWTY/SecCoderX.

1. Introduction

LLMs are increasingly used for software development, but generated code often contains serious security vulnerabilities. Existing security-alignment methods can improve security while substantially reducing functionality, making Effective Safety Rate a practical concern.

  • Effective Safety Rate evaluates secure utility by accounting for both security and functional correctness.
  • LLM-generated code frequently contains serious security vulnerabilities that create risks for downstream systems.
  • Prior secure code alignment methods often improve security at a substantial cost to generated-code functionality.
  • Many models aligned by prior methods underperform their unaligned counterparts on Effective Safety Rate.

C. Online Secure Code Generation Alignment

SecCoderX repurposes vulnerability-detection resources into online-RL prompts and a vulnerability reward model, then combines them in a training pipeline. The framework targets secure code generation while preserving functionality and reports gains over prior approaches.

  • Task and reward-resource construction: SecCoderX synthesizes vulnerability-inducing prompts from vulnerability-detection datasets for online reinforcement-learning rollouts.
  • Task and reward-resource construction: It trains a reasoning-based, CWE-conditioned vulnerability reward model to provide scalable security supervision.
  • Online RL alignment: The synthesized prompts and reward model are integrated into online reinforcement learning with a reward design targeting security and functional correctness.
  • Resources and empirical outcome: SecCoderX releases a dataset of 24k prompts spanning 24 CWE categories and 5 programming languages.
  • Resources and empirical outcome: SecCoderX improves effective secure code generation by approximately 10% over prior methods while avoiding severe ESR degradation reported for existing approaches.

2. SecCoderX Framework

SecCoderX repurposes vulnerability detection resources into vulnerability-inducing prompts and a CWE-conditioned reward model, then uses them for online RL secure-code alignment. Its synthesis pipeline expands realistic vulnerability examples across repository contexts, languages, and CWE categories.

  • Framework overview: SecCoderX uses vulnerability detection datasets to create vulnerability-inducing prompts and a vulnerability reward model for online RL.These resources address the lack of naturally eliciting coding tasks and provide security supervision for policy rollouts.
  • Reality-grounded task synthesis: Gemini-2.5-Pro infers multiple plausible repository contexts in which each vulnerable snippet could naturally arise as a functional component.Mapping one vulnerability pattern to multiple software contexts increases contextual diversity.
  • Reality-grounded task synthesis: SecCoderX assigns one of five target languages and generates prompts aligned with each repository context while likely to induce the associated CWE vulnerability.The target languages are C, C++, Java, JavaScript, and Python.
  • Reality-grounded task synthesis: 24k prompts span 24 CWE categories and 5 programming languages and serve as online RL rollout prompts.The dataset is formalized in Algorithm 1 and illustrated in Figure 2.
  • CWE-conditioned vulnerability reward model: The CWE-conditioned reward model is trained through dataset collection, reasoning-based vulnerability-detection SFT, and online RL generalization.At inference, it generates structured reasoning followed by a binary vulnerability-status prediction.
  • CWE-conditioned vulnerability reward model: Across PrimeVul, R2Vul, SVEN, and ProSec, the 8B reward model consistently ranks first or second in F1 and achieves the best overall performance.It also surpasses larger commercial models in overall F1 score and is conditioned on the target CWE category.

3. Experiment Setup

The evaluation measures security, functionality, and general coding performance across secure-code and standard code-generation benchmarks. Effective Safety Rate combines security with functionality to assess practically useful secure code.

  • Secure code generation evaluation: Secure code generation is evaluated on CyberSecEval SCG and CWEval across C, C++, Java, JavaScript, and Python.The language coverage matches prior work such as ProSec and SafeCoder.
  • Security metrics: Safety score is the proportion of test prompts for which generated code is classified as non-vulnerable at temperature 0.The benchmark vulnerability checker determines whether each generated response is secure.
  • Functionality metrics: Functionality is measured by unit-test pass fractions on CWEval and by Gemini-2.5-Flash scores normalized to [0,1] on CyberSecEval SCG.CyberSecEval SCG uses a discrete judge score from 0 to 5 because it lacks unit tests.
  • General coding evaluation: HumanEval+ and MBPP+ report pass@1 and pass@10 to test whether secure alignment preserves general coding performance.These benchmarks evaluate instruction-following code generation beyond security-focused tasks.
  • Combined metric: Effective Safety Rate discounts security successes on functionally defective code and measures code that is simultaneously secure and functional.The metric treats security without functionality as insufficient for practical utility.
  • Baselines and models: SecCoderX is compared with SafeCoder and ProSec on CodeLlama-7B-Instruct, Qwen2.5-Coder-3B-Instruct, and Qwen2.5-Coder-7B-Instruct.All three are instruction-tuned open-source code models.

4. Discussion and Takeaways

SecCoderX mitigates the functionality–security paradox through online RL and a composite reward design. Its vulnerability, length, AST, and format rewards play distinct roles in balancing secure generation with functional correctness.

  • Performance Analysis: SecCoderX achieves higher Effective Safety Rate while maintaining on-par or better Safety rate than prior alignment baselines.ProSec and SafeCoder improve raw safety but substantially reduce functionality, whereas SecCoderX preserves pre-alignment utility.
  • Reward Model Ablation: Reasoning-based SFT substantially improves vulnerability detection compared with standard SFT without reasoning traces.The ablation identifies structured detection reasoning as critical for accurate vulnerability identification.
  • Reward Model Ablation: GRPO improves recall and F1 on unseen code, indicating stronger vulnerability-detection generalisation than SFT-only training.The reported comparison is from “with Reasoning SFT” to “Full.”
  • Reward Model Ablation: CWE-conditioning produces the best overall vulnerability reward model detection performance by tailoring reasoning to the target vulnerability category.The full model outperforms the version without CWE-conditioning.
  • Online RL Reward Ablation: The vulnerability reward drives security, while length and AST rewards constrain exploration to preserve functional code structure.Removing vulnerability reward sharply reduces Safety rate and ESR; removing AST reward harms functionality, while length regularizes rollout exploration.
  • Online RL Reward Ablation: Format reward has limited impact but provides a low-cost safeguard against malformed outputs.Instruction-tuned code models already generally follow formatting conventions.

5. Related Work

Related work spans functional code generation, secure-code alignment, and vulnerability detection. SecCoderX builds on these areas by redirecting vulnerability-detection resources toward secure code generation.

  • LLMs for Code Generation: LLMs achieve strong functional and efficient code generation through large-scale code pre-training and instruction fine-tuning.Recent work also uses reasoning-based RL with verifiable rewards such as unit-test execution to improve coding-task generalization.
  • Secure Code Generation Alignment: Secure-code alignment studies address the persistent risk that generated code contains vulnerabilities even as functional generation improves.The cited literature frames security as a continuing concern for increasingly capable code models.
  • Vulnerability Detection: Large-scale vulnerability-detection datasets contain millions of real-world examples but have traditionally trained classifiers rather than guided code generation.Their potential for secure-code alignment has been largely overlooked.

6. Conclusion

SecCoderX is an online RL framework that repurposes vulnerability-detection resources to align code LLMs toward secure, functional generation. Experiments report improved safety and ESR relative to unaligned models and prior methods.

  • Conclusion: SecCoderX aligns code LLMs with a CWE-conditioned, reasoning-augmented vulnerability reward model and synthesized vulnerability-inducing tasks.These resources support scalable security supervision and online RL rollouts.
  • Conclusion: 11%–16% higher safety rates than the unaligned model are reported across experiments.The result is stated as a consistent comparison against the unaligned model.
  • Conclusion: Approximately 10% ESR improvement is reported for SecCoderX, whereas prior methods consistently degrade ESR by 14%–54%.The paper presents this as a step toward functionality-preserving secure code alignment.

Impact Statement

The paper frames SecCoderX as an effort to improve the security and reliability of LLM-generated code. It also acknowledges that code-generation advances can be misused to automate insecure or malicious software production.

  • Impact Statement: The work aims to improve the security and reliability of code generated by large language models.Its stated goal is to address the functionality–security trade-off in secure code generation.
  • Impact Statement: Addressing the functionality–security trade-off may reduce vulnerability propagation and improve the safety and trustworthiness of AI-assisted software development.The paper presents these as potential benefits rather than guaranteed outcomes.
  • Impact Statement: The methods could be misused to automate insecure or malicious software artifacts at scale if deployed irresponsibly.The authors state that they focus on vulnerability reduction and release models and datasets for research purposes only.

C.1. Training Details

SecCoderX trains its vulnerability reward model through reasoning-based SFT followed by online GRPO, then aligns secure code generation with synthesized vulnerability-inducing tasks and composite rewards.

  • Vulnerability Reward Model Training: The reward model begins with reasoning-based SFT to build vulnerability detection capability and explicit reasoning.
  • Vulnerability Reward Model Training: The SFT dataset contains 37k samples spanning 144 CWE categories and nine programming languages.
  • Vulnerability Reward Model Training: Online GRPO further trains the reward model on R2Vul labels across 265 CWE categories and five programming languages.
  • Vulnerability Reward Model Training: The reward model receives binary feedback when its vulnerability prediction matches the ground-truth label, encouraging reasoning that supports correct judgments.
  • Secure Code Generation Alignment: Secure code alignment uses approximately 24k synthesized vulnerability-inducing prompts, filtered reference solutions, and online GRPO rollouts.
  • Secure Code Generation Alignment: The composite reward combines format, vulnerability, length, and AST-similarity signals before the policy update.

C.2. Evaluation Details

The evaluation covers vulnerability detection, secure code generation, functionality, and synthesis quality using controlled prompts, multiple benchmarks, and judge validation.

  • Vulnerability Detection: Vulnerability detection is evaluated against strong open and proprietary baselines on PrimeVul, R2Vul, SVEN, and ProSec.
  • Secure Code Generation: Secure code generation is tested on CyberSecEval SCG and CWEval across C, C++, Java, JavaScript, and Python.
  • Secure Code Generation: Safety Rate, functionality, and Effective Safety Rate are jointly reported to capture security alongside practical utility.
  • Functionality Evaluation: CyberSecEval generations use a fixed Gemini-2.5-Flash judge, enabling consistent relative comparisons across methods.
  • Functionality Evaluation: Human verification of 50 randomly sampled solutions found that Gemini-2.5-Flash assessed functional validity correctly in the vast majority of cases.
  • Task Synthesis: Reality-grounded task synthesis uses strong general-purpose LLMs, adaptive expansion, stratified sampling, and produces approximately 24k prompts across 24 CWE categories.
  • Structural Similarity: ASTSim compares candidate and reference abstract syntax trees after removing leaf nodes, measuring structural rather than naming similarity.

D.1. Vulnerability Reward Model Before and After SecCoderX Training

The case studies contrast vulnerability reasoning before and after SecCoderX training and secure code before and after alignment, illustrating more focused detection and bounded implementations.

  • Vulnerability Reward Model: The CWE-787 case study compares Qwen3-8B vulnerability reasoning before and after SecCoderX training.
  • Vulnerability Reward Model: After training, the reward model identifies the vulnerable code location, provides focused reasoning, and reaches the correct vulnerability conclusion.
  • Vulnerability Reward Model: The original analysis identifies several possible weaknesses, including CWE-787 out-of-bounds write, CWE-125 out-of-bounds read, and CWE-190 integer overflow.
  • Vulnerability Reward Model: The analyzed function aggregates values per batch and writes to a per-batch counts structure indexed by tensor-derived batch and value indices.
  • Vulnerability Reward Model: The code checks that batch is below num_batches, but the case-study analysis states that the write occurs before this check and that value lacks equivalent range validation.
  • Secure Code Generation: The secure-generation case study compares Qwen2.5-Coder-7B outputs before and after SecCoderX alignment on a CyberSecEval SCG task.
  • Secure Code Generation: Before alignment, malloc combined with strcpy and strcat permits unbounded copying and creates CWE-122 heap-buffer-overflow risk.
  • Secure Code Generation: After alignment, the model computes the input length once, allocates input_len + 2 bytes, copies bounded content, and writes the slash and terminator explicitly.

F. Additional Results

Additional results report ablations and comparisons with larger or proprietary models, showing the roles of reward components and SecCoderX’s competitive secure-generation performance.

  • Ablations: Ablation results evaluate how vulnerability reward-model training stages and reward components affect performance.
  • Model Comparisons: 35.82 ESR versus 31.98 shows SecCoderX-tuned Qwen2.5-Coder-7B outperforming the larger PurpCode-14B model.
  • Model Comparisons: SecCoderX achieves comparable Safety% to closed-source models while those models retain higher ESR because of stronger base functionality.
Loading 2602.07422v1…