PaperlayerView exact revision on arXiv

Source-linked AI summary

R2Vul: Learning to Reason about Software Vulnerabilities with Reinforcement Learning and Structured Reasoning Distillation

Martin Weyssow, Chengran Yang, Junkai Chen, Ratnadira Widyasari, Ting Zhang, Huihui Huang, Huu Hung Nguyen, Yan Naing Tun, Tan Bui, Yikun Li, Ang Han Wei, Frank Liauw, Eng Lieh Ouh, Lwin Khin Shar, David Lo

arXiv:2504.04699v2cs.SEcs.AIcs.CL

TL;DR

Vulnerability detection needs both accurate vulnerable-versus-safe judgments and explanations, yet LLM explanations can be shallow or wrong. R2Vul uses teacher-generated valid and flawed reasoning to preference-train small code models on a multilingual dataset. It reports improved performance over SFT and strong comparisons with larger and commercial models, while targeting function-level detection.

  • Problem

    Vulnerability detection requires binary judgments supported by sound explanations, but LLM-generated reasoning can be shallow or wrong and detection accuracy remains low.

    Source: §1 · p. 1

  • Method

    R2Vul preference-trains small code LLMs using teacher-generated valid and flawed security reasoning paired with vulnerability labels in a multilingual dataset.

    Source: §1 · pp. 1–2

  • Results

    Across five languages, the 1.5B R2Vul model improves macro-F1 over the strongest SFT baseline from 74% to 81%, surpasses its 32B teacher, and outperforms proprietary LLMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    R2Vul demonstrates that preference alignment of structured reasoning can support vulnerability detection and security-aware explanations in small models across five languages.

    Source: §1 · p. 2

    The study targets function-level detection and leaves interprocedural reasoning for future work.

    Source: §3 · p. 2

Abstract

from arXiv · show

Large language models (LLMs) have shown promising performance in software vulnerability detection, yet their reasoning capabilities remain unreliable. We propose R2Vul, a method that combines reinforcement learning from AI feedback (RLAIF) and structured reasoning distillation to teach small code LLMs to detect vulnerabilities while generating security-aware explanations. Unlike prior chain-of-thought and instruction tuning approaches, R2Vul rewards well-founded over deceptively plausible vulnerability explanations through RLAIF, which results in more precise detection and high-quality reasoning generation. To support RLAIF, we construct the first multilingual preference dataset for vulnerability detection, comprising 18,000 high-quality samples in C\#, JavaScript, Java, Python, and C. We evaluate R2Vul across five programming languages and against four static analysis tools, eight state-of-the-art LLM-based baselines, and various fine-tuning approaches. Our results demonstrate that a 1.5B R2Vul model exceeds the performance of its 32B teacher model and leading commercial LLMs such as Claude-4-Opus. Furthermore, we introduce a lightweight calibration step that reduces false positive rates under varying imbalanced data distributions. Finally, through qualitative analysis, we show that both LLM and human evaluators consistently rank R2Vul model's reasoning higher than other reasoning-based baselines.

1 Introduction

R2Vul addresses weak vulnerability-detection accuracy and unreliable explanations by aligning small code LLMs to prefer valid over flawed security reasoning. Its 18,000-sample multilingual dataset supports preference tuning, and the reported evaluations show strong detection and explanation-preference results.

Evaluation

  • Across five languages, the 1.5B R2Vul model improves macro-F1 over the strongest SFT baseline from 74% to 81%, surpasses its 32B teacher, and outperforms proprietary LLMs.Under extreme class imbalance, R2Vul also produces fewer false positives than SFT; lightweight threshold calibration further reduces false positives with modest recall loss.Source: §1 · p. 2
  • LLM evaluators and security experts preferred R2Vul explanations over CoT and MSIVD in up to 94% of a 100-sample qualitative evaluation.The broader comparison covers five languages and includes static analyzers, fine-tuning methods, reasoning models, and proprietary LLMs.Source: §1 · p. 2

R2Vul

  • R2Vul distills teacher-generated preferences between valid and flawed reasoning into small student LLMs, improving detection accuracy and reasoning capabilities.Unlike positive-only SFT, the contrastive signal teaches the student to recognize and reject misleading explanations.Source: §1 · pp. 1–2

Dataset

  • The 18,000-sample preference corpus pairs high-confidence vulnerability labels with structured reasoning across C#, JavaScript, Java, Python, and C.For each function, a teacher generates reasoning and a label-flipped counterpart supplies a flawed explanation for preference alignment.Source: §1 · pp. 1–2

2 Related Work

Prior vulnerability-detection work includes prompting and fine-tuning methods, but limitations in explanation quality, language coverage, and dataset quality motivate R2Vul's preference-based approach and multilingual corpus.

2.1 Reasoning and Instruction Tuning for VD

  • Prior CoT explanations can be unfaithful, while RAG adds security artifacts but lacks released replication packages in the cited studies.Earlier SFT approaches lacked structured reasoning or cross-lingual generalization, and MSIVD, VulLLM, and LLMVulExp focus on C/C++.Source: §2.1 · p. 2
  • R2Vul introduces RLAIF preference alignment between valid and flawed structured reasoning, supports five languages without inference-time retrieval, and reports stronger performance than prior approaches.The cited comparisons include MSIVD, VulLLM, CodeBERT, GPT-4.1, and Claude-4-Opus.Source: §2.1 · p. 2

2.2 VD Datasets

  • The 18,000-sample dataset pairs valid and flawed reasoning with explicit CWE/CVE metadata across five languages, with label correctness reported as high as 93%.Prior datasets are described as low-correctness, small, C/C++-restricted, or lacking CWE/CVE links needed for security-grounded reasoning.Source: §2.2 · p. 2

3 R2VUL: Structured Reasoning Distillation

R2Vul trains small code LLMs for function-level vulnerability detection and security-aware explanations through teacher-generated structured reasoning and preference-based distillation. Its paired valid and flawed traces supply contrastive supervision optimized with ORPO.

3 R2VUL: Structured Reasoning Distillation

  • R2Vul targets function-level detection and assumes access to a larger teacher model, with structured reasoning generation followed by RLAIF-based reasoning distillation.Interprocedural reasoning is left for future work.Source: §3 · p. 2

3.1 Structured Reasoning Generation

  • For each labeled function, a teacher generates a valid trace conditioned on its true label and a flawed trace by swapping that label.The paired traces create direct contrastive supervision; class-specific templates make the vulnerable and safe reasoning paths structurally clear.Source: §3.1 · pp. 2–3
  • Vulnerable-case reasoning identifies code constructs, explains vulnerability mechanisms and impact, and links issues to CWE/CVE context; safe-case reasoning justifies the absence of known issues.The teacher's vulnerable reasoning is grounded in the function and its linked CWE/CVE metadata.Source: §3.1 · p. 3

3.2 Reasoning Distillation with RLAIF

  • ORPO combines an SFT loss on valid reasoning with an odds-ratio preference term that favors valid over flawed explanations.This approach is presented as simpler to implement and cheaper to train than methods such as PPO or DPO, while achieving superior or comparable performance.Source: §3.2 · p. 3
  • R2Vul is presented as the first use of RLAIF with structured reasoning distillation for vulnerability detection.The method applies preference tuning to a domain-specific detection task rather than NLP or code generation.Source: §3.2 · p. 3

4 Experimental Setup

The evaluation uses a curated multilingual preference dataset, controlled training and test splits, multiple baseline families, and standard detection metrics. It also tests external Java generalization and confidence-threshold calibration.

Dataset

  • The preference dataset spans C#, JavaScript, Java, Python, and C, and pairs 18,000 labeled samples with teacher-generated valid and flawed reasoning traces.A manual check of 100 samples yielded 93% label correctness; the corpus uses 80/10/10 train/validation/test splits and includes experiments at varied class-imbalance ratios.Source: §4 · pp. 3–4

External Test Set

  • An independent Java test set contains 106 balanced functions, including 53 expert-verified vulnerable examples from non-overlapping commits or CVEs.The vulnerable examples were individually checked by two security-expert authors, who agreed on all 53.Source: External Test Set · p. 4

Baselines and Models

  • Baselines include four SAST tools, CLS, CoT, SFT, MSIVD, VulLLM, and CodeBERT, while R2Vul, SFT, CLS, and CoT are tested at 0.5B, 1.5B, and 7B student sizes.The comparison includes rule-based analyzers and LLM-based detection or explanation methods.Source: Baselines · p. 4, Teacher and Student LLMs · p. 4

Metrics and Calibration

  • Macro-F1, precision, recall, false positives, and false positive rate are the principal evaluation metrics.The calibration procedure compares likelihoods for vulnerable and safe reasoning-template prefixes and predicts vulnerable when confidence meets threshold τ.Source: Metrics · p. 4

5 Results and Analysis

R2VUL outperforms leading baselines across languages and generalizes to an external Java test set. Its performance scales with more training data, remains stronger than SFT under class imbalance, and supports a calibration trade-off between false positives and recall; evaluators also prefer its explanations.

5.1 Main Results

5.2 Data Ablation Study

  • At 25% data, SFT matches R2VUL, but at 100% R2VUL reaches 80.9% macro-F1 versus SFT’s 74.4%; R2VUL improves steadily as data increases while SFT stagnates.Figure 2 compares seed-averaged macro-F1 for the two methods across four training-data fractions and three random seeds.Source: §5.2 · p. 6, Figure 2 · p. 6

5.3 Class Imbalance Impact

  • At 1:1 imbalance, R2VUL yields 27.74% FPs versus SFT’s 35.48%, a 28% relative reduction; both FPRs plateau beyond 1:5, with R2VUL consistently lower.The comparison uses the Python test set; both methods still produce considerable false positives, especially under high imbalance.Source: §5.3 · p. 6, Figure 3 · p. 6

5.4 Model Calibration

  • On Python, τ=0.73 cuts FPR from 28% to 20% at 1:1 while retaining 77.56% recall and 78.57% F1; τ=0.96 reaches 20% FPR at 1:10 with 66.03% recall.The reported thresholds illustrate the precision–recall trade-off available by adjusting the inference decision threshold.Source: §5.4 · p. 7, Table 3 · p. 7, Figure 4 · p. 7

5.5 Qualitative Assessment of Reasoning

  • R2VUL earns near-perfect completeness and clarity scores with Claude-3.5-Sonnet; both LLM judges rank it first in up to 94% of cases, and human evaluators prefer it in most cases.The comparison evaluated R2VUL (1.5B), Qwen2.5-1.5B with CoT, and MSIVD using LLM judges and two security practitioners.Source: §5.5 · p. 7, Table 4 · p. 7

6 Conclusion and Future Work

R2VUL uses RLAIF with ORPO to distill high-quality structured reasoning from teacher-generated explanations, alongside an 18,000-sample multilingual preference dataset. Across five languages, it outperforms a range of tools and models, while the study identifies language and project types beyond its scope.

6 Conclusion and Future Work

  • Across five languages, R2VUL surpasses SAST tools, CoT, CLS, SFT, recent reasoning models, and proprietary LLMs including GPT-4.1 and Claude-4-Opus.A lightweight calibration step mitigates high false positives.Source: §6 · pp. 7–8
  • RLAIF with ORPO distills high-quality reasoning into student LLMs by contrasting valid and flawed structured explanations generated by a teacher LLM.The paper presents R2VUL as the first preference-tuning approach for vulnerability detection.Source: §6 · p. 7
  • The authors contribute an 18,000-sample high-quality multilingual preference dataset for vulnerability detection.The study targets C#, JavaScript, Java, Python, and C.Source: §6 · p. 7
  • Human and LLM evaluators prefer R2VUL’s explanations in more than 90% of cases.Source: §6 · p. 8
  • The study targets five languages and leaves Go, Rust, and mixed-language projects for future work.Future work could also investigate fine-tuning large-scale reasoning LLMs.Source: §6 · p. 8

A.1 Preference Dataset

The preference dataset is built from patched NVD vulnerabilities in five programming languages, with GPT-4o re-annotation used to select high-confidence vulnerable functions. The final dataset combines these with safe functions, is split by language, and receives valid and flawed reasoning generated by a teacher model.

A.1 Preference Dataset

  • The dataset mines NVD entries linked to GitHub vulnerability-fixing commits and covers C#, JavaScript, Java, Python, and C.Functions modified before a fix are initially labeled vulnerable, while other functions are labeled non-vulnerable.Source: Data Collection · p. 10
  • GPT-4o scores pre- and post-commit function pairs from 0 to 4, and the dataset retains vulnerable functions scoring 4.Processing filters test functions, deduplicates by MD5 hash, and removes functions longer than 4,096 tokens.Source: Data Collection · p. 10
  • Validated vulnerable functions are combined with safe functions to form a balanced dataset, then split into train, validation, and test sets at 80:10:10 while preserving language distributions.Valid and flawed reasoning is generated for all samples using Qwen2.5-Coder-32B-Instruct.Source: Data Collection · p. 10
  • In an expert sanity check, 93% of sampled vulnerable labels were confirmed correct, 1% rejected, and 6% marked uncertain.Three authors assessed 100 samples, with 20 from each language.Source: Annotation Sanity Check · p. 10

A.2 Implementation Details.

The experiments fine-tune CLS, SFT, and R2VUL using LoRA, with ORPO used for R2VUL. They run on a single H100 GPU, with specified training and inference settings.

A.2 Implementation Details.

  • CLS, SFT, and R2VUL use LoRA with r = 8, α = 16, and dropout 0.05; R2VUL uses ORPO with λ = 0.3.The learning rates are 5e-5 for CLS and 3e-4 for SFT/ORPO.Source: §A.2 · p. 11
  • Training uses batch sizes of 16 for CLS and 2 for SFT/ORPO, with maximums of 10 and 5 epochs, respectively.Source: §A.2 · p. 11
  • Experiments run on one NVIDIA H100 GPU with 80GB memory; inference uses temperature 0.2 and a maximum generation length of 2,048 tokens.Inputs are capped at 512 tokens for CodeBERT and 4,096 tokens for other LLMs, with prompts capped at 32,768 tokens.Source: §A.2 · p. 11

A.3 Main Results Details

The main-results analysis reports precision, recall, and F1, and tests R2VUL against SFT with paired bootstrap procedures. It also tracks ORPO validation reward accuracy and loss across epochs for three λ settings.

A.3 Main Results Details

  • Paired bootstrap tests compare R2VUL and SFT on per-language and macro-averaged metrics using 10,000 resamples.Each model’s predictions from three runs with identical seeds are concatenated before testing.Source: §A.3 · p. 11
  • Figure 5 plots validation reward accuracy and loss across epochs for ORPO with λ values 0.1, 0.5, and 1.0.Source: Figure 5 · p. 11
  • The study reports statistical significance at p < 0.05 for paired bootstrap tests across experiments.Source: §A.3 · p. 11

A.4 Hyperparameter Sensitivity Analysis

The analysis varies ORPO’s balance between supervised fine-tuning and preference tuning, then examines training dynamics and test macro-F1. Peak scores occur after four to five epochs, with the best result at λ = 0.6 after five epochs.

A.4 Hyperparameter Sensitivity Analysis

  • Across λ settings, validation loss begins rising after three epochs, alongside increasing reward accuracy, indicating overfitting during training.The authors say the small gap between λ = 0.5 and λ = 1.0 suggests that pushing λ higher may offer no added benefit.Source: §A.4 · p. 11
  • The highest test macro-F1 occurs at λ = 0.6 after five epochs, while lower and higher λ values perform worse.The authors note that their λ = 0.3 setting performed well, but differs from the best value observed in this analysis.Source: §A.4 · p. 11
  • Peak test macro-F1 scores consistently occur after four to five epochs across all tested λ values.The analysis varies λ to explore low-to-high contributions of preference tuning in ORPO.Source: §A.4 · p. 11

A.5 Additional Class Imbalance Experiments

Additional imbalance experiments find that R2Vul outperforms SFT across Java class-imbalance ratios. Qualitative examples show more precise reasoning in some cases, alongside misses by both models; the appendix also lists prompts used for relabeling, reasoning, inference, and assessment.

A.5 Additional Class Imbalance Experiments

  • R2Vul outperforms SFT at every tested class-imbalance ratio on the Java test set, with results similar to Python.The figure reports false-positive rates and corresponding false-positive counts at each ratio.Source: §A.5 · p. 11, Figure 7 · p. 12

A.6 Qualitative Analysis Examples

  • The examples report erroneous MSIVD outputs and conclude that preference tuning improves precision, while both compared models retain failure cases.The authors characterize comprehensive vulnerability reasoning as complex.Source: §A.6 · p. 12
  • In the C# example, R2Vul identifies insufficient URL escaping as an injection risk, while Qwen2.5-1.5B misses the vulnerability.The comparison concerns the vulnerable function shown in Table 8.Source: §A.6 · p. 12, Table 8 · p. 13
  • For Java path traversal, R2Vul explains how unsanitized input can create directories in unauthorized locations, while Qwen2.5-1.5B misses the flaw.R2Vul maps the issue to CWE-22; Qwen2.5-1.5B instead focuses on code-quality issues without security implications.Source: §A.6 · p. 12, Table 9 · p. 15
  • In the JavaScript example, neither model identifies the type-confusion vulnerability that enables XSS through an attacker-supplied object.Both give incorrect explanations, and both miss the input-type validation added by the patch.Source: §A.6 · p. 12, Table 10 · p. 16
  • For Python, R2Vul identifies arbitrary-code execution risk from untrusted pickle data, while Qwen2.5-1.5B overlooks the core deserialization threat.R2Vul maps the vulnerability to CWE-502.Source: §A.6 · p. 12, Table 11 · p. 17
  • In C, both models miss the infinite loop caused by negative fread returns satisfying the loop condition; R2Vul is marginally closer in code focus.R2Vul instead alleges a buffer-overflow risk in fwrite, while Qwen2.5-1.5B gives a generic list of possible issues.Source: §A.6 · p. 12, Table 12 · p. 18

A.7 Prompt Templates

  • The appendix lists prompts for dataset relabeling, structured-reasoning generation, model inference, and LLM-as-a-judge assessment.Tables 13–16 contain the relabeling, reasoning, inference, and assessment prompts, respectively.Source: §A.7 · p. 12, Table 13 · p. 19, Table 14 · p. 19
Loading 2504.04699v2…