PaperlayerView exact revision on arXiv

Source-linked AI summary

From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detection

Youpeng Li, Fuxun Yu, Xinda Wang

arXiv:2602.14012v2cs.CRcs.AIcs.SE

TL;DR

Post-training for LLM-based vulnerability detection remains underexplored despite its use in general coding tasks. This paper systematically investigates the pipeline from SFT through preference optimization and on-policy RL, finding that GRPO outperforms common alternatives and identifying VD-specific training and evaluation guidance.

  • Problem

    How post-training methods, data choices, stage interactions, rewards, and evaluation protocols affect LLM-based vulnerability detection remains insufficiently studied.

    Source: §1 · p. 2

  • Method

    The paper systematically studies cold-start SFT, off-policy preference optimization, and on-policy RL, alongside data curation, reward design, and evaluation choices.

    Source: §1 · pp. 2–3

  • Results

    GRPO models surpass SFT, preference-optimization methods, and specialized VD LLMs, while rejection sampling, root-cause-aligned rewards, and stage-aware choices yield VD-specific gains.

    Source: §1 · p. 3, §8 · p. 22

  • Takeaways & Limitations

    The findings support treating VD post-training as a multi-stage pipeline, with different data, initialization, reward, and evaluation choices suited to different stages.

    Source: §1 · p. 3

    Experiments focus on the Qwen3-4B backbone and C/C++ data, and potential pre-training/evaluation leakage may bias absolute performance.

    Source: §7 · p. 22

Abstract

from arXiv · show

The integration of LLMs into vulnerability detection (VD) has shifted the field toward more interpretable and context-aware analysis. While post-training techniques have shown promise in general coding tasks, their systematic application to VD remains underexplored. In this paper, we present the first comprehensive investigation into the post-training pipeline for LLM-based VD, demonstrating that on-policy RL with GRPO consistently outperforms SFT, off-policy preference optimization methods, and specialized VD LLMs. Our study further reveals VD-specific post-training guidelines and insights beyond common practices: (1) For data curation, contrary to the widespread use of rationalization-based supervision in prior VD work, SFT based on rejection sampling proves more effective, as rationalization can introduce hallucinations; in RL training, the inherently skewed difficulty distribution of vulnerabilities leads difficulty-aware data filtering to drastically reduce data coverage, causing non-negligible performance loss, and undermines curriculum learning, while pair-based data scheduling can partially mitigate this. (2) For stage interactions, unlike preference optimization typically applied to lightly trained SFT models, increasing SFT epochs consistently benefits off-policy preference optimization in VD tasks; however, excessive SFT suppresses self-exploration in on-policy RL, limiting its gains. (3) For reward mechanisms, naively treating vulnerability classification correctness as reward signals leads to reward hacking, whereas fine-grained root-cause judgments provide more reliable credit assignment; specification-based rewards further improve efficiency at the cost of additional design and generation effort. (4) For evaluation protocols, LLM-as-a-Judge based on root-cause analysis offers a more robust alternative, albeit with variability across judge models.

1. Introduction

The paper addresses the underexplored use of post-training for vulnerability detection (VD), where existing approaches face limits in model capabilities and training and evaluation practices remain unsettled. It systematically studies the pipeline and reports that GRPO and several VD-specific choices outperform or improve on common alternatives.

1. Introduction

  • Prompt-based VD approaches are bounded by base-model code understanding and instruction following, while proprietary reasoning concealment and large model scale constrain some deployments.The passage particularly notes resource-constrained settings and sensitive codebase analysis.Source: §1 · p. 2
  • The paper identifies five unresolved areas in VD post-training: evaluation scope, data curation, stage interaction, reward design, and evaluation protocols.These include whether the best SFT checkpoint is also the best initialization for later stages and how reward granularity affects RL.Source: §1 · p. 2
  • The study examines data curation, stage interactions, reward mechanisms, and evaluation protocols across cold-start SFT, off-policy preference optimization, and on-policy RL.It also introduces a context-level VD dataset and quantifies differences among LLM judges and correctness criteria.Source: §1 · pp. 2–3
  • GRPO models significantly outperform SFT, DPO, ORPO, and specialized VD LLMs, while rejection-sampled CoT, root-cause rewards, and pair-based scheduling improve specific pipeline stages.Rationalization can introduce hallucinated vulnerability information; excessive SFT can suppress the exploration needed by on-policy RL.Source: §1 · p. 3
  • The paper contributes OpenVul, an open-source post-training framework with code, datasets, and models, alongside VD-specific guidelines for future research.Its investigation spans cold-start SFT through off-policy preference optimization and on-policy RL.Source: §1 · p. 3

2. Preliminaries

The preliminaries describe how supervised and preference-based methods use fixed supervision, while GRPO uses feedback from current-policy rollouts. They also motivate a context-aware dataset by outlining limitations in existing VD datasets and the paper’s context-enrichment approach.

2.1. Problem Formulation.

  • The vulnerability-reasoning dataset is represented by queries paired with preferred and dispreferred responses, typically generated by prompting strong teacher LLMs.The notation distinguishes the query set from the two response sets.Source: §2.1 · p. 4
  • SFT learns from queries and chosen responses, DPO optimizes pairwise preferences using an SFT reference policy, and ORPO combines SFT and odds losses without a reference model.The discussion frames static supervision as unable to distinguish valid from flawed outputs around subtle vulnerability patches.Source: §2.1 · p. 4
  • GRPO addresses static supervision by sampling response groups from current-policy rollouts and using reward-derived relative advantages to guide learning.The prose describes on-the-fly feedback as a way to incorporate the current policy’s behavior.Source: §2.1 · p. 4

2.2. Context-aware Dataset Construction

  • Existing VD datasets may lack context, provide it for only a minority of samples, rely on known vulnerability locations, or share context across vulnerable and patched versions.Some datasets also lack enough context-level data for post-training or benchmarking.Source: §2.2 · p. 5
  • The paper enriches function-level C/C++ vulnerability-patch pairs with dependency context using build information and Code Property Graphs, reporting over 95% context-extraction accuracy in a 400-sample author inspection.Clang-based builds are used for dependency resolution, with Joern as a fallback when compilation fails.Source: §2.2 · p. 5

3. Post-training Pipeline for LLM-based Vulnerability Detection

The pipeline moves from cold-start SFT through off-policy preference optimization and on-policy RL, examining data curation, training, reward design, and evaluation at each stage.

3.1.1. SFT Data Curation

  • SFT data curation contrasts rejection sampling, which filters teacher-generated reasoning but may discard difficult cases, with rationalization, which can produce superficial post-hoc justifications.Rejection sampling retains high-quality reasoning without external guidance; rationalization provides ground-truth metadata to the teacher but risks matching known answers rather than genuine deduction.Source: §3.1.1 · p. 7

3.1.2. LLM-Judge Filtering Protocol

  • The rejection-sampling judge checks whether root-cause reasoning matches ground truth, generates eight responses per query, and retains queries with at least one correct response.Source: §3.1.2 · p. 7

3.2. Off-policy Preference Optimization

  • DPO preferences pair responses sampled from the SFT model, while ORPO preference data comes from a teacher and its objective combines SFT with preference optimization.DPO samples eight candidate responses per query and pairs preferred and dispreferred responses judged for the same query.Source: §3.2.1 · p. 8, §3.2.2 · p. 8

3.3.1. RL Data Curation

  • RL filtering excludes vulnerable–patched pairs with SFT Pairwise Pass@1 of 0 or 1, using eight response pairs to retain cases with mixed outcomes.Pairwise correctness requires both identifying the vulnerability and recognizing its absence in the patched code, reducing class imbalance from prediction bias.Source: §3.3.1 · p. 9

3.3.1. RL Data Curation; 3.3.2. On-policy Optimization

  • Curriculum scheduling orders queries by Pairwise Pass@1, while pairwise scheduling also batches each vulnerable sample with its patched version; GRPO then trains from online rollouts scored by a judge.Random sampling may create batch difficulty variance and vulnerable/patched imbalances; the GRPO pipeline starts from an SFT policy and uses reward feedback for optimization.Source: §3.3.1 · p. 9, §3.3.2 · p. 9

3.3.3. Multi-granularity Reward Systems

  • Reward systems range from binary detection and CWE prediction to root-cause reasoning judgments against CVE descriptions, commit messages, and patch diffs.CWE predictions receive credit when predicted and true IDs have a direct parent-child relationship; root-cause judgments aim to reduce reward hacking from correct guesses supported by flawed reasoning.Source: §3.3.3 · p. 10
  • Specification-based rewards use sample-specific criteria for prediction accuracy, localization precision, and reasoning quality, weighted 0.4, 0.2, and 0.4, respectively.Prediction accuracy is a correctness gate; localization and reasoning provide incremental rewards, allowing the judge to distinguish reasoning quality and reduce stochasticity.Source: §3.3.3 · pp. 10–11

3.4. Evaluation Protocol

  • Evaluation ranges from binary-label and CWE matching to root-cause matching, which requires alignment with ground-truth vulnerability information; treating unknown vulnerabilities in patched code as true negatives reports an upper bound on precision.Because datasets identify only the target vulnerability, the protocol treats other unknown vulnerabilities in patched samples as true negatives; the study also plans to assess bias across judge models.Source: §3.4 · p. 11

4. Experimental Setup

The experiments use a commit-date split to limit leakage, evaluate Qwen3-4B with multiple post-training methods, and measure paired vulnerability-and-patch detection alongside sample-level performance.

4. Experimental Setup

  • The dataset is split into training, validation, and test sets at an 8:1:1 ratio by commit date to avoid leakage.The paper says balancing the pair-based data distribution is important for reducing bias and supporting fair evaluation.Source: §4 · p. 12
  • Qwen3-4B is the backbone; DeepSeek-R1-0528 generates reasoning data, while GPT-OSS-120B serves as the default reward judge and sample specification generator.GPT-OSS-120B is also used for LLM-judge-based filtering and evaluation.Source: §4 · p. 12
  • Pairwise Pass@k measures joint correctness on vulnerable and patched code, while the evaluation also reports P-V, P-B, P-R, and sample-level F1.P-V, P-B, and P-R describe prediction consistency across vulnerability-patch pairs; P-C is omitted as equivalent to P-Pass@1.Source: §4 · p. 12
  • The post-training framework uses Hugging Face TRL with DeepSpeed and vLLM for training and inference efficiency.SFT and preference optimization use four A100 GPUs; GRPO uses a distributed eight-GPU setup, plus two GPUs for vLLM.Source: §4 · p. 12

5. Experimental Evaluation and Results

The experiments compare SFT data curation strategies and test how SFT intensity affects subsequent preference optimization and on-policy RL. Rejection sampling outperforms rationalization, while the effects of more SFT differ between off-policy and on-policy training.

5.1. RQ1: How do different SFT data curation methods impact VD LLM performance?

  • 201.7% relative P-Pass@1 improvement: rejection-sampled SFT outperforms rationalization, which can prompt hallucinated CVEs or vulnerability types.Removing teacher-generated reasoning improves rationalization-based P-Pass@1 by 37.0% relative to reasoning SFT, but it remains substantially below rejection-sampled SFT.Source: Figure 7 · p. 13, §5.1 · p. 13

5.2. RQ2: How does the extent of SFT influence the performance of VD LLMs subsequently trained through preference optimization?

  • As SFT epochs rise, DPO and ORPO improve: DPO P-Pass@1 rises 34.9% and ORPO 14.5% from 1- to 5-epoch initialization, with DPO eventually surpassing ORPO.ORPO is more robust to initialization, whereas DPO scales more effectively with a stronger SFT foundation.Source: Figure 8 · p. 14, §5.2 · pp. 14–15

5.3. RQ3: How does the extent of SFT influence the performance of VD LLMs subsequently trained through on-policy RL?

  • At epoch 5, GRPO P-Pass@1 falls 9.1% versus epoch 3; excessive SFT suppresses exploration and P-Pass@8, while GRPO without SFT also underperforms.In CVE-2024-46769, epoch-3 GRPO considers memory-allocation failure, while epoch-5 GRPO misses the vulnerability.Source: Figure 9 · p. 15, §5.3 · pp. 15–16, Figure 10 · p. 16

5.4. RQ4: How do data filtering and scheduling influence the effectiveness of RL training?

Across RL data curation, reward design, and model comparisons, the study finds that full-data training avoids filtering losses, fine-grained rewards improve training, and GRPO outperforms other post-training baselines. Paired scheduling helps where curriculum learning does not, while current specialized VD models can be undermined by rationalization or coarse rewards.

5.4. RQ4: How do data filtering and scheduling influence the effectiveness of RL training?

  • Filtering extreme-difficulty training data reduces performance versus the full dataset, while the filtered-data GRPO model still improves P-Pass@1 by 15.7% over the initial checkpoint.The authors frame filtering as a trade-off between computational efficiency and peak performance.Source: §5.4 · pp. 16–17, Table 4 · p. 16
  • Curriculum learning does not improve performance, but paired-data scheduling consistently outperforms original-data scheduling under both filtering settings.The authors suggest paired batches help the model learn subtle differences between vulnerabilities and patches.Source: §5.4 · p. 17

5.5. RQ5: How does the granularity of reward systems impact the efficiency of RL?

  • Prediction- and detection-based rewards can induce reward hacking or flawed updates, whereas reasoning- and specification-based rewards improve RL training stability and efficiency.Specification-level assessment adds the overhead of designing and generating sample specifications, so reasoning-based reward is the default.Source: §5.5 · pp. 17–18

5.6. RQ6: How do zero-shot LLMs and post-training methods perform in VD?

  • 183% relative improvement in P-Pass@1 over Qwen3-4B, and a 47.01% P-Pass@8, make Qwen3-4B-OpenVul the top post-training model in the reported comparison.Under specification-based reward guidance, it also surpasses Qwen3-235B-A22B and gemini-2.5-flash on P-Pass@1.Source: Table 5 · p. 18, §5.6 · p. 19
  • R2VUL and ReVD underperform the original Qwen3-4B, while DPO and ORPO rank among the stronger off-policy methods.The authors attribute the former decline to rationalization-induced hallucinations and MARCO's below-SFT performance to its coarse detection reward.Source: §5.6 · p. 19
  • Qwen3-4B-OpenVul remains behind DeepSeek-V3.1 but performs close to its teacher, DeepSeek-R1-0528.The authors suggest higher-quality data distillation could further narrow the gap between small specialized VD models and general-purpose LLMs.Source: §5.6 · p. 19

5.7. RQ7: To what extent do different judge LLMs impact the evaluation accuracy?

Judge-model choice affects evaluation accuracy, but independent judges show strong agreement. Across evaluation granularities, binary and CWE-level scores can misrepresent whether a model has correctly reasoned about a vulnerability.

5.7. RQ7: To what extent do different judge LLMs impact the evaluation accuracy?

  • GPT-OSS-20B and GPT-OSS-120B achieve over 99% judgment accuracy on 400 samples, while GPT-4.1 mini mislabels 59 vulnerable-sample predictions as correct (14.75%).The study selects GPT-OSS-120B as its default judge, citing evaluative precision and output stability.Source: Table 6 · p. 19, §5.7 · pp. 19–20
  • Cohen’s Kappa of 0.8537 between the reward and independent evaluation judges supports the authors’ conclusion that gains are not due to judge overfitting.GPT-OSS-120B supplies RL rewards; Qwen3.5-122B-A10B independently assesses final performance.Source: Table 7 · p. 20, §5.7 · p. 20

5.8. RQ8: How does model assessment shift across multiple evaluation granularities?

  • Reasoning-level evaluation is needed to verify genuine vulnerability understanding because binary and CWE-level assessment can respectively inflate or bias performance estimates.The authors also argue this granularity can prevent RL from rewarding correct labels reached through flawed reasoning.Source: §5.8 · pp. 20–21
  • Of 138 vulnerabilities detected at the binary-label level, 88.4% are mislabeled at the CWE level, showing binary comparison overestimates performance.Among 16 correct CWE predictions, 56.3% rely on root-cause analyses unrelated to ground truth; meanwhile, 6.6% of CWE-mismatched predictions identify the correct root cause.Source: Figure 12 · p. 20, §5.8 · pp. 20–21

6. Related Work

Prior VD work has used prompting approaches, including structural information, explicit reasoning, and multi-agent collaboration. Post-training work includes SFT, off-policy preference optimization, and on-policy RL, with static preference data limiting exploration in cited methods.

6.1. Prompt-based LLMs for VD

  • Prompt-based VD has progressed from structure-aware prompting and explicit reasoning to multi-agent collaboration, but reported challenges include out-of-distribution generalization and noise accumulation.Multi-agent methods use role-specific agents, debates, sequential decisions, or refinement loops for cross-verification.Source: §6.1 · p. 21

6.2. Post-training LLMs for VD

  • Post-training VD work spans instruction-augmented SFT, preference optimization, and RL, while cited off-policy methods generally rely on static preference datasets that restrict exploration of diverse reasoning paths.Examples include VulLLM and MSIVD for SFT, Smart-LLaMA-DPO, ReVD, and R2Vul for preference optimization, and RLFD and GRPO-based work for RL.Source: §6.2 · p. 21

7. Discussion and Future Work

The study’s evaluation covers a representative method set using Qwen3-4B and C/C++ data, and notes potential pretraining–evaluation leakage as a caveat for absolute performance claims.

Evaluation Scope

  • Experiments are limited to representative methods, the Qwen3-4B backbone, and C/C++ data; cross-language datasets with full repository context remain challenging to construct.The authors omit some generic extensions and hyperparameters they consider not directly relevant to VD objectives.Source: §7 · p. 22

Evaluation Validity

  • Potential pretraining–evaluation data leakage may bias absolute performance, so the authors frame their goal as demonstrating relative improvements.They identify more robust evaluation protocols as future work given scarce recent vulnerability data and rapid LLM evolution.Source: §7 · p. 22

8. Conclusion

The study examines post-training across SFT, off-policy preference optimization, and on-policy RL for vulnerability detection, and finds that data curation, training stages, rewards, and evaluation all matter.

8. Conclusion

  • On-policy RL enables models to surpass SFT-based models and state-of-the-art zero-shot LLMs.The paper also finds that fine-grained, root-cause-aligned rewards outperform coarse binary-level and CWE-level signals in training efficiency and accuracy.Source: §8 · p. 22
  • Rejection sampling is more effective than rationalization for SFT data curation because rationalization can introduce reasoning hallucinations.The paper reports that heavy SFT benefits off-policy preference optimization but can inhibit the self-exploration needed for on-policy GRPO.Source: §8 · p. 22
  • Fine-grained, root-cause-aligned rewards outperform coarse binary-level and CWE-level signals in training efficiency and accuracy.The paper also highlights reasoning-level evaluation protocols as necessary to mitigate performance overestimation.Source: §8 · p. 22
  • The study covers cold-start SFT, off-policy preference optimization, and on-policy RL, and provides an open-source framework and datasets.The authors present these resources as a standardized foundation and practical guidelines for developing specialized vulnerability-detection LLMs.Source: §8 · p. 22

A. Details of Dataset

Figure 13 shows the distributions of the dataset’s most common vulnerability types and projects.

A. Details of Dataset

B. Details of CWE Matching

The paper’s CWE matching treats a prediction as correct when its CWE is the ground truth or its direct parent or child in the CWE-1000 taxonomy.

B. Details of CWE Matching

  • A CWE prediction is correct when it matches the ground truth or shares a direct parent-child relationship with it in the CWE-1000 taxonomy.For ground-truth CWE-416, predictions of CWE-416 or its direct parent CWE-825 count as correct; higher-level CWE-119, CWE-118, and CWE-664 do not.Source: B. Details of CWE Matching · p. 30, Figure 14 · p. 30

C. Implemetation Details of LLM Prompts

The appendix documents prompts for vulnerability detection, data curation, judging, and specification generation. Its rubrics assess vulnerable and patched code using reasoning, evidence, and verdict criteria.

C.1. Prompt for LLM-based VD; C.2.1. Rejection Sampling

  • The detection prompt asks the LLM to analyze C/C++ code and context, support identified flaws with evidence, map them to CWE identifiers, and report whether vulnerabilities are present.The rejection-sampling teacher uses the same prompt as the detector.Source: §C.1 · p. 31, §C.2.1 · p. 32

C.2.2. Rationalization

  • The rationalization prompt gives the teacher hidden vulnerability ground truth but instructs it to justify its decision using only the code’s internal logic.The teacher is instructed to simulate a blind audit without explicitly mentioning the external metadata.Source: Figure 16 · p. 32, §C.2.2 · p. 33

C.3.1. Vulnerable Version

  • For vulnerable code, the reasoning rubric marks an analysis correct only when it identifies vulnerability and explains its root cause; recognizing vulnerability but misidentifying the cause is incorrect.The latter distinction is retained in the rubric but treated as incorrect in RL rewards and performance evaluation.Source: §C.3.1 · p. 33

C.3.2. Patched Version

  • For patched code, UNKNOWN is acceptable when the detector reports other potential vulnerabilities, since only removal of the target CVE is confirmed by ground truth.The rubric distinguishes this from correctly identifying the absence of the original CVE.Source: §C.3.2 · p. 33

C.4.1. Vulnerable Version

  • The vulnerable-code specification checklist tests whether the detector identifies the vulnerability, cites code changed by the patch, and explains the exploit mechanism.Its dimensions are Verdict Recall, Evidence Identification, and Reasoning Mechanism.Source: Figure 19 · p. 36, §C.4.1 · pp. 37–38

C.4.2. Patched Version

  • The patched-code specification checklist tests whether the detector avoids reporting the fixed flaw, identifies the safeguard code, and explains how the change prevents exploitation.Its dimensions are Verdict Absence, Evidence of Safeguards, and Reasoning Resolution.Source: Figure 20 · p. 37, §C.4.2 · p. 38

C.5.1. Vulnerable Version

  • For vulnerable code, the judge scores verdict, localization evidence, and reasoning about both the attack mechanism and its technical consequence.The evidence rubric distinguishes exact code matches, partial matches, and misses.Source: §C.5.1 · p. 38, Figure 21 · p. 39

C.5.2. Patched Version

  • For patched code, the judge scores whether the fixed vulnerability is absent, the safeguard code is identified, and the safety logic is understood.The output is a structured, justified assessment of post-patch precision.Source: §C.5.2 · p. 38, Figure 22 · p. 40

Related papers

Loading 2602.14012v2…