PaperlayerView exact revision on arXiv

Source-linked AI summary

Making Small Code LLMs More Efficient for Secure Code Review with Neuro-Symbolic Fine-Tuning

Mostafijur Rahman Akhond, Mahmuda Khatun, Gias Uddin

arXiv:2610.35477v1cs.SE

TL;DR

Generated code raises review demands, while static analyzers provide limited explanation and model-generated explanations can be hard to verify. secSemScan combines analyzer evidence with locally deployable models and OGSM training data; fine-tuning improves prompted verdict accuracy across three models, though independent vulnerability detection is not evaluated.

  • Problem

    Static analyzers offer limited triage explanation and language-model explanations can be difficult to verify, leaving open whether small local models can express checkable security evidence.

    Source: §1 · pp. 1–2

  • Method

    secSemScan pairs analyzer or fix-diff evidence with locally deployable models trained to produce schema-constrained security monologues, using OGSM’s 1,993 C/C++ functions across 22 CWEs.

    Source: §1 · p. 2

  • Results

    Fine-tuning improved all three models on the prompted task; Qwen2.5-Coder-7B verdict accuracy rose from 4.8% to 96.6%.

    Source: §7 · p. 15

  • Takeaways & Limitations

    The results suggest fine-tuning helps small models produce more structured, evidence-linked security reasoning, while independent detection and effectiveness on constrained hardware remain unestablished.

    Source: §9 · p. 18

    Because prompts supply verdicts, CWEs, and oracle evidence, the evaluation measures evidence-grounded narration rather than independent vulnerability detection or developer utility.

    Source: §6 · p. 15

Abstract

from arXiv · show

As AI coding assistants generate more code, the need to review it for security flaws grows. Existing approaches leave a gap: static analyzers identify program sites but offer little explanation, while language-model explanations can be difficult to verify. Small open-weight code models could make security review available without sending proprietary code to a hosted service. We study whether fine-tuning helps these models turn security evidence into explanations that developers and tools can check. We present secSemScan, which pairs a static analyzer with a locally deployable model that writes a schema-constrained security monologue naming program sites, assigning a CWE, and giving a machine-parseable verdict. We also introduce OGSM (Oracle-Grounded Security Monologue), a dataset of 1,993 C/C\texttt{++} functions covering 22 CWEs in the taint and lifetime families. Each record pairs a function with analyzer-derived or fix-diff-derived evidence and a monologue. We fine-tune three open-weight code models in two stages, first on the output format and then on the monologue content. Because evaluation prompts supply the verdict and evidence, our results measure how well models use that information, not whether they detect vulnerabilities independently. On 355 test records, fine-tuning raised Qwen2.5-Coder-7B's verdict accuracy from 4.8\% to 96.6\%; DeepSeek-Coder-6.7B reached 78.9\%, and StarCoder2-7B reached 43.7\%. Qwen2.5-Coder-7B cited 87.5\% of Tier~1 oracle sites, and every line it cited belonged to the oracle evidence. The results suggest that fine-tuning helps small models produce more structured, evidence-linked security reasoning.

1 Introduction

Generated code increases the need for review, while existing analyzers and language-model explanations each leave gaps in checkability or locality. secSemScan pairs analyzer evidence with a locally deployable model, and evaluation shows strong prompted-narration results while leaving independent detection untested.

1 Introduction

  • Static analyzers identify sites but offer limited triage explanation, while hosted LLM explanations may be unfaithful and require sending code externally.The open question is whether small locally deployable models can express security evidence in a form tools can check.Source: §1 · pp. 1–2
  • secSemScan uses Semgrep traces or fix-diff evidence to train a local open-weight model to produce schema-constrained monologues naming sites, assigning a CWE, and giving a parseable verdict.A fresh analyzer run checks named sites at inference.Source: §1 · p. 2
  • OGSM contains 1,993 C/C++ functions across 22 CWEs, pairing verdicts and evidence with monologues for evaluation of creation and usefulness.The evaluation prompts supply each record’s verdict, CWE, and oracle trace.Source: §1 · p. 2
  • Qwen2.5-Coder-7B reached 96.6% verdict accuracy after fine-tuning, versus 4.8% before, on 355 test records.The prompts supplied the verdict, CWE, and oracle trace, so this evaluates narration from evidence rather than independent vulnerability detection.Source: §1 · p. 2
  • Qwen2.5-Coder-7B cited 87.5% of Tier 1 oracle sites, and every line it cited was an oracle site.DeepSeek-Coder-6.7B reached 78.9% verdict accuracy and StarCoder2-7B reached 43.7%.Source: §1 · p. 2
  • The evaluation does not test whether the approach finds vulnerabilities missed by the analyzer’s rules.The paper also contributes a two-stage fine-tuning recipe and rule-based evaluation checks that need no judge model.Source: §1 · pp. 2–3

2 Preliminaries

The preliminaries motivate checking model explanations against program evidence, define the evidence tuples for two weakness families, and specify monologue and verdict semantics.

2.1 Motivating Example

  • Panel (a) gives conflicting unchecked accounts of a use-after-free, whereas panel (b) checks the named free and use sites against a fresh Semgrep trace.The checked trace supports the insecure verdict, illustrating the design’s focus on verifiable sites.Source: §2.1 · p. 3

2.2 Weakness Families and Evidence Tuples

  • Taint weaknesses use a source–sink pair, while lifetime weaknesses use an allocation–free–use tuple to represent the relevant program evidence.The analyzer can verify these site tuples, unlike an unstructured explanation alone.Source: §2.2 · p. 4

2.3 Security Monologue and Verdict States

  • A security monologue names weakness sites by identifier and line, assigns a CWE, and ends with a machine-parseable verdict that the oracle checks.An insecure state requires oracle confirmation of a path and reproduced sites; a secure state requires a SAFE report and no oracle path.Source: §2.3 · p. 4

3 Dataset

OGSM is a C/C++ dataset that pairs functions with security labels, evidence, and site-grounded monologues. Its evidence combines analyzer-confirmed traces with fix-diff-derived records, and its scale and composition shape how results should be interpreted.

3 Dataset

  • Each OGSM record pairs a function with a verdict, CWE, vulnerable-site evidence, and a schema-constrained monologue that reproduces the evidence sites.Source: §3 · p. 4

3.1 Construction

  • OGSM draws on DiverseVul, PairVul, and MVD and retains 1,993 records after candidate selection, merging, and per-CWE capping.Candidate functions shorter than 5 or longer than 200 lines were excluded.Source: §3.1 · p. 4
  • Tier 1 records use Semgrep-confirmed traces, whereas Tier 2 records use fix-diff evidence that is not verified as a source, sink, or allocation site.The dataset has 670 Tier 1 and 1,296 Tier 2 records; 27 merged multi-CWE vulnerable records carry no trace.Source: §3.1 · p. 4

3.2 Statistics and Splits

  • OGSM contains 1,993 records across 22 CWEs and 433 CVEs, with 1,531 taint-track records and 462 lifetime-track records.Its 355-record test split is CVE-disjoint, but PairVul contributes 127 of 178 vulnerable test records and appears in both training and test.Source: §3.2 · p. 6, Table 2 · p. 6

3.3 Comparison with Related Datasets

  • OGSM differs from compared vulnerability datasets by including Semgrep traces for Tier 1 records and unfiltered LLM-generated narrations.The compared datasets contain 5.8K to 349K records, versus OGSM’s 1,993.Source: §3.3 · p. 6, Table 3 · p. 6

4 Methodology

secSemScan pairs static analysis with a locally deployable model that turns oracle evidence into structured security monologues. The methodology defines the oracle traces, monologue schema, two-stage training, and inference checks.

4.1 Oracle Analysis

  • Semgrep runs taint and lifetime rules to produce site-level traces; Tier 1 labels come from rule firings, while Tier 2 evidence comes from fix diffs.The same rules run at inference, so Tier 1 agreement with the oracle is not independent evidence.Source: §4.1 · pp. 6–7

4.2 Monologue and Prompt Design

  • Monologues turn static-oracle traces into readable, checkable narration that identifies required sites by identifier and line, explains the CWE, and ends with a parseable verdict.The fixed prompt supplies the function, CWE, oracle mode, trace, and verdict; the illustrative lifetime example names free and use sites but is not an OGSM record.Source: §4.2 · pp. 7–8

4.3 Model Training

  • All three models use LoRA in two stages: 300 synthetic skeletons teach format until compliance exceeds 90%, then 1,248 OGSM records teach content.The second stage starts with a fresh adapter on the format-aligned weights; the paper does not compare this ordering with single-stage training.Source: §4.3 · p. 8

4.4 Inference

  • Inference checks insecure-verdict sites with fresh Semgrep, but secure claims rely on the model; supplied verdicts, CWEs, and traces make evaluation narration-focused, not vulnerability detection.An output without a verdict line counts as wrong.Source: §4.4 · p. 9

5 Evaluation

The evaluation tests whether models can turn supplied verdicts and evidence into structured, grounded narrations, not detect vulnerabilities independently. Fine-tuning substantially improves Qwen2.5-Coder-7B’s verdict accuracy, while evidence-use checks, decoding tests, and a five-function kernel case study qualify the results.

5.1 Experimental Setup

  • The test set has 355 CVE-disjoint records, but PairVul contributes 127 of 178 vulnerable records and also appears in training; prompts supply verdicts, CWEs, and evidence.The evaluation therefore measures narration from supplied information rather than vulnerability detection alone.Source: §5.1 · p. 9

5.2 Performance of the Finetuned Models

  • On 355 test records, Qwen2.5-Coder-7B fine-tuning raised VA from 4.8% to 96.6% and CWE accuracy from 5.1% to 96.6%; other models reached 78.9% and 43.7% VA.Table 4 reports overall performance; DeepSeek-Coder-6.7B and StarCoder2-7B improved by less than Qwen2.5-Coder-7B.Source: Table 4 · p. 10, §5.2 · p. 10

5.3 Grounding of Monologues in the Oracle Evidence

  • Qwen2.5-Coder-7B cited 70 of 80 Tier 1 sites (87.5%), and all 70 cited lines were oracle sites; Tier 2 evidence quotation and code-identifier rates were 57.4% and 94.6%.Fine-tuning raised Tier 1 site citation from 60.0% but did not change the Tier 2 quotation or identifier rates.Source: §5.3 · pp. 11–12, Table 6 · p. 12
  • Rule-based checks assess narration surface rather than explanation correctness: a sampled Tier 2 narration passed checks despite describing the data flow incorrectly.The authors did not review explanations against a reference.Source: §5.3 · p. 13

5.4 Robustness to Decoding Choices

  • With five samples at temperature 0.8 and top-p 0.95, fine-tuning raised pass@1 from 16.7% to 96.6% on Qwen, 31.6% to 87.2% on DeepSeek, and 24.0% to 37.5% on StarCoder2.Source: Table 7 · p. 13, §5.4 · p. 14

5.5.1 Protocol.

  • The kernel case study uses five pre-fix functions, excludes CVEs overlapping OGSM, and treats the later fix commits as ground truth.Semgrep was run on the same functions, without fix information supplied to secSemScan.Source: §5.5.1 · p. 14, §5.5 · p. 14

5.5.2 Results.

  • secSemScan labeled all five kernel functions VULNERABLE with the correct CWE although Semgrep fired no rule; the 95% interval for the rate spans about 57% to 100%.All five verdicts were in the unverified_trust state, with fix commits confirming claims made only by the model.Source: Table 8 · p. 14, §5.5.2 · pp. 14–15
  • The kernel examples cover taint and lifetime tracks, and each monologue names the identifier changed by the corresponding security fix.The examples describe a missing lock for hdev and an unchecked overflow for octx->next_offset.Source: §5.5.2 · p. 15

6 Threats to validity

The evaluation uses checkable grounding and consistency properties, but does not directly measure explanation quality or developer utility. Its scope and precision are limited by the dataset, evidence tiers, and subset sizes.

Construct validity

  • The rule-based usefulness checks assess oracle-site citation, identifier grounding, and CWE consistency, not explanation quality or developer utility.Because prompts supply the verdict, CWE, and oracle evidence, the evaluation measures evidence-grounded narration rather than vulnerability detection.Source: §6 · p. 15

Internal validity

  • Tier 2 evidence may retain PairVul-specific characteristics despite CVE-disjoint splits, and LLM-generated training narrations may introduce noise.Fine-tuned and untuned models are compared under identical prompts, schemas, and decoding settings.Source: §6 · p. 15

External validity

  • OGSM covers C/C++ and two weakness tracks, limiting generalization to other languages and vulnerability classes.The held-out kernel case study is small, and experiments use one software and hardware environment.Source: §6 · p. 15

Statistical validity

  • Estimates from small subsets, including 55 Tier 1 records, are less precise; the 12 Qwen2.5-Coder-7B records without verdicts come from six functions.The authors report 95% Wilson intervals where possible and note that these errors are not independent.Source: §6 · p. 15

7 Discussion

Fine-tuning improved verdict accuracy and evidence-grounding checks, with gains that varied by model and decoding. The results concern narration from supplied evidence, while a small kernel study offers only preliminary evidence beyond OGSM.

7 Discussion

  • Qwen2.5-Coder-7B rose from 4.8% to 96.6% verdict accuracy; DeepSeek-Coder-6.7B reached 78.9%, and StarCoder2-7B reached 43.7%.The Qwen result was unchanged across five sampled outputs per record.Source: §7 · p. 15
  • Qwen2.5-Coder-7B cited 70 of 80 Tier 1 oracle sites, and all 70 cited lines were oracle sites.Its header matched the record’s track in all 343 outputs with a header, and 165 of 172 eligible VULNERABLE outputs named the same CWE in body and verdict.Source: §7 · p. 16
  • One of 14 reviewed narrations passed the checks despite failing to explain the vulnerability, and Tier 2 evidence-quotation rates were 57.4% for both Qwen variants.These checks test reproduced details, and the unchanged Tier 2 rate suggests gains were not uniform across evidence types.Source: §7 · p. 16
  • 96.6% pass@1, pass@3, and pass@5 for Qwen2.5-Coder-7B; single-output accuracy remained model- and decoding-dependent.DeepSeek-Coder-6.7B reached 87.2% pass@1 and 96.1% pass@5, while StarCoder2-7B reached 37.5% and 89.3%, respectively.Source: §7 · p. 16
  • In a five-function kernel case study, secSemScan assigned the correct verdict and CWE to all five functions, while Semgrep produced no finding.The sample is too small to establish performance on unseen vulnerabilities; experiments on NVIDIA L40S GPUs did not measure memory, latency, or power on constrained hardware.Source: §7 · p. 16

8 Related Work

Related work spans rule-based analysis, vulnerability-prediction models, reasoning-trained open-weight models, and neuro-symbolic systems. secSemScan focuses on checkable narration of supplied static evidence for two C/C++ weakness families.

8.1 Static Analysis for Vulnerability Detection

  • Semgrep can analyze isolated source snippets without a build step, unlike CodeQL’s compilable-codebase requirement, but rule-based analyzers detect only encoded patterns.Six static C analyzers missed 47% to 80% of 192 known vulnerabilities in 27 real-world programs.Source: §8.1 · p. 16

8.2 LLM-Based Vulnerability Detection

  • Vulnerability detection progressed from graph and encoder models to decoder-only code models, but benchmark accuracy may not predict held-out-CVE or cross-domain performance.SecVulEval addresses context-aware C/C++ statement-level detection and localization; secSemScan instead narrates supplied analyzer traces or fix-diff evidence.Source: §8.2 · pp. 16–17

8.3 Scope and Effectiveness of Open-weight LLMs

  • R2Vul [33] uses teacher-written reasoning with reinforcement learning, while VulLLM [7] adds fix-patch localization and GPT-4-generated interpretation to binary detection.R2Vul reports a 1.5B student exceeding its 32B teacher and commercial LLMs in macro-F1 across five programming languages.Source: §8.3 · p. 17
  • SemCoder [6] grounds monologue reasoning in executable Python traces; secSemScan substitutes static-oracle grounding for execution traces.CWE-Trace [38] reports limits to fine-tuning on kernel samples and concludes it shifts output thresholds without producing reliable CWE-classification reasoning.Source: §8.3 · p. 17
  • Unlike R2Vul and VulLLM, secSemScan requires monologues to reproduce named taint or lifetime sites from analyzer traces or fix-diff evidence.This narrows the task to two C/C++ weakness families whose evidence is a small tuple of sites, enabling claim-by-claim checks.Source: §8.3 · p. 17

8.4 Neuro-Symbolic Approaches

  • Other neuro-symbolic systems combine LLMs and program analysis, but none of the four compared systems trains a locally deployable model on statically verified evidence.SecSemScan evaluates unfiltered generated narrations against supplied evidence; its prompts include the oracle trace, so inference-time analyzer comparison does not test independent confirmation.Source: §8.4 · pp. 17–18

9 Conclusion

Fine-tuning improved structured security monologues from supplied evidence, with Qwen2.5-Coder-7B showing the largest reported gains. The results do not establish independent vulnerability detection or practical performance on constrained hardware.

9 Conclusion

  • Qwen2.5-Coder-7B reached 96.6% verdict accuracy and cited 87.5% of Tier 1 oracle sites; DeepSeek-Coder-6.7B reached 78.9%, and StarCoder2-7B reached 43.7%.The prompts included verdict and evidence, so these results do not establish independent vulnerability detection.Source: §9 · p. 18

Related papers

Loading 2610.35477v1…