PaperlayerView exact revision on arXiv

Source-linked AI summary

JevVibe: Efficient Classification-Guided Secure Code Generation

Arshak Rezvani, Sasha Behrouzi, Ahmad-Reza Sadeghi

arXiv:2609.34963v1cs.CRcs.AI

TL;DR

Functionally correct generated code can still contain security weaknesses, so repair systems may first diagnose the weakness type; generating and parsing CWE labels also raises efficiency and validity questions. The paper evaluates Jev’s direct-choice CWE classification and uses it to guide repair, finding advantages over open-weight baselines, metric-dependent results against GPT-5.6-Sol, and a higher detector-measured pass rate with Jev-guided repair.

  • Problem

    Generated code can meet functional requirements yet contain security weaknesses, and using generated text that must be parsed for a CWE diagnosis raises questions about validity, speed, and cost.

    Source: §1 · p. 1

  • Method

    The authors evaluate Jev’s direct fixed-choice CWE classification against autoregressive models and use predicted diagnoses to guide code repair in JevVibe.

    Source: §3.1.1 · p. 3, §3.2 · p. 4

  • Results

    Jev outperforms six open-weight baselines across reported metrics; against GPT-5.6-Sol, it leads on Top-3 and Top-5 while GPT-5.6-Sol leads on Top-1 and Macro-F1, and Jev-guided repair raises pass rate from 0.661 to 0.707.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Jev-guided repair improves aggregate detector-measured security, but both repair approaches can introduce vulnerabilities and neither improves security consistently across languages.

    Source: §7 · p. 9

    The downstream evaluation uses one repair model and only each classifier’s Top-1 diagnosis, and the classification and repair results rely on one benchmark and one external evaluator.

    Source: §7 · p. 9

Abstract

from arXiv · show

Large language models can generate functionally correct code that still contains security weaknesses, motivating repair pipelines that first diagnose a weakness type before deciding how to fix it. The Common Weakness Enumeration (CWE) provides a standardized vocabulary for such diagnoses, but asking an autoregressive language model to generate a CWE label and extracting it from the response raises questions about output validity, speed, and cost, as well as accuracy. We evaluate Jev, a decision model that instead selects directly from a declared set of candidates and returns a probability for each, against six open-weight autoregressive models and a frontier proprietary model, GPT-5.6-Sol, on a controlled 50-way CWE classification task over 1,916 CyberSecEval benchmark examples. Jev outperforms all six open-weight baselines on every classification and ranking metric, while its comparison with GPT-5.6-Sol depends on the metric: GPT-5.6-Sol achieves higher Top-1 accuracy and Macro-F1, whereas Jev achieves higher Top-3 and Top-5 accuracy and a nearly identical MRR, at $6.27\times$ lower median API latency and $55.9\times$ lower estimated API cost. We further build JevVibe, a diagnosis-guided repair agent that uses predicted CWE labels to repair code generated by Qwen2.5-Coder-32B-Instruct. With Jev providing the diagnosis, the agent increases the detector-measured security pass rate from 63.5% before repair to 70.7%, compared with 66.1% for LLM-guided repair. These results show that JevVibe is effective at improving the security of generated code, with Jev providing reliable and efficient CWE classification.

1. Introduction

The paper studies whether direct fixed-choice CWE classification can provide a more efficient diagnosis step for secure code repair than generating and parsing CWE labels. It evaluates Jev against autoregressive models and uses Jev’s diagnoses in a repair agent.

Motivation

  • The study asks whether generating and parsing a CWE label is an efficient way to make a fixed-choice diagnosis, beyond prediction accuracy.The question concerns the evaluated prompting and parsing approach, not autoregressive models’ general ability to predict CWE labels.Source: §1 · p. 1

Approach

  • The authors build JevVibe, a diagnosis-guided repair agent, and compare Jev with six open-weight models and GPT-5.6-Sol on 1,916 CyberSecEval examples.The classification task uses a fixed catalog of 50 CWE classes; the repair stage uses Qwen2.5-Coder-32B-Instruct-generated code.Source: §1 · pp. 1–2

Main results

  • Jev beats all six open-weight baselines on reported classification and ranking metrics; versus GPT-5.6-Sol, Jev wins Top-3 and Top-5 while GPT-5.6-Sol wins Top-1 and Macro-F1.Jev’s MRR is nearly identical to GPT-5.6-Sol’s; its median API latency is 6.27× lower and estimated API cost is 55.9× lower.Source: §1 · pp. 1–2
  • Jev-guided repair raises detector-measured security pass rate from 63.5% before repair to 70.7%, compared with 66.1% for LLM-guided repair.Both repair approaches also cause some previously passing examples to fail.Source: §1 · p. 2

Contributions

  • The paper examines classification quality, output coverage, API latency and cost, and how classifier choice affects repairs and newly introduced vulnerabilities.The authors present the work as three contributions, including an analysis of these classification and repair dimensions.Source: §1 · p. 2

2. Preliminaries

The preliminaries introduce CWE as a vocabulary for weakness types and situate the work between CWE classification and secure code generation. They describe Jev’s typed-choice approach and note limits in the available independent evidence about the model.

2.1. Common Weakness Enumerations, Classification, and Secure Code Generation

  • CWE identifiers name weakness types rather than individual vulnerability instances, providing a standardized vocabulary for describing security issues in code.The paper connects prior CWE classification work with secure code generation by using predicted weaknesses to guide repair and an external CWE-aligned detector to evaluate it.Source: §2.1 · p. 2
  • Prior CWE classification studies include prompted LLMs, often using a restricted or curated set of weakness types rather than the full taxonomy.Secure code generation research instead steers models during generation or adapts them through training; generated code is typically evaluated for known CWE-associated patterns.Source: §2.1 · p. 2

2.2. Jev and Typed Choice Prediction

  • Jev evaluates a declared set of candidates in parallel, and its typed Choice interface returns schema-conforming answers with a probability for each candidate.The paper uses the interface for a single decision over 50 CWE classes.Source: §2.2 · p. 2
  • Jev’s developers describe RLCD training as targeting calibrated decision probabilities, but the paper notes no independent peer-reviewed evaluation of Jev at the time of writing.The model description relies on developer documentation, whereas the reported performance results come from the authors’ experiments.Source: §2.2 · p. 2

3. JevVibe

JevVibe evaluates typed CWE classification and uses predicted weakness labels to guide code repair. The study repurposes CyberSecEval for 50-class classification and compares autoregressive and typed-choice approaches alongside a controlled repair evaluation.

3. JevVibe

  • The study compares a typed decision model with open-weight and proprietary autoregressive models for 50-class CWE classification, then evaluates classifier-guided repair of generated code.The repair agent uses each classifier’s predicted CWE to guide automatic repair.Source: §3 · p. 2

3.1.1. Benchmark and CWE Taxonomy.

  • CyberSecEval’s 1,916 Instruct tasks provide origin_code snippets and associated CWE identifiers, which the study uses as classification inputs and benchmark labels.The benchmark’s 50 CWE identifiers form a fixed catalog with names and descriptions shared across classifiers.Source: §3.1.1 · p. 3

3.1.2. Autoregressive CWE Classification.

  • Autoregressive models receive the code and full 50-class catalog and return up to five ranked CWE identifiers in a fixed JSON schema, with a regex fallback for parsing.Invalid or unrecoverable predictions count as classification failures; the resulting lists do not provide normalized probability distributions.Source: §3.1.2 · p. 3

3.1.3. Jev Choice Classification.

  • Jev’s typed Choice interface requires predefined CWE candidates and returns a selected class plus a probability for each candidate, which is sorted to rank the classes.Responses missing the expected choice or probability map are treated as inference failures.Source: §3.1.3 · p. 3

3.1.4. Classification Metrics.

  • Classification metrics include Top-1, Top-3, Top-5, MRR, Macro-F1, and coverage; prediction metrics use valid classifications, while coverage uses all 1,916 examples.Additional diagnostic analyses report Top-1 accuracy by frequent CWE class and programming-language subset, with language results treated descriptively.Source: §3.1.4 · p. 3
  • API efficiency is assessed using end-to-end median and 95th-percentile latency and estimated experiment cost, with provider pricing and token usage informing cost calculations.Provider token counts are not compared directly because billing and tokenization conventions differ.Source: §3.1.4 · p. 3

3.2. Building a CWE-Guided Repair Agent

  • The repair comparison uses the same baseline programs and Qwen2.5-Coder-32B-Instruct generator and repair model across unmodified, LLM-guided, and Jev-guided conditions.The repair prompt and decoding parameters are held constant; outputs are assessed with CyberSecEval’s official detector, using security pass rate and paired fix/regression measures.Source: §3.2 · p. 4
  • The repair workflow diagnoses generated code with an LLM-based detector or Jev and supplies the Top-1 CWE and its definition to the repair model.Target-CWE agreement is reported diagnostically because generated code need not retain the original benchmark weakness.Source: §3.2 · p. 4, Figure 2 · p. 5

4. Results

Classification results split by metric: GPT-5.6-Sol leads on Top-1 and Macro-F1, while Jev leads on Top-3 and Top-5 and is far more efficient. In controlled repair, Jev-guided repair raises security pass rate more than LLM-guided repair, though repairs can also regress secure examples.

4.1. CWE Classification Performance

  • GPT-5.6-Sol scores 0.495 Top-1 accuracy and 0.360 Macro-F1, while Jev scores 0.458 and 0.269; Jev leads Top-3 and Top-5 at 0.766 and 0.836.Their MRR values are nearly identical: 0.628 for GPT-5.6-Sol and 0.624 for Jev. Jev also outperforms all six open-weight models, including Qwen2.5-Coder-32B-Instruct, on the reported classification and ranking metrics.Source: §4.1 · p. 5, Figure 3 · p. 6
  • Jev reduces estimated API cost from $15.83 to $0.283 and median end-to-end request latency from 1.531 s to 0.244 s versus GPT-5.6-Sol.The median latency difference is 6.27×; 95th-percentile latency is 3.737 s for GPT-5.6-Sol and 0.292 s for Jev.Source: §4.1 · p. 5, Table 2 · p. 6
  • Class-level results vary across the 50 CWE classes, and no classifier dominates every programming-language subset.Language-wise comparisons are descriptive, not evidence of intrinsic language-specific capability differences, because subsets contain different CWE mixtures.Source: §4.1 · pp. 6–7, Figure 4 · p. 7

4.2. Repair Agent Performance

  • Jev-guided repair raises security pass rate from 0.635 unrepaired to 0.707, versus 0.661 with LLM-guided repair.The repair model, prompt, decoding configuration, and evaluation procedure were held fixed while the diagnostic source changed.Source: §4.2 · p. 7, Figure 5 · p. 8
  • On six of eight language subsets Jev-guided repair has the highest pass rate, while LLM-guided repair leads on C++ and the unrepaired baseline on C#.The diagnostic rankings on generated baseline code are higher for Jev: strict Top-1, Top-3, and Top-5 agreement are 0.336, 0.568, and 0.645, versus 0.230, 0.433, and 0.484 for LLM-guided diagnosis; these are target-CWE agreement scores, not conventional detection accuracy.Source: §4.2 · pp. 7–8
  • Jev-guided repair fixes 228 of 700 baseline-vulnerable generations (32.6%), versus 129 (18.4%) for LLM-guided repair, but regresses 89 previously secure examples versus 79.Net improved examples are +139 for Jev-guided repair and +50 for LLM-guided repair; the higher Jev-guided pass rate reflects more fixes, not fewer regressions.Source: §4.2 · p. 7

5. Related Work

Prior CWE work classifies descriptions or source code, predicts through the CWE hierarchy, or prompts instruction-tuned LLMs. JevVibe instead compares typed fixed-choice decisions with autoregressive models for flat CWE classification and diagnosis-guided repair.

5. Related Work

  • Prior CWE classification ranges from text classifiers for vulnerability descriptions to pretrained code models, hierarchical CWE-path prediction, and instruction-tuned LLM prompting [11],.Prior work reports that CWE classification lags binary security-patch detection.Source: §5 · p. 8
  • This paper evaluates a typed decision model against autoregressive models on unpaired source code classified into a flat 50-class CWE catalog, including downstream diagnosis-guided repair.The authors report no prior evaluation of this typed-decision comparison on CWE classification or diagnosis-guided repair.Source: §5 · p. 8
  • Secure-generation work steers models with learned prompts or activation-level interventions [3], or fine-tunes on vulnerability fixes [4], synthetic CWE-paired data [14], or selected security-relevant neurons [5].These methods evaluate known weakness patterns rather than outputting a reusable CWE label; HexaCoder embeds CWE-targeted changes in training weights [4].Source: §5 · p. 8

6. Discussion

The strongest classifier depends on whether exact Top-1 prediction or higher-ranked alternatives matter, and Top-1 accuracy alone does not determine repair outcomes. Jev's efficiency and probability outputs have important scope limits: they do not establish architecture-independent speed advantages or calibrated confidence.

6. Discussion

  • GPT-5.6-Sol leads Top-1 accuracy (0.495) and Macro-F1 (0.360), while Jev leads Top-3 (0.766) and Top-5 accuracy (0.836); their MRR scores are nearly identical (0.628 and 0.624).Despite lower Top-1 accuracy, Jev-guided repair has a higher security pass rate than LLM-guided repair (0.707 vs. 0.661).Source: §6 · pp. 8–9
  • The measured 6.27× lower median latency and 55.9× lower estimated cost apply to the evaluated systems and API configurations, not architecture-independent efficiency.Jev requires a predefined candidate set and output type and does not support open-ended text generation.Source: §6 · p. 9
  • Jev's probability outputs do not establish calibration; testing that claim requires a dedicated calibration study across multiple decision domains.The experiments do not establish whether reported confidence consistently corresponds to empirical correctness.Source: §6 · p. 9

7. Conclusion

Jev outperforms all evaluated open-weight models on classification and ranking metrics, while its results against GPT-5.6-Sol vary by metric. In JevVibe, Jev-guided repair raises the detector-measured security pass rate, though the evaluation has important benchmark and evaluator limitations.

  • Jev outperforms every evaluated open-weight model across classification and ranking metrics; GPT-5.6-Sol leads Top-1 accuracy and Macro-F1, while Jev leads Top-3 and Top-5 with nearly identical MRR.Source: §7 · p. 9
  • 55.9× lower estimated API cost and 6.27× lower median latency distinguish Jev from GPT-5.6-Sol.Source: §7 · p. 9
  • When guiding a diagnosis-and-repair agent, replacing an autoregressive CWE classifier with Jev raises overall security pass rate from 0.661 to 0.707 and fixes 228 initially vulnerable programs, versus 129 with LLM-guided repair.Both approaches occasionally introduce new vulnerabilities, and neither improves security consistently across all languages.Source: §7 · p. 9
  • Jev is presented as useful for quick, structured choices from a fixed set, not as a general replacement for autoregressive models.Source: §7 · p. 9
  • The downstream evaluation uses one repair model and Top-1 diagnoses, and both result sets rely on one benchmark and one external evaluator with imperfect judgments.Jev is commercial and vendor-documented, and cost and latency reflect one provider’s pricing at the time of experimentation.Source: §7 · p. 9

Related papers

Loading 2609.34963v1…