PaperlayerView exact revision on arXiv

Source-linked AI summary

Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferences

Mohammad Saqib Hasan, Saikat Chakraborty, Santu Karmaker, Niranjan Balasubramanian

arXiv:2506.00419v2cs.CRcs.SE

TL;DR

LLM-generated code can contain security issues, while secure-code training faces challenges in obtaining broad, high-quality data and aligning models to localized security differences. The paper distills security-reasoned code preferences from frontier LLMs and introduces LPO; evaluations report reduced insecurity and improved code quality. The authors also identify noise and representativeness limits in the synthetic data.

  • Problem

    Secure-code training faces challenges in acquiring broad, high-quality data and aligning models to security differences localized within otherwise similar code.

    Source: §3.2 · p. 3

  • Method

    The paper distills security-reasoned secure/insecure code preferences using security knowledge and frontier LLMs, and introduces LPO for localized preference optimization.

    Source: §4 · p. 4, §4.1 · p. 4, §9 · p. 9

  • Results

    LPO reduces security issues by 19−40% on four secure coding benchmarks and improves code quality by 3−10% on two coding benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that DiSCo and LPO contribute substantially to improving code security and quality on multiple benchmarks.

    Source: §9 · p. 9

    DiSCo is derived from closed-source frontier LLMs, and its data may be noisy and unrepresentative of source code in the wild.

    Source: Limitations · p. 10

Abstract

from arXiv · show

LLM generated code often contains security issues. We address two key challenges in improving secure code generation. First, obtaining high quality training data covering a broad set of security issues is critical. To address this, we introduce a method for distilling a preference dataset of insecure and secure code pairs from frontier LLMs, along with a security reasoning that explains the issues and the fix. The key idea here is to make use of security knowledge sources to devise a systematic prompting strategy that ensures broad coverage. Second, aligning models to secure code requires focusing on localized regions of code. Direct preference optimization methods, like SimPO, are not designed to handle these localized differences and turn out to be ineffective. We address this with a new localized preference optimization algorithm that masks the security related tokens in both the winning (secure) and losing (insecure) responses. To prevent loss in code quality, we also add a regularizer. Evaluations show that both training on our dataset, DiSCo, and the new preference optimization algorithm, LPO, yield substantial reductions in code insecurity while also improving overall code quality. Code and dataset are available at https://github.com/StonyBrookNLP/disco-lpo.

1 Introduction

The introduction presents DiSCo and LPO as responses to challenges in obtaining broad, high-quality security training data and aligning models to localized security differences. It reports improvements in security and code quality and situates the work among code-generation alignment and security methods.

1 Introduction

  • Prior secure-code tuning faces difficulty collecting scalable data with broad CWE coverage and aligning models that treat security-relevant and other tokens equally.The authors also note that models are not trained to reason about security during code generation.Source: §1 · p. 1
  • DiSCo uses security knowledge sources and analyzer feedback to create 10k preference pairs with reasoning about security issues and fixes.The dataset is described as Python data covering a wider range of known security issues than previous datasets.Source: §1 · p. 2
  • LPO focuses preference optimization on localized security differences and adds an SFT loss over other tokens to help preserve code quality.The paper identifies SimPO as an existing method not well-suited to these localized preferences.Source: §1 · p. 2
  • LPO reduces security issues by 19−40% on four secure coding benchmarks and improves code quality by 3−10% on two coding benchmarks.The introduction also reports significant improvements from DiSCo training with standard SFT.Source: §1 · p. 2

2 Related Works

  • Related work spans code-generation alignment and generated-code security, while the authors position their approach as preference tuning over security-relevant tokens.The paper also describes a data distillation pipeline incorporating domain knowledge and reasoning over security.Source: §2 · pp. 2–3

3 Secure Code Generation

This section frames secure code generation as balancing code utility with security, and identifies data acquisition and model alignment as challenges. It presents a knowledge-guided distillation pipeline and refinement process alongside localized preference optimization.

3 Secure Code Generation

  • LLM-generated code security is a concern, and the task is framed as maximizing utility while minimizing security issues.The passage defines utility through compilation and unit tests and security through tools such as CodeQL and Bandit.Source: §3 · p. 3, §3.1 · p. 3

3.2 Challenges in secure code generation

  • Training secure code generation requires prompt, secure-code, and insecure-code tuples, but manual curation is costly and automated sources can be noisy, small, narrow in vulnerability coverage, and codebase-specific.The cited prior work reports poor data quality from documentation and tool inaccuracies.Source: §3.2 · p. 3
  • Secure and insecure responses often differ only in small code regions, making standard preference optimization ill-suited and creating a risk of reduced code utility.The example given is avoiding a potentially necessary insecurity-prone function such as os.popen.Source: §3.2 · p. 3

3.3 Our solution

  • DiSCo distills secure/insecure code pairs with security reasoning, using controlled synthesis for broad CWE coverage and analyzers to reduce noise.LPO focuses on log-probability differences for security-related tokens and regularizes other tokens with supervised fine-tuning.Source: §3.3 · pp. 3–4

4 DiSCo: Distilling Secure Code Alignment Dataset

  • Simple frontier-LLM prompting can yield easy, commonly covered CWE instances and secure code that still contains vulnerabilities, motivating knowledge-base prompts and analyzer-guided refinement.DiSCo instances pair a task prompt with insecure code and reasoning, plus secure code and reasoning about its security.Source: §4 · p. 4

4.1 Generating Prompts for Distillation

  • Prompt generation combines curated CWE-ID, issue, and description tuples with package instances to sample prompts for broad CWE and application coverage.The security information is curated from information hubs and documents, including the CWE website and analyzer documentation.Source: §4.1 · p. 4

4.2 Distillation with Refinement:

  • Analyzer feedback and one refinement reduce generated secure examples with security issues from 37.4% to 12.7%, while three iterations reduce them to 9.4% but worsen synthetic-data quality.The reported remaining issues include an unfixed target issue or other issues outside the prompt's scope.Source: §4.2 · pp. 4–5

5 Preference Optimization for Secure Code Generation

The training pipeline first uses supervised fine-tuning to generate secure code with security reasoning, then preference-tunes models to favor secure over insecure code. LPO localizes preference loss to security-relevant tokens and adds an SFT regularizer over other secure-code tokens.

5 Preference Optimization for Secure Code Generation

  • The pipeline applies supervised fine-tuning before preference tuning, which favors secure responses over insecure responses for the same prompt.The preference-tuning stage uses LPO to exploit the localized nature of security-related differences.Source: §5 · p. 5

5.1 Supervised Fine-tuning with Security Reasoning

  • The SFT stage trains models to generate secure code and reasoning from the prompt, using reasoning assembled from the CWE ID, issue, description, and r+ and r−.The reasoning is intended to help models consider possible security issues and insecure code when generating the secure version.Source: §5.1 · p. 5

5.2 Localized Preference Optimization

  • LPO masks security-relevant tokens in secure and insecure responses, concentrating preference loss where the security issue differs.The masks mark differing tokens; the shared reasoning trace is masked out.Source: §5.2 · p. 5
  • Because localization alone can lead to non-parseable or incoherent code, LPO adds an SFT regularizer over secure-response tokens outside the security mask.The paper attributes these outputs to reward hacking and defines the regularizer using the complement of the secure-response mask.Source: §5.2 · p. 5

6 Experimental Setup

The experiments evaluate DiSCo and LPO for security and coding utility across several models and testbeds. The reported results indicate that DiSCo-trained models perform strongly on security benchmarks, with LPO outperforming other fine-tuning strategies in nearly all tested setups.

6 Experimental Setup

  • The evaluation is designed to test whether DiSCo and LPO improve both security and code quality in billion-scale models, including comparison with frontier models.The paper also assesses the effect of individual LPO components and analyzes security issues in generated code.Source: §6 · p. 5, §7 · p. 6

6.1 Datasets

  • The final DiSCo dataset contains 9,987 instances covering 431 insecurity categories, built from 10,000 sampled prompts drawn from combinations of security issues and Python libraries.The pipeline uses 534 issues from CWE and analyzer documentation and manually identifies 75 security-prone Python libraries; one analyzer-feedback refinement leaves 12.7% of the refined dataset with security issues.Source: §6.1 · pp. 5–6

6.2 Models

  • The study evaluates Phi-2, CodeLlama, Mistral, and Starcoder-2 alongside GPT-4o, GPT-4o-Mini, and Claude-3.5-Sonnet.For smaller models, settings include DiSCo SFT and LPO, with off-the-shelf, SafeCoder, DPO, and SimPO baselines; larger models are compared zero-shot with security-aware prompting.Source: §6.2 · p. 6

6.3 Evaluation

  • Security is evaluated on four testbeds using insecurity and issues per 100 generations, while coding utility is measured with pass@1 and pass@5.Security analyzers CodeQL and Bandit identify issues in parsable generations; lower security metrics and higher pass@k indicate better performance.Source: §6.3 · p. 6, §6.1 · p. 6

6.4 Training Setup

  • Training uses LoRA, with LPO evaluated using β = 10.0, γ = 5.4, and α = 0.05 across models.Training evaluation took around 48 hours on four A6000 GPUs.Source: §6.4 · p. 6

1) DiSCo improves secure code generation

  • DiSCo-trained models give the best security performance across model and dataset combinations, while LPO yields substantial gains over baselines and SafeCoder.The results caption also reports consistent coding-utility gains over base models; LPO has the best security performance in almost all dataset/model setups.Source: 1) DiSCo improves secure code generation · pp. 6–7, Table 1 · p. 7

2) LPO is effective for secure code generation

LPO improves secure code generation across nearly all tested model–benchmark combinations and retains or improves coding utility. The analyses show a security–utility trade-off, broad CWE coverage, and gains on unseen CWEs, while examining LPO components and DiSCo refinement.

2) LPO is effective for secure code generation

  • LPO outperforms other fine-tuning strategies in 15 of 16 model–security benchmark combinations and reduces insecurity by 2–30% versus SimPO.The reported reductions are across cases; the source does not assign individual values to particular combinations.Source: 2) LPO is effective for secure code… · p. 7
  • DiSCo training improves pass@k by about 3–10 points, while LPO on DiSCo improves it by roughly 3–6 points over off-the-shelf models.The authors report these gains on standard code-generation benchmarks.Source: 2) LPO is effective for secure code… · p. 7
  • Frontier models outperform StarCoder2 LPO in coding performance, but StarCoder2 LPO performs better on secure coding benchmarks.The comparison covers Security Eval and HumanEvalX.Source: Table 2 · p. 8, 2) LPO is effective for secure code… · p. 8

8 Analyses

  • Security-focused alignment can trade code utility for security, although LPO with DiSCo provides the best security–utility trade-off against baseline and SafeCoder models.Manual error inspection found some alignment models added more complex code, including extra try-except blocks or wrapper functions.Source: §8 · p. 8
  • LPO eliminates issues in 8 of 32 CWE categories, versus SafeCoder’s 1, and reduces most frequent CWEs; SafeCoder eliminates CWE-89.CWE-78 and CWE-259 are among the most frequent categories shown.Source: §8 · p. 8, Figure 2 · p. 8
  • Removing localization worsens security and code generation; removing regularization favors security at a utility cost, while removing reasoning sharply lowers coding ability.The ablations also find that reasoning removal slightly worsens security, and SFT before LPO slightly helps security while improving code utility.Source: §8 · p. 8, Table 3 · p. 9
  • On SecurityEval and LLMSecEval, LPO trained on DiSCo without the evaluation CWEs performs worse than full-DiSCo training but still gains over the base model.The authors attribute the remaining generalization to DiSCo’s broad coverage and the interconnection of CWEs and security issues.Source: Table 4 · p. 9, §8 · p. 9
  • Single-round refinement reduces erroneous data from about 37% to 12.7%; further refinement reduces errors by 3% but deteriorates security and coding utility.Keeping erroneous instances after one refinement has little effect on security and slightly reduces code quality.Source: Table 5 · p. 9, §8 · p. 9

9 Conclusions

The paper presents DiSCo, a 10k-instance preference dataset, and LPO, a preference loss for localized security issues in code; evaluations report improved security and quality. The authors identify limitations in the synthetic data and Python-only evaluation, and flag risks from vulnerable examples.

9 Conclusions

  • The work distills 10k secure–insecure code preference instances with security reasoning and introduces LPO to account for localized security issues.The conclusion reports that DiSCo and LPO both contribute to improved security and quality across multiple benchmarks.Source: §9 · p. 9
  • Future work could improve DiSCo through retrieval of security documentation and improve LPO through better reasoning.Source: §9 · p. 9

Limitations

  • The evaluation is limited to Python; synthetic data from closed-source frontier models may be noisy and unrepresentative of source code in the wild.The authors also describe the evaluated secure-code benchmarks as relatively small and easy.Source: Limitations · p. 10

Ethical Consideration

  • The insecure code in DiSCo could be used to train and deploy models that generate vulnerable code, creating risks of misuse or unsafe coding agents.The authors advise appropriate care when using the dataset.Source: Ethical Consideration · p. 10

A Prompts

The prompts generate insecure and secure code with security reasoning, refine secure outputs using analyzer feedback, and specify formats for benchmark code-generation tasks.

Data Generation Prompt

  • Data-generation prompts draw on security-issue information and request vulnerable code, vulnerability reasoning, a minimally changed secure version, fix reasoning, and an instruction summary.The issue is intended to remain implicit in the instruction summary.Source: A Prompts · p. 13

Data Refinement Prompt

  • The refinement prompt supplies secure code and Bandit and CodeQL findings, then requests fixed code and reasoning.The prompt is for refining secure DiSCo examples with analyzer-discovered issues.Source: A Prompts · p. 13

Reasoning R Template

  • The reasoning template combines a CWE identifier, issue, description, and reasoning about the secure and insecure versions..Source: A Prompts · pp. 13–14

Prompts for Secure Code Generation

  • Secure-code benchmark prompts vary by dataset format, requesting completion from a docstring, partial code snippet, or instruction.The docstring and instruction variants are also used for coding benchmarks.Source: A Prompts · p. 14

Prompts for Frontier LLMs

  • Frontier-model prompts include a secure-code reminder for incomplete code and a separate autocomplete format that asks only to complete code.The incomplete-code prompt explicitly says to keep code security in mind.Source: A Prompts · p. 14

B Dataset Details

The section points to a table of DiSCo dataset statistics and a figure showing a detailed training-set example.

Dataset example

Dataset statistics

C Mask Computation

The section computes security-token masks by comparing secure and insecure code, and describes six evaluation benchmarks spanning secure and general code generation.

C Mask Computation

  • Token-level differences between functionally equivalent secure and insecure code identify tokens for the security masks used in the loss.The method assumes tokens unique to the insecure version relate to insecurity and unique tokens in the secure version improve security; it computes the difference after target-model tokenization using difflib.Source: C Mask Computation · pp. 14–15

D Evaluation Benchmark Details

  • Six benchmarks assess performance: four target code security and two assess regular code generation.The passage introduces the benchmark set but does not name all six in the supplied text.Source: D Evaluation Benchmark Details · p. 15
  • Security Eval contains 121 partially completed Python prompts, each exposing a CWE during code generation.The prompts derive from analyzer documentation, security-issue documentation, or expert authoring.Source: D Evaluation Benchmark Details · p. 15
  • The selected Asleep at the Keyboard subset contains 29 Python prompts from its diversity-of-weakness evaluation set.The broader benchmark varies weakness, prompt, and domain; the selected examples are Python-only.Source: D Evaluation Benchmark Details · pp. 15–16
  • The evaluations also include 81 Python LLMSecEval instructions and 200 held-out DiSCo vulnerability prompts.LLMSecEval prompts request vulnerable code; the synthetic set consists of natural-language instructions for code susceptible to vulnerabilities.Source: D Evaluation Benchmark Details · p. 16
  • MBXP tests simple multilingual instruction-to-code problems with test cases, while HumanEvalX uses code-completion prompts and public tests to assess functional correctness..Source: D Evaluation Benchmark Details · p. 16

E Baselines

The section reviews supervised security tuning and preference-optimization baselines. These methods differ in their training objectives and use of reference models.

SafeCoder

  • SafeCoder combines instruction tuning with unlikelihood training on intent, insecure code, and corresponding secure code, applying security masks during finetuning..Source: E Baselines · p. 16

DPO

  • DPO reparameterizes the RLHF loss to optimize a policy without hosting a separate reward model, using preferred and rejected responses relative to a reference policy.The reference model is the unaligned base model, and β controls deviation from it.Source: E Baselines · p. 16

SimPO

  • SimPO eliminates reference log-probabilities, making reference modeling unnecessary, and uses β and γ as reward scale and target reward margin..Source: E Baselines · pp. 16–17

F Setup Details

The paper reports LoRA-based training and sampling settings, then evaluates security with two analyzers and code-generation utility with pass@k. Its security metrics capture both the share of insecure generations and the number of issues per 100 generations.

F Setup Details

  • Models used LoRA with r = 16 and α = 32; evaluation ran on 4 A6000 GPUs, with about 48 GPU-hours for training and evaluation.DiSCo generation and refinement cost approximately 100 USD in GPT-4o API use; evaluation cost around 20 USD across OpenAI and Anthropic APIs.Source: F Setup Details · p. 17

G Evaluation Details

  • Security evaluation combines CodeQL and Bandit, checks for any detected issue rather than only the test-set CWE, and excludes unparsable generations.The two analyzers have only a subset of intersecting rules, so their combination identifies more possible security patterns.Source: G Evaluation Details · p. 17
  • Insecurity measures the percentage of evaluation code with any detected issue, while Issues per 100 scales the mean issue count across generations by 100.The former captures absolute reduction in insecure coding; the latter provides a bug-level view.Source: G Evaluation Details · pp. 17–18
  • Code-generation utility is measured with an unbiased pass@k estimator based on n generated samples and c correct samples.The estimator accounts for the probability that a selected subset of k samples contains only incorrect samples; security evaluations use five generations at T = 0.4, while code-generation evaluations use five at T = 0.2 and T = 0.6 for pass@1 and pass@5, respectively.Source: F Setup Details · p. 17, G Evaluation Details · p. 18

H CWE Analysis

The CWE analysis spans 32 categories: LPO reduces most top categories but leaves common issues, while SafeCoder and LPO differ on SQL injection. The refinement example also shows how added security layers can make code unnatural and potentially reduce utility.

H CWE Analysis

  • CWE-78 (OS command injection) and CWE-259 (hard-coded credentials) remain common and are not fully eliminated by LPO.The authors suggest hard-coded credentials persist because DiSCo examples involving authentication contain them even after refinement.Source: H CWE Analysis · p. 18
  • Across the 32-CWE analysis, LPO reduces most top categories but leaves CWE-78 and CWE-259, while SafeCoder eliminates CWE-89 (SQL injection) that LPO retains.The authors attribute the remaining CWE-78 and CWE-259 issues likely to their presence in DiSCo after refinement, and suggest limited cohesive DiSCo examples for CWE-89.Source: H CWE Analysis · p. 18, Figure 4 · p. 19

I Over-engineering via Refinement

  • Further refinement can add unnatural security layers, such as lexical analysis and excess exception handling, to code that still contains a vulnerability.The example concerns CWE-78 and may deteriorate code utility when used as training data.Source: I Over-engineering via Refinement · p. 18, Figure 5 · p. 19
Loading 2506.00419v2…