PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models for Code: Security Hardening and Adversarial Testing

Jingxuan He, Martin Vechev

arXiv:2302.05319v5cs.CRcs.LGcs.PLcs.SE

TL;DR

Language models for code can generate functionally correct but insecure programs, motivating security control that does not sacrifice correctness. The paper formulates controlled code generation and proposes SVEN, which uses trainable continuous prefixes and curated security-fix data without changing LM weights. On CodeGen with 2.7B parameters, SVEN changes secure-generation rates from 59.1% to 92.3% for hardening and 36.8% for adversarial testing while closely matching original functional-correctness scores.

  • Problem

    Language models often generate insecure code despite functional correctness, creating a need to control code security while retaining usefulness.

  • Method

    SVEN uses property-specific continuous prefixes, specialized region-based losses, and a manually curated security-fix dataset while keeping the LM’s weights unchanged.

  • Results

    On a 2.7B-parameter CodeGen LM, secure-generation rates changed from 59.1% to 92.3% with hardening and 36.8% with adversarial testing, while functional correctness closely matched the original LM.

  • Takeaways & Limitations

    Controlled code generation enables both security hardening and adversarial testing of code-generating LMs under a functional-correctness constraint.

  • Takeaways & Limitations

    SVEN does not generalize to certain evaluated CWEs and currently covers only Python and C/C++.

Abstract

from arXiv · show

Large language models (large LMs) are increasingly trained on massive codebases and used to generate code. However, LMs lack awareness of security and are found to frequently produce unsafe code. This work studies the security of LMs along two important axes: (i) security hardening, which aims to enhance LMs' reliability in generating secure code, and (ii) adversarial testing, which seeks to evaluate LMs' security at an adversarial standpoint. We address both of these by formulating a new security task called controlled code generation. The task is parametric and takes as input a binary property to guide the LM to generate secure or unsafe code, while preserving the LM's capability of generating functionally correct code. We propose a novel learning-based approach called SVEN to solve this task. SVEN leverages property-specific continuous vectors to guide program generation towards the given property, without modifying the LM's weights. Our training procedure optimizes these continuous vectors by enforcing specialized loss terms on different regions of code, using a high-quality dataset carefully curated by us. Our extensive evaluation shows that SVEN is highly effective in achieving strong security control. For instance, a state-of-the-art CodeGen LM with 2.7B parameters generates secure code for 59.1% of the time. When we employ SVEN to perform security hardening (or adversarial testing) on this LM, the ratio is significantly boosted to 92.3% (or degraded to 36.8%). Importantly, SVEN closely matches the original LMs in functional correctness.

1 INTRODUCTION

The paper introduces controlled code generation to steer language models toward secure or unsafe code while preserving functional correctness. It proposes SVEN, a lightweight prefix-based method, and reports strong security control across hardening and adversarial testing.

  • Security Hardening and Adversarial Testing: Controlled code generation adds a binary security property to guide an LM toward secure or unsafe code.The task supports both security hardening and adversarial testing while targeting functional correctness.
  • Our Solution: SVEN: SVEN keeps LM weights unchanged and learns property-specific continuous prefixes that steer generation through the model’s hidden-state computation.The prefixes are lightweight, comprising approximately 0.1% of the LM parameters in the reported experiments.
  • Our Solution: SVEN: SVEN balances security control with functional correctness by applying specialized losses to changed and unchanged code regions.Changed regions receive security-oriented objectives, while unchanged regions are regularized toward the original LM’s next-token distributions.
  • Training Data: The manually curated training dataset contains approximately 1.6k programs selected to improve data quality and suitability for code-completion scenarios.The curation addresses limited generalizability and undesirable artifacts in existing vulnerability datasets.
  • Evaluation: 92.3% secure-code generation under security hardening and 36.8% under adversarial testing compare with 59.1% for the original 2.7B-parameter CodeGen LM.SVEN’s pass@k scores closely match the original LMs on HumanEval, indicating preserved functional correctness.

2 BACKGROUND AND RELATED WORK

Large language models generate functionally correct code but remain vulnerable to security failures. This section situates controlled code generation among code-generation and classical vulnerability-analysis tasks.

  • Code Generation with Large Language Models: Transformer-based code LMs generate tokens autoregressively from prompts, updating hidden states and next-token probabilities at each step.Generation proceeds left to right until an end token or length budget is reached.
  • LMs’ Benefits in Programming Productivity: Code completion engines such as GitHub Copilot demonstrate the practical reach of large code LMs.Codex powers Copilot, which has been used by more than 1M developers and 5K businesses.
  • Code Security and Vulnerability: Existing vulnerability work spans detection, repair, and injection, but these tasks operate retrospectively on code that has already been written.Controlled generation instead targets code completion before the user finishes writing the program.
  • Security of LMs for Code: Evaluations report that state-of-the-art LMs generate insecure code for approximately 40% of security-sensitive cases.These evaluations cover Copilot and other LMs across vulnerabilities including MITRE’s top-25 weaknesses.
  • Enhancing or adversarially degrading security: Copilot introduced a scheme blocking insecure coding patterns, while poisoning attacks can increase insecure cryptographic-parameter suggestions.The paper compares its approach with both prevention and poisoning-based strategies.

3 CONTROLLED CODE GENERATION

Controlled code generation adds a binary security property to a code-generation prompt. The property directs the LM toward secure or unsafe code while the task remains code completion rather than retrospective analysis.

  • Controlled Code Generation: The binary property c ∈ {sec, vul} requests secure code for security hardening or unsafe code for adversarial testing.The same formulation supports both improving and degrading an LM’s security behavior.
  • Controlled Code Generation: Controlled code generation preserves the original code-completion setting while conditioning generation on the requested property.Programs are generated left to right as with a standard LM.
  • Differences from Related Security Tasks: Unlike vulnerability detection, controlled generation reverses the input-output relationship by generating code from a requested property rather than predicting a property for complete code.Repair and injection additionally assume that a complete program is already known to be unsafe or secure.

4 SVEN: INFERENCE, TRAINING, AND DATA

SVEN controls security through lightweight, property-specific continuous prefixes attached to frozen language models. It trains these prefixes on curated security fixes using region-specific objectives that balance security control with functional correctness.

  • Inference: SVEN keeps LM weights frozen and learns separate continuous prefixes for secure and vulnerable generation.The selected prefix is prepended as the LM’s initial hidden states and influences later computations through attention.
  • Inference: SVEN adds approximately 0.1% parameters, and its prefixes can be attached or detached independently during inference.The secure and vulnerable prefixes are trained jointly but can be retained separately after training.
  • Training: Training separates changed security-sensitive regions from unchanged neutral regions, enforcing the requested property in the former and preserving the original LM distribution in the latter.The objectives include conditional language modeling, contrastive security-versus-vulnerability learning, and KL-divergence regularization.
  • Training: Each training sample is a program, token mask, and security property, paired with a version having the opposite property.Masks can be defined at program, line, or character level using diffs between insecure and secure code.
  • Constructing High-quality Training Dataset: Existing commit-level vulnerability datasets contain false positives, irrelevant artifacts, and project-specific fixes that limit suitability for daily code completion.SVEN’s curation removes samples that fail its quality criteria.
  • Constructing High-quality Training Dataset: Manual curation produces 1,606 programs, or 803 insecure-secure pairs, covering nine CWEs in C/C++ and Python.The dataset is split randomly 9:1 into training and validation, prioritizing quality over quantity.

5 SVEN: USE CASES

SVEN supports two use cases: hardening an LM with the secure prefix and adversarially testing it with the vulnerable prefix. The latter also raises ethical concerns because preserved functionality can make malicious behavior difficult to detect.

  • Security Hardening: Security hardening always supplies SVENsec to improve the target LM’s reliability in generating secure programs.The target may be an open-source or non-public LM, assuming the user can train SVEN on it.
  • Adversarial Testing: SVENvul enables adversarial testing intended to assess an LM’s security level and support debugging of weak points and attack vectors.The stated intent is benign evaluation rather than deployment of unsafe generation.
  • Potential Ethical Concerns: SVENvul can be misused to redistribute modified open-source LMs or operate malicious code-completion services and plugins.Preserving functional correctness is described as critical to hiding the malicious purpose.
  • Adversarial Testing: SVEN differs from poisoning attacks because it acts on trained LMs without requiring poisoned data or fine-tuning during the attack.The comparison also notes that SVEN evaluates diverse CWEs and stronger LMs than the cited poisoning work.

6 EXPERIMENTAL EVALUATION

SVEN provides strong security control across models and settings while preserving functional correctness. Its evaluations also show robustness, cross-model applicability, and generalization to some unseen CWEs.

  • Main Experiments: SVENsec raises CodeGen-2.7B’s security rate from 59.1% to 92.3% on the main CWEs.Across CodeGen model sizes, SVEN consistently provides strong security control.
  • Main Experiments: SVENsec almost always increases or maintains security rates across individual CodeGen-2.7B scenarios, reaching nearly 100% for several CWEs.The only reported exception is CWE-416 1-c, where SVENsec decreases the security rate by 11.3%.
  • Main Experiments: SVEN closely matches the original CodeGen models’ HumanEval pass@k scores, with only slight reductions in some cases.The authors conclude that SVEN preserves functional correctness while controlling security.
  • Ablation Studies: Increasing w_CT to 4 balances security control and functional correctness, whereas values above 4 significantly deteriorate functional correctness.The weighting controls the security-related training objective relative to preservation of model behavior.
  • Ablation Studies: Setting w_KL to 1.6 produces desirable security control and functional correctness, while larger values hinder security control through excessive constraint.The KL divergence loss constrains prefixes to preserve functional correctness.

7 CONCLUSION

The work addresses security hardening and adversarial testing for code-generating LMs through controlled code generation and SVEN. SVEN steers generation toward secure or unsafe code while closely maintaining functional correctness.

  • SVEN addresses security hardening and adversarial testing through controlled code generation, which conditions generation on a binary security property.The property specifies whether the LM should generate secure or unsafe code.
  • SVEN learns continuous prefixes that steer program generation without altering the LM’s weights.The prefixes are property-specific and guide generation toward the requested security property.
  • SVEN trains on a curated high-quality dataset and applies specialized loss terms separately to changed and unchanged code regions.The training procedure uses changed/unchanged regions to enforce security control while preserving original behavior.
  • SVEN achieves strong security control while closely maintaining the original LM’s functional correctness.

A MORE DETAILS ON EXPERIMENTAL SETUP

The supplementary setup describes excluded or adapted evaluation scenarios, prompt perturbations, and SVEN’s lightweight training configuration. It also covers evaluations on CWEs unseen during training.

  • Changes to Individual Evaluation Scenarios: The evaluation excludes CWE-079 2-c and CWE-476 1-c for task-specific modeling and evaluation reasons.CWE-079 2-c cannot be modeled as left-to-right completion, while CWE-476 1-c always produces unsafe code under its prompt.
  • Changes to Individual Evaluation Scenarios: The supplementary scenarios include Python conversions of CWE-078 and CWE-022 from their original C/C++ versions.The conversions reflect the prevalence of Python samples for these CWEs in training.
  • Hyperparameters and Training Cost: SVEN’s prefix parameters are approximately 0.1% of total LM parameters, with model-specific prefix lengths and KL-regularization weights.The training configuration fixes w_CT at 4.0 and varies w_KL across models.
  • Detailed Descriptions for Prompt Perturbations: Prompt perturbations modify metadata, formatting, spelling, comments, and wording around the base scenario “con”.Listed perturbations include changing the Python author flag, indentation, misspellings, comments, and the words “remove” or “delete”.

B MORE RESULTS ON SECURITY

Additional security results break down individual CWE scenarios across CodeGen model sizes and temperatures. For SVENsec, lower-temperature evaluation yields 100% security on all but two scenarios.

  • SVENsec achieves a 100% security rate for every individual scenario except CWE-476 0-c and CWE-787 0-c at lower temperature.Figure 19 reports the breakdown for the 2.7B model at temperature 0.1.
  • Lower temperature makes the LM more certain and tends to produce either fully secure or fully insecure outcomes for an individual scenario.
  • Figures 20 and 21 provide individual-scenario security breakdowns for CodeGen-350M and CodeGen-6.1B at temperature 0.4.
  • Table 7 reports absolute counts of valid, secure, non-compiled or non-parsed, and duplicate programs for Figure 10.The statistics are averaged across 10 runs.

C MORE EXAMPLES OF GENERATED CODE

The supplementary examples contrast unsafe patterns from the LM and SVENvul with safer patterns from SVENsec across command execution, formatting, web output, deserialization, credentials, and SQL queries.

  • CWE-089, SQL Injection: LM and SVENvul construct SQL commands with string formatting, while SVENsec uses query parameters to prevent SQL injection.
  • CWE-078, OS Command Injection: LM and SVENvul construct OS commands using string formatting, whereas SVENsec passes arguments as a list to enable escaping and quoting.
  • CWE-787, Out-of-bounds Write: SVENsec increases security for float formatting by generating fixed-width format specifiers such as %.3f.The unsafe example allocates 10 bytes although sprintf may produce up to 47 characters for %f from float.
  • CWE-079, Cross-site Scripting: LM and SVENvul naively embed username in a web page, while SVENsec almost always escapes username before insertion.
  • CWE-502, Deserialization of Untrusted Data: SVENsec is more likely to use yaml.safe_load for untrusted YAML input, despite the training set containing no safe_load calls.The paper attributes this generalization to other training functions whose names are prefixed with safe_.
  • CWE-798, Hard-coded Credentials: LM frequently hard-codes passwords, whereas SVENsec more often retrieves the correct password from local configuration files.
Loading 2302.05319v5…