PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations

Catherine Tony, Markus Mutas, Nicolás E. Díaz Ferreyra, Riccardo Scandariato

arXiv:2303.09384v1cs.SEcs.IRcs.LG

TL;DR

The security of code generated from natural-language prompts has not been extensively investigated, despite LLMs learning from potentially vulnerable open-source code. LLMSecEval provides 150 CWE-related prompts paired with secure examples and demonstrates their use with GPT-3 and Codex. The prompts received high naturalness scores, though the dataset covers 18 of the Top 25 CWE scenarios and may not suit language-specific weaknesses across languages.

  • Problem

    The security of code generated from natural-language descriptions has not been extensively investigated, and a dataset of security-relevant prompts had not been documented to the authors’ knowledge.

    Source: Motivation · p. 1

  • Method

    LLMSecEval pairs 150 natural-language prompts describing Top 25 CWE scenarios with secure code examples and demonstrates an evaluation application using GPT-3, Codex, and CodeQL.

    Source: Contribution · p. 1, §IV · p. 3

  • Results

    All prompts scored 4 or higher on naturalness; 138 of 150 scored at least 3 on adequacy, and 135 of 150 scored at least 3 on conciseness.

    Source: Language-related Metrics · p. 4, Content-related Metrics · p. 4

  • Takeaways & Limitations

    The prompts can support security evaluation of LLM-generated code and exploratory prompt-engineering studies for secure code generation.

    Source: §VI · p. 4

    The dataset covers only 18 of the Top 25 CWEs from 2021, and language-agnostic prompts for language-specific weaknesses may not suit evaluation across different programming languages.

    Source: §VII · p. 4

Abstract

from arXiv · show

Large Language Models (LLMs) like Codex are powerful tools for performing code completion and code generation tasks as they are trained on billions of lines of code from publicly available sources. Moreover, these models are capable of generating code snippets from Natural Language (NL) descriptions by learning languages and programming practices from public GitHub repositories. Although LLMs promise an effortless NL-driven deployment of software applications, the security of the code they generate has not been extensively investigated nor documented. In this work, we present LLMSecEval, a dataset containing 150 NL prompts that can be leveraged for assessing the security performance of such models. Such prompts are NL descriptions of code snippets prone to various security vulnerabilities listed in MITRE's Top 25 Common Weakness Enumeration (CWE) ranking. Each prompt in our dataset comes with a secure implementation example to facilitate comparative evaluations against code produced by LLMs. As a practical application, we show how LLMSecEval can be used for evaluating the security of snippets automatically generated from NL descriptions.

I. INTRODUCTION

Because LLMs generate code from natural-language descriptions while their training sources can contain vulnerable code, assessing the security of generated code is important. LLMSecEval addresses the lack of documented datasets for this assessment with 150 prompts and secure code examples, alongside a demonstration using GPT-3 and Codex.

Motivation:

  • Open-source training code can contain vulnerabilities, and around 85% of security APIs on GitHub were observed to be misused.Source: Motivation · p. 1
  • A dataset of security-relevant natural-language prompts was needed to assess whether LLMs produce secure implementations, but none had been proposed or documented to the authors’ knowledge.Source: Motivation · p. 1
  • LLMSecEval provides 150 prompts describing software prone to Top 25 CWE vulnerabilities, each paired with a secure code example for comparison with LLM-generated code.Source: Contribution · p. 1
  • The authors demonstrate an application that uses GPT-3 and Codex to generate code from the prompts and CodeQL to identify vulnerabilities.Source: Contribution · p. 1

Contribution:

Prior LLM code datasets emphasized functional correctness, while security-focused work examined code completion or vulnerability repair rather than secure code generation from natural-language prompts.

Contribution:

  • HumanEval and two datasets by Austin et al. [11] evaluate functional or semantic correctness, not the security of LLM-generated code.Source: §II · p. 1
  • Pearce et al. [12] evaluated security by having Copilot complete incomplete code scenarios and analyzing the results.Source: §II · p. 2
  • A later study used the same scenarios for vulnerability repair, focusing on incomplete code with comments rather than natural-language prompts for secure generation.Source: §II · p. 2

III. CREATION OF NL PROMPTS

The authors used CWE-related code scenarios as the basis for generating natural-language prompts, selecting functional programs from an existing Copilot dataset and translating them into descriptions.

III. CREATION OF NL PROMPTS

  • MITRE’s Top 25 CWE list served as the reference for prompts describing code scenarios prone to security flaws; Codex generated descriptions from CWE-related snippets.Source: §III · p. 2
  • A CWE-20 example prompt describes a Python registration page that accepts user details and stores them in a database.Source: §III · p. 2
  • Figure 1 presents the NL-prompt creation process.Source: Figure 1 · p. 2, §III · p. 2

A. Data Source

  • The source dataset contained 54 code scenarios covering 18 of the 2021 Top 25 CWEs, with three scenarios per CWE; seven architectural issues were excluded.Source: A. Data Source · p. 2
  • The scenarios were incomplete snippets, sometimes with natural-language comments, and Copilot generated 25 confidence-ranked completions per scenario, yielding 1,084 valid programs.Source: A. Data Source · p. 2
  • The authors selected the top three functionally correct C or Python samples for each scenario, producing a base corpus of 162 programs.Source: A. Data Source · p. 2
  • Because 40% of the original 1,084 programs contained vulnerabilities, the selected samples were also considered likely vulnerable; the authors say they took steps to prevent those flaws influencing prompts.Source: A. Data Source · p. 2

B. NL Prompts using Codex

Codex translated the selected programs into natural-language descriptions, which the authors manually filtered and edited before converting them into prompts for code generation.

B. NL Prompts using Codex

  • The authors used Codex’s code-davinci-002 model to translate programs into descriptions, limiting outputs to 100 tokens after longer settings produced repeated and invalid results.Source: B. NL Prompts using Codex · p. 2

C. Manual Curation of Responses

IV. DATASET DESCRIPTION

LLMSecEval packages 150 prompts covering 18 Top 25 CWE scenarios with source metadata, quality scores, and secure Python implementations.

IV. DATASET DESCRIPTION

  • The dataset contains 150 NL prompts covering 18 of the Top 25 CWE scenarios.The prompts are provided in CSV and JSON formats.Source: §IV · p. 3
  • Each record identifies the CWE, source-code filepath, vulnerability status, and source language.The dataset marks 85 prompts as generated from vulnerable code; the 150 prompts originate from 83 Python and 67 C programs.Source: §IV · p. 3
  • Four quality-metric scores are included so users can select prompts according to their own requirements.The metrics are described in Section V.Source: §IV · p. 3
  • Each prompt has a corresponding secure Python implementation, created mostly manually and checked with CodeQL.The examples are intended to support comparisons with LLM-generated code.Source: §IV · p. 3

V. NL PROMPTS QUALITY ANALYSIS

The authors assess prompt language and content quality with four metrics, manually scored by two authors, and report strong inter-rater agreement.

V. NL PROMPTS QUALITY ANALYSIS

  • Naturalness, expressiveness, adequacy, and conciseness assess language quality and content richness or relevance.The adopted metrics were proposed by Hu et al..Source: §V · p. 3
  • Two authors manually assigned each metric a score from 1 to 5 using Hu et al.’s criteria.They assessed inter-rater reliability with weighted Cohen’s Kappa.Source: §V · p. 3
  • Kappa was 0.98 for naturalness, 0.83 for expressiveness, 0.8 for adequacy, and 0.88 for conciseness, indicating strong agreement.Disagreements were resolved through verbal discussion.Source: §V · p. 4
  • Fig. 3 presents language- and content-related scores; frequencies below 2 are not labeled.The figure covers the four metrics described in the text.Source: §V · p. 3, Figure 3 · p. 4

Language-related Metrics:

The prompts scored well on naturalness and generally well on expressiveness, though some descriptions included details that could hinder understanding.

Language-related Metrics:

  • All prompts scored at least 4 for naturalness, reflecting fluent English sentences after removal of unnecessary whitespace and special characters.The authors report only a few such formatting issues.Source: Language-related Metrics · p. 4
  • All prompts scored at least 3 for expressiveness, and a majority scored 4 or higher.Some descriptions scored lower because needless function names and implementation details could hinder understanding.Source: Language-related Metrics · p. 4
  • Overall, the authors characterize the prompts as high quality in language fluency.Source: Language-related Metrics · p. 4

Content-related Metrics:

Most prompts scored at least 3 for adequacy and conciseness, while lower-scoring prompts lacked relevant code information or included unnecessary background.

Content-related Metrics:

  • 138 of 150 prompts scored 3 or higher on adequacy.The lower-scoring prompts were too abstract and omitted relevant information from their source code.Source: Content-related Metrics · p. 4
  • The adequacy scores below 3 reflected prompts that were too abstract or omitted relevant code information.Source: Content-related Metrics · p. 4
  • 135 of 150 prompts scored 3 or higher on conciseness.The lower-scoring prompts included unnecessary background about built-in method calls without adding much value.Source: Content-related Metrics · p. 4
  • Lower conciseness scores reflected unnecessary background information about built-in method calls.Source: Content-related Metrics · p. 4

VI. DATASET USAGE FOR SECURE CODE GENERATION

LLMSecEval is intended to support security research on code-generation models by testing whether code produced from natural-language prompts contains CWE-related vulnerabilities. Its application demonstrates evaluation of generated code using CodeQL queries.

VI. DATASET USAGE FOR SECURE CODE GENERATION

  • The dataset supports checking whether code generated from natural-language prompts introduces CWE-related vulnerabilities.Its prompts can also serve as a baseline for designing descriptions that lead to secure implementations.Source: §VI · p. 4
  • The prompts can support exploratory prompt-engineering studies for secure code generation.They can serve as a baseline for descriptions leading to secure implementations.Source: §VI · p. 4
  • The application evaluates code generated by GPT-3 and Codex with built-in CodeQL queries detecting 18 of the Top 25 CWEs.Users select a model and programming language, upload prompts, and download generated code; query results can be stored locally.Source: §VI · p. 4

VII. LIMITATIONS AND FUTURE IMPROVEMENTS

LLMSecEval contains 150 language-agnostic prompts for 18 of the 2021 Top 25 CWE scenarios, each paired with a secure code example. The paper notes limits in CWE coverage and cross-language suitability, and describes plans for expansion and security unit tests.

VII. LIMITATIONS AND FUTURE IMPROVEMENTS

  • The dataset covers only 18 of the 2021 Top 25 CWEs, with plans to expand coverage and update it annually.The authors also plan to design security unit tests tailored to each prompt.Source: §VII · p. 4
  • Language-agnostic prompts for CWEs relevant only to specific programming languages may not suit evaluation across different languages.Source: §VII · p. 4

VIII. CONCLUSION

  • LLMSecEval contains 150 prompts covering 18 of the 2021 Top 25 CWE scenarios, each paired with a secure code example.An application demonstrates its use to assess code generated by GPT-3 and Codex.Source: §VIII · p. 4
Loading 2303.09384v1…