PaperlayerView exact revision on arXiv

Source-linked AI summary

FRANC: A Lightweight Framework for High-Quality Code Generation

Mohammed Latif Siddiq, Beatrice Casey, Joanna C. S. Santos

arXiv:2307.08220v2cs.SEcs.LG

TL;DR

LLM-generated code can contain vulnerabilities and other quality issues, while improving models through training is time- and resource-intensive. FRANC filters, ranks, and repairs generated code using static rules, quality scores, and prompt engineering. The authors report applying it to Java and Python code from five LLMs and conclude that it helps produce vulnerability-free, comparatively high-quality output.

  • Problem

    Generated code can contain vulnerabilities and other quality issues, while improving code-generation models through training is time- and resource-intensive.

    Source: §I · p. 1

  • Method

    FRANC filters generated snippets, ranks them using code-quality tools, and prompts the model to repair detected issues.

    Source: §I · p. 1

  • Results

    The authors demonstrated FRANC on Java and Python code generated by five LLMs and report that it improves generated-code quality.

    Source: §IX · p. 10

  • Takeaways & Limitations

    The authors conclude that FRANC helps obtain vulnerability-free, comparatively high-quality generated source code.

    Source: §IX · p. 10

    The evaluation covered transformer-based LLMs and default generation lengths, so results may not generalize to other inference hyperparameters.

    Source: §VII · p. 10

Abstract

from arXiv · show

In recent years, the use of automated source code generation utilizing transformer-based generative models has expanded, and these models can generate functional code according to the requirements of the developers. However, recent research revealed that these automatically generated source codes can contain vulnerabilities and other quality issues. Despite researchers' and practitioners' attempts to enhance code generation models, retraining and fine-tuning large language models is time-consuming and resource-intensive. Thus, we describe FRANC, a lightweight framework for recommending more secure and high-quality source code derived from transformer-based code generation models. FRANC includes a static filter to make the generated code compilable with heuristics and a quality-aware ranker to sort the code snippets based on a quality score. Moreover, the framework uses prompt engineering to fix persistent quality issues. We evaluated the framework with five Python and Java code generation models and six prompt datasets, including a newly created one in this work (SOEval). The static filter improves 9% to 46% Java suggestions and 10% to 43% Python suggestions regarding compilability. The average improvement over the NDCG@10 score for the ranking system is 0.0763, and the repairing techniques repair the highest 80% of prompts. FRANC takes, on average, 1.98 seconds for Java; for Python, it takes 0.08 seconds.

I. INTRODUCTION

LLM-generated code can contain vulnerabilities and other quality issues, while improving models through training requires substantial data and resources. FRANC instead filters, ranks, and repairs generated code without requiring model retraining.

I. INTRODUCTION

  • LLM-generated code can contain code smells and vulnerabilities that may negatively affect deployed systems’ security and reliability.Prior work cited in the paper found harmful coding patterns in fine-tuning samples and vulnerabilities in GitHub Copilot output.Source: §I · p. 1
  • Improving training data quality requires vetting datasets, while fine-tuning and pre-training demand substantial computing resources.The paper notes that fine-tuning requires at least one GPU and gives a 2.7-billion-parameter model example using a 24 GB GPU.Source: §I · p. 1
  • FRANC filters generated code, ranks snippets by measured quality, and uses generated-knowledge prompting to repair quality issues.The framework is described as lightweight, configurable, and model-agnostic.Source: §I · p. 1
  • The evaluation used five code-generation models and prompts from existing benchmarks plus 70 StackOverflow-derived prompts; the paper also reports a dataset of 70 prompts.The contributions include automated filtering, prompt-engineering repairs, and an empirical comparison with existing generation and infilling models.Source: §I · p. 2

A. LLM-based Code Generation

  • The paper situates code generation within transformer-based language models, which are fine-tuned on code-related datasets for software-engineering tasks.Earlier sequence-to-sequence approaches used RNNs and LSTMs; transformers use self-attention.Source: A. LLM-based Code Generation · p. 2

B. Code Quality and Code Smells

The paper treats code quality in terms of requirements, standards, and the absence of smells, with special attention to the first generated suggestion. Its motivating example shows why a secure alternative lower in the model’s ranking may need to be surfaced.

B. Code Quality and Code Smells

  • Code quality is described broadly as code free of bugs and conformant to requirements, while the chosen quality indicators depend on vendor demands.Coding standards vary by language; PEP-8 is given as a Python example.Source: B. Code Quality and Code Smells · p. 2
  • Code smells indicate poor design or implementation practices and can hinder maintenance or reliability; security smells are patterns that may signal vulnerabilities.A security smell may indicate vulnerability without constituting one by itself.Source: B. Code Quality and Code Smells · p. 2
  • FRANC focuses on coding-standard adherence and the absence of code and security smells, prioritizing the top generated snippet because engineers often focus on it.The paper’s examples include an unhandled ValueError and use of MD5, identified as a risky cryptographic algorithm.Source: B. Code Quality and Code Smells · p. 2

C. Motivating Example

  • In the motivating example, GitHub Copilot’s top-ranked code contains SQL injection, while a lower-ranked suggestion uses a parameterized query to avoid it.The paper argues for lightweight ranking because developers are biased toward the first suggestion they see.Source: C. Motivating Example · pp. 2–3

III. OUR FRAMEWORK (FRANC)

FRANC is a five-phase, model-agnostic framework whose phases can be tailored to the programming language and model. It starts with a developer prompt, uses an existing model to generate suggestions, and then filters, ranks, and repairs those suggestions.

III. OUR FRAMEWORK (FRANC)

  • FRANC’s five phases are model-agnostic and can be tailored to the programming language and LLM in use.Its design relies on the possibility of lower-ranked secure alternatives and on treating repair as a code-generation task with an explicit fix instruction.Source: §III · p. 3
  • Static filtering removes unnecessary code and syntax errors, leaving eligible snippets; later phases rank the survivors and repair problematic code through engineered prompts.The ranking is configurable, and repair prompts instruct the model to fix quality issues such as a buffer overflow at a specified line.Source: §III · p. 3

A. Phase 1: Prompt Creation

  • The engineer’s prompt specifies the expected behavior of the code to be generated.The example prompt includes an import, a function declaration, and a comment describing intended functionality.Source: A. Phase 1: Prompt Creation · p. 3

B. Phase 2: Code Generation

  • FRANC passes the prompt to an existing open- or closed-source LLM, whose sorted suggestions form the suggestion inventory.Generated output may repeat the prompt or include unnecessary or syntactically incorrect code.Source: B. Phase 2: Code Generation · pp. 3–4

C. Phase 3: Static Filtering

FRANC uses language-configurable rules to clean and check generated snippets, then ranks eligible code by a configurable quality score. If the best-ranked snippet remains below an acceptable threshold, it can ask the model to repair detected issues.

C. Phase 3: Static Filtering

  • Static filtering removes code that does not match the prompt’s intent and checks whether each remaining snippet is syntactically correct.The rules are configurable for the language; the example Python rule removes code after the prompted function.Source: C. Phase 3: Static Filtering · p. 4

D. Phase 4: Quality-based Ranking

  • The ranker sorts snippets by a weighted average of quality factors, with higher scores representing higher quality and ties preserving the model’s original order.Factors can represent attributes such as security, performance, or code smells; the best-scoring snippet is presented first.Source: D. Phase 4: Quality-based Ranking · p. 4

E. Phase 5: LLM-based Code Quality Repair

  • When the top-ranked snippet’s quality score is below threshold τ, FRANC uses static-analyzer findings to prompt the model to fix the detected issue.Repair is treated as code generation with an instruction targeting a problematic location.Source: E. Phase 5: LLM-based Code Quality… · p. 4
  • FRANC offers three repair-prompt structures: append a fix comment, append that comment and the original prompt, or provide code only up to the first issue.Prompts can address multiple issues, but FRANC attempts repair only once.Source: E. Phase 5: LLM-based Code Quality… · pp. 4–5

IV. FRAMEWORK EVALUATION

The evaluation examines FRANC’s static filtering, quality-based ranking, code repair, and runtime overhead. It tests the framework on Python and Java code generated by five models.

IV. FRAMEWORK EVALUATION

  • The evaluation applies FRANC’s components to Python and Java code generated by five different LLMs to assess the framework’s applicability and usefulness.The authors chose the two languages as popular programming languages and to demonstrate the configurable architecture across models.Source: §IV · p. 5
  • RQ1 measures how effectively the static filter corrects and removes non-compilable suggestions.The filter uses lightweight rules to clean Python and Java code before checking compilability.Source: §IV · p. 5
  • RQ2 compares FRANC’s quality-based ranking of filtered snippets with each model’s original ranking.The ranker sorts snippets that passed the static filter.Source: §IV · p. 5
  • RQ3 examines whether the code-generation model can repair quality issues through generated-knowledge prompting.FRANC uses the existing model to fix a snippet with quality problems.Source: §IV · p. 5
  • RQ4 measures the time overhead added by filtering, ranking, and repair.The framework adds these phases on top of static analyzers and an existing code-generation model.Source: §IV · p. 5

A. Prompts Creation

The evaluation draws on existing code-generation benchmarks and SOEval, a prompt set created from Stack Overflow questions. Prompts ask models to produce a method or function body from documentation and a signature.

A. Prompts Creation

  • The prompt collection combines four existing code-generation benchmarks with SOEval, a dataset created from Stack Overflow questions.The prompts ask for a method or function body based on a docstring or JavaDoc and the method or function signature.Source: A. Prompts Creation · p. 5
  • SOEval began with 500 popular, recent Python questions and 500 Java questions, then applied inclusion and exclusion criteria to select prompts.Selected questions had to ask how to do something, include code, and have an accepted answer with code; the criteria excluded open-ended, API-finding, configuration, and syntax-specific questions.Source: A. Prompts Creation · pp. 5–6
  • SOEval contains 28 Java prompts and 42 Python prompts after screening the initial 1,000 questions.Source: A. Prompts Creation · pp. 5–6
  • The full evaluation set contains 1,151 prompts: 594 for Java and 557 for Python.Source: A. Prompts Creation · p. 6

B. Code Generation Models

The study uses four open-source code-generation models and one closed-source model. It builds a large suggestion inventory by requesting multiple completions for each prompt, with token limits tailored to model output.

B. Code Generation Models

  • The suggestion inventory used five code-generation models, the first four open-source and the last closed-source; the experiment used the OpenAI version released in May 2023.Source: B. Code Generation Models · p. 6

Suggestion Inventory Creation:

  • Each model was instructed to generate 10 suggestions per prompt, producing an inventory of 102,100 snippets: 41,010 Java and 61,090 Python.The open-source models had a 128-token generation limit, while GPT-3.5-Turbo had a 512-token limit to accommodate explanations in its output.Source: Suggestion Inventory Creation · p. 6
  • The 128-token limit for open-source models was set at about 2.5 times the 50-token average of insecure examples in SecurityEval; GPT-3.5-Turbo’s higher limit accounted for explanations.Source: Suggestion Inventory Creation · p. 6

C. Static Filtering

FRANC cleans generated snippets with language-specific rules, ranks compilable snippets using a quality score, and attempts to repair the top-ranked snippet when it falls below a threshold. The evaluation also notes that the rule descriptions are abbreviated in the paper’s main text.

C. Static Filtering

  • The static filter applies six cleanup rules before syntax checking, including removing extraneous text or code and iteratively completing incomplete Java code.A seventh rule replaces the old problematic code with the repaired source when a prompt is for quality repair.Source: C. Static Filtering · p. 6

D. Quality-based Ranking

  • For Python, FRANC uses Bandit to detect security smells, and for Java it uses SpotBugs to detect code smells; its quality score promotes compilable, smell-free suggestions.The ranker sorts the eligible snippets that passed filtering.Source: D. Quality-based Ranking · pp. 6–7
  • Rule descriptions are superficial in the paper’s main text because of space constraints; examples and fuller details appear in supplementary materials.Source: D. Quality-based Ranking · p. 6

E. Repair by Generated Knowledge Prompting

  • When the top suggestion’s quality score is below threshold τ=1, FRANC uses smell descriptions and locations to prompt the same model to generate 10 repair suggestions.The evaluation used three prompt structures and created 1,023 repair prompts, 341 per structure.Source: E. Repair by Generated Knowledge… · p. 7

V. EVALUATION RESULTS

FRANC’s static filter increased compilability, while its quality-based ranking improved suggestion ordering for both Java and Python. The ranking also increased the number of prompts whose first suggestion was compilable, functionally correct, and smell-free.

A. RQ1: Static Filter Effectiveness

  • 8.6%–46% for Java and 12.8%–43% for open-source Python, with 10.8% for GPT-3.5: static filtering increased compilable suggestions.For Python, the 12.8%–43% range applies to open-source models; GPT-3.5’s improvement is 10.8%.Source: A. RQ1: Static Filter Effectiveness · p. 7
  • Java’s gains were more noticeable: less than 2% of prompts had at least one compilable Java snippet on average, versus 85.4% for Python.The reported averages are 1.7% for Java and 85.4% for Python.Source: A. RQ1: Static Filter Effectiveness · p. 7

B. RQ2: Quality-based Ranking Effectiveness

  • 9.6% for Java and 7% for Python: FRANC’s average NDCG@10 percentage increase over original rankings; both differences were significant (p<0.0001).Table II compares average NDCG@10 scores for original model rankings and FRANC rankings.Source: Table II · p. 8, B. RQ2: Quality-based Ranking… · p. 8
  • FRANC also increased the number of prompts whose first suggestion was compilable, functionally correct, and free of smells.The study manually analyzed prompts and found more with rel1 = 3 after ranking.Source: B. RQ2: Quality-based Ranking… · p. 8

C. RQ3: Code Repair Effectiveness

FRANC repairs a problematic top-ranked suggestion through prompt engineering, with different repair prompt structures performing best for Java and Python. The techniques produced good suggestions for up to 80% of prompts.

C. RQ3: Code Repair Effectiveness

  • P3 performed best for Java, generating good prompts 2.2%–56.1% of the time; P2 performed better for Python, at 35.71%–79.2%.For Python, P2’s difference from P1 was close, and GPT-3.5 was an exception to P2 performing better.Source: C. RQ3: Code Repair Effectiveness · p. 8
  • Overall, models repaired Python samples better than Java samples; the authors suggest heavier Python training as one possible explanation.The proposed explanation is explicitly framed as possible.Source: C. RQ3: Code Repair Effectiveness · p. 8
  • Up to 80% of prompts had at least one good suggestion after FRANC’s repair process.A good prompt has at least one snippet with Q(ci) = 1.Source: C. RQ3: Code Repair Effectiveness · p. 8

D. RQ4: Extra Overhead from the Framework

FRANC adds under two seconds of average runtime, with ranking as its most time-consuming phase. Its discussion reports different outcomes across model families and code-repair tasks.

D. RQ4: Extra Overhead from the Framework

  • Running all FRANC phases took 1.984502 seconds on average for Java (σ = 0.06) and 0.084635 seconds for Python (σ = 0.003); ranking was the most time-consuming phase because it ran external quality-scoring tools.Source: D. RQ4: Extra Overhead from the… · p. 9

VI. DISCUSSION

  • 22% of prompts had at least one compilable InCoder snippet after filtering, although InCoder showed lower NDCG@10 improvement than other models.FRANC is geared toward synthesis but also improved the top-ranked code quality for this infilling model.Source: §VI · p. 9
  • GPT-3.5 performs better than most models at generating suggestions, and FRANC significantly improves their ranking; open-source models’ repair responses vary by language.The paper reports that GPT-3.5 better understands different repair scenarios.Source: §VI · p. 9
  • For Python, LLMs were better at repairing XML validation, subprocess API, and Flask debug=True issues, but less capable on CWE-327, CWE-22, and CWE-732.Source: §VI · p. 9
  • For Java, LLMs resolved array-toString, suspicious reference comparison, and ignored-return-value smells, but struggled with infinite loops, out-of-bounds indexing, and useless control flow.Source: §VI · p. 9

VII. THREATS TO VALIDITY

The paper notes validity limits tied to the models, inference settings, and manual relevance scoring. It distinguishes FRANC from approaches that modify or fine-tune code-generation models and calls for evaluation with other languages and models.

VII. THREATS TO VALIDITY

  • Results may not generalize beyond transformer-based LLMs or the tested inference settings: defaults used 128 tokens for open-source models and 512 for ChatGPT.The authors also identify Bandit and SpotBugs as the external tools used.Source: §VII · p. 10
  • Manual relevance scoring is prone to bias, though peer review showed strong agreement; the manually curated StackOverflow prompt dataset is presented as a proxy for real-world prompts.Source: §VII · p. 10

VIII. RELATED WORK

  • Unlike approaches that change or fine-tune models, FRANC filters, ranks, and repairs existing outputs without retraining and lets developers configure quality-score weighting.The paper contrasts its approach with property-specific guidance and a self-refinement method that fine-tunes models.Source: §VIII · p. 10

IX. CONCLUSION AND FUTURE WORK

  • The authors report improving Java and Python code from five LLMs and identify evaluation with other programming languages and learning models as future work.Source: §IX · p. 10
Loading 2307.08220v2…