PaperlayerView exact revision on arXiv

Source-linked AI summary

How Effective Are Neural Networks for Fixing Security Vulnerabilities

Yi Wu, Nan Jiang, Hung Viet Pham, Thibaud Lutellier, Jordan Davis, Lin Tan, Petr Babkin, Sameena Shah

arXiv:2305.18607v2cs.SEcs.AIcs.CR

TL;DR

Java vulnerability repair lacks sufficient automation and evaluation coverage, especially for a widely used language whose vulnerabilities are not well served by existing benchmarks. The paper compares LLMs and DL-based APR techniques on two real-world benchmarks, adds transformed evaluations to address training-data overlap, and finds that current systems fix few vulnerabilities while fine-tuning improves LLM performance.

  • Problem

    Java vulnerability repair lacks adequate benchmarks and evidence comparing LLMs with DL-based APR techniques, despite the need for faster automated fixes.

  • Method

    The study evaluates five LLMs, four APR-data-fine-tuned LLMs, and four DL-based APR techniques on Vul4J, VJBench, and transformed vulnerabilities designed to mitigate training-data overlap.

  • Results

    Existing LLMs and APR models fix very few Java vulnerabilities overall; Codex is the strongest system, while fine-tuning improves LLM vulnerability-fixing capabilities.

  • Takeaways & Limitations

    Improving automated Java vulnerability repair requires larger vulnerability-repair datasets, fine-tuning LLMs with such data, and code-simplification transformations.

  • Takeaways & Limitations

    Java vulnerabilities are diverse, so the findings might not generalize to all Java vulnerabilities.

Abstract

from arXiv · show

Security vulnerability repair is a difficult task that is in dire need of automation. Two groups of techniques have shown promise: (1) large code language models (LLMs) that have been pre-trained on source code for tasks such as code completion, and (2) automated program repair (APR) techniques that use deep learning (DL) models to automatically fix software bugs. This paper is the first to study and compare Java vulnerability repair capabilities of LLMs and DL-based APR models. The contributions include that we (1) apply and evaluate five LLMs (Codex, CodeGen, CodeT5, PLBART and InCoder), four fine-tuned LLMs, and four DL-based APR techniques on two real-world Java vulnerability benchmarks (Vul4J and VJBench), (2) design code transformations to address the training and test data overlapping threat to Codex, (3) create a new Java vulnerability repair benchmark VJBench, and its transformed version VJBench-trans and (4) evaluate LLMs and APR techniques on the transformed vulnerabilities in VJBench-trans. Our findings include that (1) existing LLMs and APR models fix very few Java vulnerabilities. Codex fixes 10.2 (20.4%), the most number of vulnerabilities. (2) Fine-tuning with general APR data improves LLMs' vulnerability-fixing capabilities. (3) Our new VJBench reveals that LLMs and APR models fail to fix many Common Weakness Enumeration (CWE) types, such as CWE-325 Missing cryptographic step and CWE-444 HTTP request smuggling. (4) Codex still fixes 8.3 transformed vulnerabilities, outperforming all the other LLMs and APR models on transformed vulnerabilities. The results call for innovations to enhance automated Java vulnerability repair such as creating larger vulnerability repair training data, tuning LLMs with such data, and applying code simplification transformation to facilitate vulnerability repair.

1 INTRODUCTION

Java vulnerability repair needs faster, language-specific automation because existing benchmarks and techniques leave important gaps. This study compares LLMs and DL-based APR techniques, finding that current systems fix few vulnerabilities and struggle across many CWE types.

  • Motivation: 60 to 79 days is the average time to fix a vulnerability, leaving opportunities for exploitation.A Log4Shell fix was first deployed 12 days after disclosure, during which attacks were reported.
  • Motivation: Java lacks vulnerability benchmarks and repair solutions despite its widespread use and severe vulnerabilities.Existing benchmarks and techniques often target C/C++ or binaries, while C/C++-focused buffer-overflow techniques are largely irrelevant to type-safe Java.
  • Research Questions: LLMs learn from large source-code corpora, whereas DL-based APR models learn from bug-fix pairs, motivating a direct comparison for Java vulnerability repair.LLM training data can be orders of magnitude larger because LLMs do not require paired bugs and fixes.
  • Research Questions: Fine-tuning with general APR data is studied because the lack of Java vulnerability data makes vulnerability-specific LLM fine-tuning unrealistic.The study asks whether more data plus fine-tuning outperforms data-format matching between pre-trained LLMs and APR models.
  • Methodological Contribution: VJBench-trans tests whether transformed, semantically equivalent vulnerabilities reduce the risk that LLM training data overlaps with evaluation data.Code transformations can also make some vulnerabilities easier to fix, and Codex and fine-tuned CodeT5 are more robust to transformations.
  • Findings: 10.2 (20.4%) vulnerabilities is Codex’s average, the highest fixing result among the evaluated systems.The study evaluates five LLMs, four fine-tuned LLMs, and four APR techniques on Vul4J and VJBench.
  • Findings: VJBench exposes failures on many CWE types, including CWE-325 Missing cryptographic step and CWE-444 HTTP request smuggling.The benchmark adds coverage that reveals limitations not captured by narrower existing evaluations.

2 NEW BENCHMARK OF JAVA VULNERABILITIES

Vul4J provides an important but limited foundation for Java vulnerability repair evaluation. VJBench extends it with reproducible vulnerabilities, broader CWE coverage, and additional cases suitable for automated repair.

  • Limitations of Vul4J: 79 vulnerabilities from 51 projects cover 25 CWE types in Vul4J, but its CWE coverage is limited.Only 35 vulnerabilities are applicable to state-of-the-art learning-based APR systems because those systems fix single-hunk bugs.
  • Benchmark Construction: VJBench selection requires Java-only vulnerabilities, tests that fail before and pass after the fix, vulnerability-specific patches, and non-duplication with Vul4J.These criteria target reproducible real-world vulnerabilities suitable for repair evaluation.
  • Benchmark Construction: The benchmark construction starts from 7,116 GitHub projects and filters to 400 Java projects containing 933 unique vulnerabilities before manual fixing-commit analysis.The source data were downloaded from the National Vulnerability Database on May 13, 2022.
  • VJBench Construction: 42 new reproducible vulnerabilities from 30 open-source projects form VJBench.The dataset contains 27 multi-hunk and 15 single-hunk vulnerabilities.
  • VJBench Coverage: VJBench covers 23 CWE types, introduces 12 types absent from Vul4J, and supplements four CWE types with sparse Vul4J coverage.The supplemented types are CWE-78, CWE-200, CWE-310, and CWE-863.
  • Study Dataset: 15 VJBench single-hunk vulnerabilities combine with 35 applicable Vul4J vulnerabilities to yield 50 vulnerabilities used in the study.The 15 new single-hunk vulnerabilities cover twelve CWE types, including six new types not present in Vul4J.

3 LARGE LANGUAGE MODELS AND APR TECHNIQUES

The study evaluates five pretrained code LLMs, four fine-tuned LLMs, and four learning-based APR techniques for Java vulnerability repair. The models differ in training data, architectures, and patch-generation setups.

  • Large Language Models As Is: Five LLMs—Codex, PLBART, CodeT5, CodeGen, and InCoder—are selected for code-generation capability and substantial source-code training.
  • Large Language Models As Is: Codex is a 12B-parameter GPT-3-based model, and the study uses its insertion mode because preliminary results favored it.
  • Large Language Models As Is: The evaluated LLMs use varied architectures and training corpora, including CodeT5’s 770M parameters and 5.2 million code functions, CodeGen’s 6B parameters, PLBART’s 400M parameters, and InCoder’s 6B parameters.
  • Fine-Tuned Large Language Models: Fine-tuned LLMs are included because fine-tuning adapts pretrained models to downstream tasks, using general APR data rather than vulnerability-specific data.
  • APR Techniques: The study evaluates four Java APR techniques—CURE, Recoder, RewardRepair, and KNOD—with training sizes ranging from 82.87K to 3.51M APR instances.

4 CODE TRANSFORMATION

The paper transforms vulnerability code to reduce training-test overlap while preserving semantic equivalence. It combines identifier renaming and code-structure transformations to create the VJBench-trans benchmark.

  • Transformation Objective: The study transforms existing vulnerabilities into semantically equivalent forms that are intended to be unseen by evaluated LLMs and APR techniques.
  • Identifier Renaming: Identifier renaming replaces project-defined variables, functions, and classes with synonyms while preserving external-library and default Java identifiers.
  • Code Structure Change: Code-structure changes include condition flipping, loop conversion, conditional-statement transformation, function chaining, function-argument passing, and code-order changes.
  • Transformation Validation: Transformations are applied together when applicable, and equivalence is checked by reproducing vulnerabilities with the same tests and applying semantically equivalent patches.
  • VJBench-trans: VJBench-trans contains 150 transformed Java vulnerabilities from three transformation settings, with searches finding no public code identical to the transformed buggy functions.

5 EXPERIMENT SETUP

The experiment benchmarks LLMs, fine-tuned LLMs, and APR techniques on 50 single-hunk Java vulnerabilities using generated patches and project tests. It also compares input formats and validates patch correctness manually.

  • Evaluation Workflow: The study builds VJBench with 42 vulnerabilities, benchmarks original and new datasets, and generates 10 patches per vulnerability for each language model.
  • Datasets and Scope: The benchmark contains 50 single-hunk vulnerabilities: 35 from Vul4J and 15 from VJBench, evaluated with perfect fault localization.
  • Input Setups: LLMs are tested with buggy lines either commented in the input or omitted, and the best-performing setup is reported for each model.
  • Patch Generation: Each model generates 10 candidates; Codex is run 25 times to average its nondeterministic results, while other LLMs are run once.
  • Patch Validation: Top-10 patches are tested against project test cases, and plausible patches are manually inspected to distinguish correct patches from over-fitted patches.

6 RESULTS AND FINDINGS

Across Java vulnerability benchmarks, existing LLMs and APR techniques fix few vulnerabilities, while fine-tuning improves LLM repair capability but does not resolve compilation and vulnerability-context limitations. Codex performs best overall, yet models struggle with complex changes and several CWE types.

  • LLMs vs. APR Techniques: 10.2 vulnerabilities (20.4%) is Codex’s average result, the best fixing capability among the evaluated LLMs and APR techniques.The evaluation considers the top ten generated patches for each vulnerability.
  • LLMs Fine-Tuned with APR Data: Fine-tuning with general APR data improves all four LLMs’ vulnerability-fixing capabilities, with fine-tuned InCoder fixing 9 vulnerabilities.General APR data helps adapt models to patch generation despite the scarcity of vulnerability-specific training data.
  • LLMs Fine-Tuned with APR Data: 79.7% is Codex’s compilation rate, exceeding fine-tuned InCoder’s 55.2% and Recoder’s 57.6%.CodeT5 improves from 6.4% to 46.8% and CodeGen from 35.8% to 47.2% after fine-tuning, while fine-tuned PLBART slightly declines.
  • Vulnerability Types: Except Codex, models generally fix vulnerabilities requiring simple changes such as deleting statements or replacing variable or method names.Examples include changing a keyword, removing code, or replacing a variable or method name.
  • Vulnerability Types: VJBench shows that models fix none of six newly added CWE types, including missing cryptographic steps and HTTP request smuggling.The benchmark exposes limitations on CWE-172, CWE-325, CWE-347, CWE-444, CWE-668, and CWE-1295.
  • Limitations and Implications: Models often miss project-wide syntax, type, method, and vulnerability-context information needed to generate compilable or targeted patches.Observed failures include modifying a final variable, calling an invalid method, supplying an unsupported argument, and making unrelated changes to security-sensitive code.

7 THREATS TO VALIDITY

The study identifies threats involving benchmark coverage, developer-patch correctness, and possible training-data overlap, and describes mitigations for each.

  • Benchmark coverage: Java vulnerability diversity limits how broadly benchmark findings may generalize.The authors mitigate this threat by expanding the existing benchmark with a new vulnerability dataset.
  • Ground truth: Developer patches serve as ground truth, but incorrect fixes could make the evaluation ground truth wrong.The study restricts analysis to reproducible NVD vulnerabilities with tests showing that fixed versions are no longer exploitable.
  • Training-data overlap: Code transformations create semantically equivalent vulnerabilities absent from the presumed training data to test repair on unseen programs.The transformed programs are then evaluated with Codex to assess whether it can repair vulnerabilities it has not seen.

8 RELATED WORK

Prior work applies deep learning and LLMs to vulnerability repair, but this paper emphasizes real-world Java evaluation and the distinct challenges of vulnerability fixing.

  • DL-based vulnerability fixing: Earlier DL vulnerability-repair studies often target C vulnerabilities or synthetic Juliet examples rather than production-like real-world vulnerabilities.The paper evaluates real-world vulnerabilities to better reflect the repair quality researchers and developers may encounter in production code.
  • LLM-based vulnerability fixing: Prior LLM work used zero-shot prompting on small sets of hand-crafted and real-world C vulnerabilities, incorporating static-analysis or sanitizer feedback.The present work differs by studying Java and evaluating LLMs alongside DL-based APR tools and fine-tuned LLMs.
  • Benchmarks: Existing real-world vulnerability datasets may contain only fixing-commit snippets without test cases, limiting evaluation to code matching rather than practical APR.The cited datasets therefore cannot support automated program repair in practice.
  • LLM application domain: LLMs have been explored for software-engineering tasks including repair, completion, and pair-programming, while vulnerability repair remains a distinct and difficult application domain.The paper frames vulnerability repair as an underexplored challenge for LLMs.

9 CONCLUSION

This work systematically investigates LLM and DL-based APR repair of Java vulnerabilities using real-world benchmarks and transformations. It finds that existing approaches fix very few vulnerabilities and motivates larger training data, fine-tuning, few-shot learning, and simplifying transformations.

  • Conclusion: The study evaluates five LLMs, four fine-tuned LLMs, and four DL-based APR techniques on two real-world Java vulnerability benchmarks.It also introduces VJBench and the transformed benchmark VJBench-trans.
  • Conclusion: Existing LLMs and APR models fix very few Java vulnerabilities.The conclusion motivates larger vulnerability-repair datasets, fine-tuning, few-shot learning, and simplifying transformations.
  • Conclusion: The benchmark and research artifacts are publicly available in the replication package.
Loading 2305.18607v2…