PaperlayerView exact revision on arXiv

Source-linked AI summary

GraftyVul: Synthesising Insecure Programs Through Real-World Vulnerability Grafting

Omri Ram, Mitchell Horner, Ron Van der Meyden, Alsharif Abuadbba, Hammond Pearce

arXiv:2608.27928v1cs.CRcs.SE

TL;DR

Existing vulnerability datasets often trade off diversity, reproducibility, and realism. GraftyVul grafts real-world findings into executable open-source programs and validates them with build, test, and exploit checks. It produces broad, verified coverage and supports semantic-fidelity analysis and automated-repair evaluation.

  • Problem

    Existing vulnerability datasets sacrifice at least one of diversity, reproducibility or executability, and realism.

  • Method

    GraftyVul uses LLM-driven agents to graft industry vulnerability findings into curated open-source projects and verify the resulting exploits against seeded flags.

  • Results

    212 verified, exploitable samples span five languages and 23 CWE categories, while semantic embeddings preserve source-vulnerability signatures and outperform standard code embeddings on cross-language clone and CWE classification.

  • Takeaways & Limitations

    GraftyVul provides a reproducible-exploit dataset for broad language and CWE coverage and supports benchmarking automated vulnerability repair tools.

  • Takeaways & Limitations

    Target programs are demonstration or reference applications rather than production systems, and evaluating production-grade hosts remains future work.

Abstract

from arXiv · show

Vulnerability datasets underpin a wide range of security research, including vulnerability detection, automated remediation, and secure code generation. However, existing datasets sacrifice at least one of three desirable properties: diversity (of language or vulnerability type), reproducibility/executability, or realism. We therefore present GraftyVul, a system that constructs vulnerable programs by grafting real-world vulnerabilities into open-source projects. This grounds the dataset in vulnerabilities observed in real-world contexts while harnessing known good build and test environments, enabling exploit-verification scripts to guarantee that an introduced vulnerability successfully alters a program's behaviour. Using GraftyVul, we generate 212 verified and exploitable vulnerable programs spanning five programming languages (Python, TypeScript, Java, Go, and C#) across 23 CWE categories. To evaluate fidelity, we introduce a language- and context-agnostic semantic embedding that compares vulnerabilities by sink, mechanism and host-feature rather than surface code. This approach outperforms standard code embeddings on cross-language clone and CWE classification. These embeddings demonstrate that GraftyVul samples retain a strong semantic signature to their source vulnerability. We additionally compare GraftyVul against 13 widely used datasets, where it attains competitive diversity while being the only reproducible-exploit dataset with broad language and CWE coverage. Finally, we illustrate GraftyVul's practical utility through an industrial case study evaluating a production vulnerability remediation system.

I. INTRODUCTION

Existing vulnerability datasets often trade off language and vulnerability diversity, reproducible exploits, or realism. GraftyVul addresses this by grafting real-world findings into curated open-source projects and validating the resulting samples.

  • Existing datasets often support few languages, lack proof-of-concept exploits, or cover narrow vulnerability ranges.
  • GraftyVul grafts industry-sourced vulnerabilities into functional open-source targets rather than generating vulnerable code from patterns or scratch.
  • Pre-existing tests, build scripts, and exploit verification preserve target functionality and confirm that inserted vulnerabilities alter program behaviour.
  • The system introduces language- and context-agnostic semantic embeddings that compare vulnerabilities by sink, mechanism, and host feature.
  • Against 13 widely used datasets, GraftyVul achieves competitive diversity while uniquely combining broad language and CWE coverage with reproducible exploits.

A. Vulnerability Datasets and Benchmarks

Vulnerability datasets range from mined and curated real-world cases to rule-based synthetic programs and learning-based generation. Existing approaches provide different combinations of coverage, label quality, reproducibility, and realism.

  • Mined datasets assemble vulnerable code and fixes from NVD records, repositories, security issues, and GitHub commits, yielding broad language and CWE coverage.
  • Mined datasets are often label-only, limiting direct reproducibility of the vulnerability.
  • Curated datasets improve label quality through developer verification, package links, vulnerable–patched pairs, and LLM-assisted cleaning.
  • Reproducible datasets pair selected vulnerabilities with proof-of-vulnerability tests or PoCs that distinguish vulnerable and fixed behaviour.
  • Rule-based methods synthesize or inject flaws using predefined patterns, execution traces, or source-to-sink graph transformations.
  • Learning-based methods increasingly use LLMs to generate, inject, or reproduce vulnerable code, sometimes reconstructing build environments and synthesizing PoCs.

C. Code Similarity and Clone Detection

GraftyVul targets Type-4 vulnerable clones by transferring vulnerability semantics across unrelated contexts. Its multi-agent pipeline analyzes a source finding, implements an analogous feature in a functional repository, and verifies the result.

  • Grafting seeks Type-4 vulnerable clones that are semantically equivalent despite syntactic and contextual differences.
  • Structural clone methods remain sensitive to syntax, programming language, and context, motivating a broader semantic representation.
  • Five agents coordinate planning, synthesis, exploitation, and review to create a semantically analogous vulnerability in a target program.
  • Validation: Pre-existing functionality, buildability, and exploitability are checked through repository tests, setup scripts, hidden flags, and PoC validation.
  • Planning: The code context agent traces source-to-sink data flow and records the mechanism, feature, code blocks, and SAST metadata of the finding.
  • Planning: The instruction agent explores the target repository and produces a plan for introducing an analogous vulnerable feature.

D. Synthesis

GraftyVul uses cooperating agents to inject vulnerabilities into curated target programs and validate both functionality and exploitability. Its evaluation examines ablations, semantic similarity, dataset characteristics, and benchmarking utility.

  • Synthesis: The injection agent edits the target repository and may submit a vulnerable patch only after pre-existing tests pass.
  • Validation: Failed exploit or review checks trigger feedback and re-synthesis for up to three iterations.
  • Synthesis: The exploit agent produces a PoC that must retrieve or modify a hidden build-time flag through the introduced vulnerability.
  • Validation: The review agent checks that the exploit uses the vulnerability mechanism, acceptance criteria are met, and the graft matches the source finding.
  • Evaluation: The evaluation varies instruction-agent model and temperature, then ablates the injection agent across Sonnet and Haiku while measuring similarity, pass rate, and attrition.
  • Evaluation: The study uses target programs across several languages and frameworks, including open-source repositories and curated applications.
  • Evaluation: Synthesis rate and termination mode are analyzed per CWE and language, with validation terminating or rejecting infeasible low-quality samples.

A. Generation

GraftyVul evaluates generation by verifying both vulnerability samples and their exploits, then examining synthesis coverage, context fidelity, and resistance to invalid grafts. The resulting dataset excludes samples that exploit the validation oracle rather than a genuine target weakness.

  • Generation: 301 of 1361 infrastructure-complete executions produced a vulnerability sample and PoC, yielding a 22.1% production rate.197 executions timed out from API rate limiting and were excluded.
  • Generation: 212 final samples remained after removing 89 samples that hard-coded the seeded flag instead of exposing a genuine target vulnerability.The excluded samples passed exploit verification because the planted code exposed the flag directly.
  • Generation: 23 of 35 attempted CWE categories were successfully produced.
  • Generation: Data-flow coverage reached 100% for Go and Java, compared with 93.3% for TypeScript, 88.3% for JavaScript, and 76.8% for Python.Coverage measures the proportion of statements on Joern-extracted source-to-sink paths recovered by the context agent.
  • Generation: Contrivance remains the largest threat to dataset quality when agents are instructed to cheat.The validation feedback loop sometimes redirects cheating attempts toward genuine vulnerabilities, so actual pass rates exclude those genuine samples.
  • Generation: Two case studies show grafts preserving a source finding’s mechanism, sink, and feature role across different programs and contexts.The cases trace PyVul findings into synthesised target samples and verifying exploits.

D. Ablations

The supplied ablation passages show the instruction agent guiding the injection agent and evaluating outputs, with code fragments illustrating an unauthenticated mode parameter flow. They do not provide ablation outcomes.

  • D. Ablations: The instruction agent directs the injection agent and defines the review agent’s evaluation criteria.
  • D. Ablations: The supplied passages do not report synthesis-rate, semantic-similarity, or attrition results for the ablation.
  • D. Ablations: The shown Python route reads an unauthenticated request mode and inserts it into a consent string before template rendering.

Java SpEL Injection

The paper illustrates vulnerability grafting through Java SpEL and database-provisioning examples, while also describing evaluation of agent configurations and cheating resistance. These passages emphasize preserved vulnerability structure and configuration trade-offs.

  • Java SpEL Injection: The Java case grafts an unauthenticated caller-controlled mode into an /actuator/preview endpoint, where SpEL evaluates the inserted expression.This transfers the source vulnerability’s mechanism, sink, and feature role from Python/Jinja2 to Java/SpEL.
  • Java SpEL Injection: The database-provisioning examples construct CREATE USER or CREATE ROLE SQL statements by formatting caller-controlled names and passwords into quoted strings.
  • Java SpEL Injection: Sonnet at temperature 0.5 achieved higher semantic similarity than the higher-rate Haiku-at-temperature-0 configuration, leading to selection of Sonnet 4.6 at 0.5 for the instruction agent.The injection-agent comparison also preferred Sonnet over Haiku across semantic similarity and synthesis rate.

E. Semantic Similarity

The paper evaluates semantic similarity by representing vulnerabilities through sink, mechanism, and host feature rather than surface code. It uses AUC to test whether these representations separate related from unrelated pairs across languages and contexts.

  • The proposed embedding summarizes each vulnerability’s sink, mechanism, and surrounding feature into a language- and context-agnostic representation.
  • AUC measures the probability that a randomly chosen related pair receives higher cosine similarity than a randomly chosen unrelated pair.AUC equals 1.0 for perfect separation and 0.5 for chance.
  • On same-language BigCloneBench, raw-code embeddings outperform summaries, with AUC 0.913 against 0.855.
  • On cross-language CLCDSA, summary embeddings reach AUC 0.989 while code embeddings reach 0.903.Code embeddings degrade when language varies, whereas summaries provide language-agnostic performance.

2) CWE Classification:

The evaluation extends semantic embedding comparisons to CWE classification and GraftyVul retrieval, then situates the dataset against competing vulnerability corpora. Results show stronger cross-language CWE separation, semantic fidelity to source findings, and a distinctive combination of reproducibility and breadth.

  • 2) CWE Classification:: Summary embeddings improve CWE separation from 0.624 to 0.761 on PyVul and from 0.573 to 0.708 on OSV.The summary representation outperforms code baselines on both datasets.
  • 3) Retrieval Evaluation:: GraftyVul summary embeddings achieve AUC 0.939 and recall@10 0.467 when retrieving source findings for 212 samples.The evaluation compares retrieval using summary embeddings against raw-code baselines.
  • E. Semantic Similarity: LLM summaries capture vulnerability sink, mechanism, and surrounding feature independently of language and syntax.They support cross-language clone classification at AUC 0.989 and cross-language CWE classification at AUC 0.708.
  • F. Comparison to Other Vulnerability Datasets: GraftyVul’s Vendi diversity score is 102, within the range of large mined corpora whose samples are generally label-only.Every GraftyVul sample is executable and exploit-verified; among more diverse datasets, only ARVO provides a reproducible exploit.
  • F. Comparison to Other Vulnerability Datasets: ARVO has greater measured diversity but narrower vulnerability breadth because it focuses on memory-safety errors in C and C++.
  • F. Comparison to Other Vulnerability Datasets: GraftyVul combines reproducible, exploit-verified samples with broad language and CWE coverage, despite being less diverse than large label-only datasets.

G. Downstream Usage

GraftyVul supports industrial benchmarking of vulnerability autofix systems by combining executable targets, host-feature tests, and verified exploits. Its case studies expose fixes that secure the reported weakness but break functionality or leave sensitive exposure intact.

  • G. Downstream Usage: A single autofix system was evaluated using GraftyVul samples selected to cover CWEs common in Nullify’s open and closed pull requests.Hand-crafted assertions checked both vulnerability removal and preservation of host functionality.
  • G. Downstream Usage: The path-containment fix blocked traversal but rejected legitimate listing of the exports root.The hand-crafted test suite detected this secure-but-over-hardened behavior.
  • G. Downstream Usage: The SQL allowlist closed injection but retained unauthenticated access to the sensitive secrets table.The provided exploit caught the remaining exposure.

3) Securing requires an interface change:

The eval-injection case shows that securing an endpoint can require changing its interface when the unsafe expression input is itself part of the feature contract. Broader benchmarking evidence indicates that verified exploits and host-feature tests reveal failures beyond whether a reported weakness is removed.

  • 3) Securing requires an interface change:: The eval-injection fix replaced caller-supplied expressions with named transforms, securing the endpoint but breaking the transform feature.Because the expression input is the contract, a safe fix must narrow the interface.
  • Benchmarking AVR Tools: GraftyVul’s build scripts enable automated execution, while proof of exploit detects when a patched program remains vulnerable.This supports benchmarking remediation tools on executable behavior rather than patch labels alone.
  • Benchmarking AVR Tools: The validation module confirms that injected samples are genuinely exploitable through pre-curated validation harnesses.This provides a stronger guarantee than LLM-generated harnesses and LLM judgment, according to the paper.
  • Benchmarking AVR Tools: Grafting one finding into multiple host programs creates varied real contexts, with diversity mainly constrained by vulnerability sinks and mechanisms.The exploit-verification harness bounds the sinks, mechanisms, and CWEs the framework can express.
  • Benchmarking AVR Tools: The case studies show that checking only whether a vulnerability is addressed is insufficient for evaluating autofix correctness.Fixes may break protected features or leave the underlying exposure open.

VI. LIMITATIONS AND FUTURE WORK

The paper’s main limitations concern metric and model generalisation, input and host scope, residual contrivance, downstream evaluation breadth, and generation cost. Future work targets broader models, production-grade hosts, additional benchmarks, deterministic verification, and earlier feasibility checks.

  • Metric and model scope: The semantic metric uses a fixed summariser prompt and one model family, leaving prompt sensitivity and cross-family generalisation untested.The ablations are restricted to the Anthropic family.
  • Input and host scope: The code-context agent is integrated with one partner’s SAST pipeline, while production-grade target hosts remain unevaluated.Alternative SAST sources, manually authored contexts, and production systems are future work.
  • Threats to validity: Contrived vulnerabilities remain a residual threat because the validation module is least likely to reject contrived samples.A fraction of such samples can pass verification despite the safeguards.
  • Downstream Utility: The downstream study evaluates one autofix system with hand-crafted host-feature assertions.The paper identifies broader benchmarking frameworks and additional remediation tools as future work.
  • Cost: Failed grafting runs are the most expensive, so earlier feasibility prediction could reduce wasted model invocations.This is identified as a cost-reduction direction.
  • Conclusion: GraftyVul contains 212 verified exploitable samples across five languages and 23 CWE categories, while scaling hosts and adding deterministic verification could widen coverage.The conclusion describes these as promising directions for future expansion.
Loading 2608.27928v1…