Source-linked AI summary
LLMVul: A Vulnerability-Labeled Dataset of LLM-Generated C/C++ Functions from Real Production Repositories
Mohammad Farhad, Shuvalaxmi Dass
TL;DR
Security research lacks vulnerability-labeled evidence from LLM-generated code committed in real production projects. LLMVul mines and labels such C/C++ functions from AI-assisted GitHub development, producing 21,430 unique functions from 226 repositories and 1,540 ensemble-vulnerable functions across 17 CWE categories. The dataset preserves vulnerability, provenance, and AI-tool metadata for reproducible security and generalization studies.
Problem
Existing vulnerability datasets focus mainly on human-written code or controlled prompting, limiting study of LLM-generated code as it appears in production repositories.
Method
LLMVul mines AI-attributed C/C++ commits, extracts and deduplicates functions, and labels them with a three-tool static-analysis ensemble validated by human raters.
Results
21,430 unique C/C++ functions from 226 repositories include 1,540 ensemble-vulnerable functions spanning 17 CWE categories.
Takeaways & Limitations
Repository provenance, vulnerability labels, CWE identifiers, AI-tool attribution, and temporal metadata support reproducible research on LLM-generated code security and generalization.
Takeaways & Limitations
AI attribution depends on explicit Git metadata, and sampling the top 1,200 repositories may underrepresent smaller, private, less-established, or embedded codebases.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used to generate and assist with software development, yet existing vulnerability datasets largely focus on human-written code or controlled prompting environments. This limits the ability to study security weaknesses in LLM-generated code as it appears in real-world software projects. We present LLMVul, a vulnerability-labeled dataset of LLM-generated C/C++ functions mined from real production repositories. We mine AI-assisted development activity from GitHub over a 4 year period, from November 13, 2022 to September 3, 2026, using provenance signals such as commit metadata and AI-related authorship evidence. After filtering and deduplication, LLMVul contains 21,430 unique C/C++ functions from 226 repositories, together with repository, commit, function, provenance, and AI-tool metadata. We establish vulnerability labels using an ensemble of complementary static-analysis and pattern-based techniques and assign Common Weakness Enumeration (CWE) categories to confirmed vulnerable functions. To assess labeling reliability, we additionally conduct independent manual annotation and measure inter-rater agreement using Cohen's kappa ($k=0.79$). LLMVul contains 1,540 ensemble-vulnerable functions spanning 17 unique CWE categories, providing substantially more real-world LLM-generated vulnerable C/C++ functions than existing vulnerability-oriented LLM code benchmarks. By preserving both code-level vulnerability labels and generation/provenance metadata, LLMVul enables reproducible research on vulnerability detection, security evaluation of LLM-generated code, and analysis of vulnerability patterns in AI-assisted software development. The LLMVul dataset is publicly available at https://doi.org/10.5281/zenodo.22668216.
1 INTRODUCTION
LLM coding assistants are increasingly used in production, but security research lacks vulnerability-labeled data from code developers actually commit. LLMVul addresses this gap with labeled C/C++ functions mined from AI-assisted GitHub development.
- LLM-generated code is increasingly used in production, including security-sensitive systems software, embedded firmware, and network services.
- The dataset gap: Existing security benchmarks mainly use controlled prompts and therefore do not capture vulnerabilities in developer-committed production code.
- Prior real-world studies identify AI-attributed commits but do not provide function-level vulnerability labels, while established C/C++ vulnerability datasets contain human-authored code.
- Our work: LLMVul contains 21.4k unique C/C++ functions from 226 repositories and 1,684 LLM-attributed commits, including 1,540 vulnerable functions across 17 CWE categories.
- Our work: The dataset combines provenance and AI-tool metadata with ensemble-generated labels validated by human raters, achieving substantial agreement (𝜅= 0.79).
2 RELATED WORK
Existing AI-code security datasets span human-written benchmarks, prompted code, and repository-mined data, but differ in provenance, generation context, vulnerability coverage, and labeling methodology.
- Table 9 compares representative datasets across provenance, code-generation context, vulnerability coverage, and labeling methodology.
3 DATASET CURATION
LLMVul mines AI-attributed C/C++ commits, extracts and deduplicates function bodies, and labels them with a validated three-tool vulnerability-analysis ensemble. The resulting dataset preserves extensive repository, provenance, attribution, and vulnerability metadata.
- The curation pipeline mines LLM-attributed commits, extracts function-level code units, and applies a three-tool static-analysis ensemble validated by human raters.
- 3.1 Phase 1: Repository Mining and AI Attribution: 7,018 LLM-attributed commits were identified across 226 repositories after scanning 1,200 repositories and 321k commits.
- 3.1 Phase 1: Repository Mining and AI Attribution: 89.2% of attributed functions had strong machine-generated co-authorship signals, while 10.8% had medium developer-declared signals.
- 3.2 Phase 2: Function Extraction: Functions were extracted from added or modified diff lines with tree-sitter, retaining C/C++ definitions between five and 150 lines and storing source and commit provenance.
- 3.2 Phase 2: Function Extraction: 21,430 unique functions remained after removing duplicate bodies from 29,062 extracted functions.
- 3.3 Phase 3: Vulnerability Labeling: The ensemble labels a function vulnerable when at least two of Semgrep, Flawfinder, and pattern matching flag it; one-tool cases are retained as uncertain.
- 3.4 Data Quality and Reproducibility: The dataset records 34 features per function and supports temporal, provenance, security, and vulnerability-category analyses across the mined corpus.
4 POSSIBLE RESEARCH QUESTIONS
LLMVul is designed to support empirical study of vulnerability detection, security characteristics, dataset utility, generalization, provenance, and evolution in LLM-generated C/C++ code.
- LLMVul supports empirical studies of the security, generalization, provenance, and evolution of LLM-generated C/C++ code.
- Detection & Generalization: The proposed detection questions test whether models trained on human-written code generalize to LLM-generated functions and whether LLMVul-trained models perform better on that population.
- Security Characterization: The security-characterization questions examine CWE distributions and changes in vulnerable-function proportions across AI tools and time.
- Dataset Utility & Analysis: The dataset-utility questions assess whether attribution metadata distinguishes AI-generated from human-written code and compare the ensemble components’ effectiveness.
- Repository provenance, vulnerability labels, CWE information, attribution signals, and temporal metadata enable systematic comparisons with human-written code and tests of security-analysis generalization.
5 THREATS TO VALIDITY
LLMVul’s interpretation is constrained by both its labeling procedure and its repository sampling frame. Static-analysis uncertainty, explicit-metadata dependence, and emphasis on highly starred C/C++ projects limit how broadly its findings should be generalized.
- Labeling limitations: Three-tool static-analysis labels may contain false positives or false negatives on isolated function fragments, despite validation on 100 functions yielding κ=0.79.CodeQL was excluded because its C/C++ extractor requires compilable translation units, which do not match the function-level representation.
- Provenance limitations: Explicit Git metadata signals may miss AI-assisted functions when developers do not record AI usage, underestimating AI-generated-code prevalence.Substantial post-generation editing may also make generated code harder to distinguish from human-written code.
- Sampling limitations: Sampling the top 1,200 C/C++ repositories by star count favors mature, widely adopted projects over smaller, private, less-established, or embedded codebases.The resulting dataset may therefore not fully represent those under-sampled project types.
- Language scope: Whether LLMVul’s observed vulnerability patterns generalize beyond C/C++ to languages such as Python and JavaScript remains an open question.
6 CONCLUSION & FUTURE WORK
LLMVul provides vulnerability-labeled C/C++ functions from authentic repositories containing AI-assisted code, with vulnerability categories, human-validated labels, and provenance metadata. Future work will extend the dataset across languages and improve build and analyzer support.
- Conclusion: LLMVul captures incidental vulnerabilities in real-world AI-assisted C/C++ development and pairs labels with provenance and AI-tool metadata for systematic analysis.
- Future work: Future work will expand LLMVul to Python, Java, and Rust and develop build infrastructure supporting additional analyzers and compilable code contexts.These extensions target cross-language vulnerability analysis and improved memory-safety characterization.