Source-linked AI summary
LLMVul: A Vulnerability-Labeled Dataset of LLM-Generated C/C++ Functions from Real Production Repositories
Mohammad Farhad, Shuvalaxmi Dass
TL;DR
Security research lacks vulnerability-labeled evidence from LLM-generated code committed in real production projects. LLMVul mines and labels such C/C++ functions from AI-assisted GitHub development, producing 21,430 unique functions from 226 repositories and 1,540 ensemble-vulnerable functions across 17 CWE categories. The dataset preserves vulnerability, provenance, and AI-tool metadata for reproducible security and generalization studies.
Problem
Existing vulnerability datasets focus mainly on human-written code or controlled prompting, limiting study of LLM-generated code as it appears in production repositories.
Source: §1 · p. 1Method
LLMVul mines AI-attributed C/C++ commits, extracts and deduplicates functions, and labels them with a three-tool static-analysis ensemble validated by human raters.
Source: §1 · p. 2, §3.1 · p. 2, §3.2 · p. 3Results
21,430 unique C/C++ functions from 226 repositories include 1,540 ensemble-vulnerable functions spanning 17 CWE categories.
Source: §1 · p. 2Takeaways & Limitations
Repository provenance, vulnerability labels, CWE identifiers, AI-tool attribution, and temporal metadata support reproducible research on LLM-generated code security and generalization.
Source: §1 · p. 2, §4 · p. 5AI attribution depends on explicit Git metadata, and sampling the top 1,200 repositories may underrepresent smaller, private, less-established, or embedded codebases.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used to generate and assist with software development, yet existing vulnerability datasets largely focus on human-written code or controlled prompting environments. This limits the ability to study security weaknesses in LLM-generated code as it appears in real-world software projects. We present LLMVul, a vulnerability-labeled dataset of LLM-generated C/C++ functions mined from real production repositories. We mine AI-assisted development activity from GitHub over a 4 year period, from November 13, 2022 to September 3, 2026, using provenance signals such as commit metadata and AI-related authorship evidence. After filtering and deduplication, LLMVul contains 21,430 unique C/C++ functions from 226 repositories, together with repository, commit, function, provenance, and AI-tool metadata. We establish vulnerability labels using an ensemble of complementary static-analysis and pattern-based techniques and assign Common Weakness Enumeration (CWE) categories to confirmed vulnerable functions. To assess labeling reliability, we additionally conduct independent manual annotation and measure inter-rater agreement using Cohen's kappa ($k=0.79$). LLMVul contains 1,540 ensemble-vulnerable functions spanning 17 unique CWE categories, providing substantially more real-world LLM-generated vulnerable C/C++ functions than existing vulnerability-oriented LLM code benchmarks. By preserving both code-level vulnerability labels and generation/provenance metadata, LLMVul enables reproducible research on vulnerability detection, security evaluation of LLM-generated code, and analysis of vulnerability patterns in AI-assisted software development. The LLMVul dataset is publicly available at https://doi.org/10.5281/zenodo.22668216.
1 INTRODUCTION
LLM coding assistants are increasingly used in production, but security research lacks vulnerability-labeled data from code developers actually commit. LLMVul addresses this gap with labeled C/C++ functions mined from AI-assisted GitHub development.
- LLM-generated code is increasingly used in production, including security-sensitive systems software, embedded firmware, and network services.Source: §1 · p. 1
The dataset gap
- Existing security benchmarks mainly use controlled prompts and therefore do not capture vulnerabilities in developer-committed production code.Source: §1 · p. 1
- Prior real-world studies identify AI-attributed commits but do not provide function-level vulnerability labels, while established C/C++ vulnerability datasets contain human-authored code.Source: §1 · pp. 1–2
Our work
- LLMVul contains 21.4k unique C/C++ functions from 226 repositories and 1,684 LLM-attributed commits, including 1,540 vulnerable functions across 17 CWE categories.Source: §1 · p. 2
- The dataset combines provenance and AI-tool metadata with ensemble-generated labels validated by human raters, achieving substantial agreement (𝜅= 0.79).Source: §1 · p. 2
2 RELATED WORK
Existing AI-code security datasets span human-written benchmarks, prompted code, and repository-mined data, but differ in provenance, generation context, vulnerability coverage, and labeling methodology.
- Table 9 compares representative datasets across provenance, code-generation context, vulnerability coverage, and labeling methodology.Source: §2 · p. 2
3 DATASET CURATION
LLMVul mines AI-attributed C/C++ commits, extracts and deduplicates function bodies, and labels them with a validated three-tool vulnerability-analysis ensemble. The resulting dataset preserves extensive repository, provenance, attribution, and vulnerability metadata.
- The curation pipeline mines LLM-attributed commits, extracts function-level code units, and applies a three-tool static-analysis ensemble validated by human raters.Source: §3 · p. 2, §3.3 · p. 3
3.1 Phase 1: Repository Mining and AI Attribution
- 7,018 LLM-attributed commits were identified across 226 repositories after scanning 1,200 repositories and 321k commits.Source: §3.1 · p. 2
- 89.2% of attributed functions had strong machine-generated co-authorship signals, while 10.8% had medium developer-declared signals.Source: §3.1 · p. 2
3.2 Phase 2: Function Extraction
- Functions were extracted from added or modified diff lines with tree-sitter, retaining C/C++ definitions between five and 150 lines and storing source and commit provenance.Source: §3.2 · p. 3
- 21,430 unique functions remained after removing duplicate bodies from 29,062 extracted functions.Source: §3.2 · p. 3, §3.4 · p. 4
3.3 Phase 3: Vulnerability Labeling
- The ensemble labels a function vulnerable when at least two of Semgrep, Flawfinder, and pattern matching flag it; one-tool cases are retained as uncertain.Source: §3.3 · p. 3
3.4 Data Quality and Reproducibility
- The dataset records 34 features per function and supports temporal, provenance, security, and vulnerability-category analyses across the mined corpus.Source: Table 4 · p. 4, §3.4 · p. 5
4 POSSIBLE RESEARCH QUESTIONS
LLMVul is designed to support empirical study of vulnerability detection, security characteristics, dataset utility, generalization, provenance, and evolution in LLM-generated C/C++ code.
- LLMVul supports empirical studies of the security, generalization, provenance, and evolution of LLM-generated C/C++ code.Source: §4 · p. 5
Detection & Generalization
- The proposed detection questions test whether models trained on human-written code generalize to LLM-generated functions and whether LLMVul-trained models perform better on that population.Source: §4 · p. 5
Security Characterization
- The security-characterization questions examine CWE distributions and changes in vulnerable-function proportions across AI tools and time.Source: §4 · p. 5
Dataset Utility & Analysis
- The dataset-utility questions assess whether attribution metadata distinguishes AI-generated from human-written code and compare the ensemble components’ effectiveness.Source: §4 · p. 5
- Repository provenance, vulnerability labels, CWE information, attribution signals, and temporal metadata enable systematic comparisons with human-written code and tests of security-analysis generalization.Source: §4 · p. 5
5 THREATS TO VALIDITY
LLMVul’s interpretation is constrained by both its labeling procedure and its repository sampling frame. Static-analysis uncertainty, explicit-metadata dependence, and emphasis on highly starred C/C++ projects limit how broadly its findings should be generalized.
Labeling limitations
- Three-tool static-analysis labels may contain false positives or false negatives on isolated function fragments, despite validation on 100 functions yielding κ=0.79.CodeQL was excluded because its C/C++ extractor requires compilable translation units, which do not match the function-level representation.Source: §5 · p. 5
Provenance limitations
- Explicit Git metadata signals may miss AI-assisted functions when developers do not record AI usage, underestimating AI-generated-code prevalence.Substantial post-generation editing may also make generated code harder to distinguish from human-written code.Source: §5 · p. 5
Sampling limitations
- Sampling the top 1,200 C/C++ repositories by star count favors mature, widely adopted projects over smaller, private, less-established, or embedded codebases.The resulting dataset may therefore not fully represent those under-sampled project types.Source: §5 · p. 5
Language scope
- Whether LLMVul’s observed vulnerability patterns generalize beyond C/C++ to languages such as Python and JavaScript remains an open question.Source: §5 · p. 6
6 CONCLUSION & FUTURE WORK
LLMVul provides vulnerability-labeled C/C++ functions from authentic repositories containing AI-assisted code, with vulnerability categories, human-validated labels, and provenance metadata. Future work will extend the dataset across languages and improve build and analyzer support.
Conclusion
- LLMVul captures incidental vulnerabilities in real-world AI-assisted C/C++ development and pairs labels with provenance and AI-tool metadata for systematic analysis.Source: §6 · p. 6
Future work
- Future work will expand LLMVul to Python, Java, and Rust and develop build infrastructure supporting additional analyzers and compilable code contexts.These extensions target cross-language vulnerability analysis and improved memory-safety characterization.Source: §6 · p. 6