The Data Problem in Software Vulnerability Analysis: Artifacts, Quality, and Consumption

Motivation

• Reported corpus failures include over 50% accuracy loss on real-world code, up to 68% near-duplicate train/test pairs, below-60% label accuracy, and roughly one in six patch-derived CVEs not recoverable from static code alone [45].Source: §1 · p. 2

The evidence ladder

• The evidence ladder treats labels as claims, patches as narrowing a flaw's location, and runnable tests or PoCs as reachability demonstrations; reasoning explains without extending that order, while traces are observational records outside it.Source: §1 · p. 2

Approach

• The study examines artifacts, quality, and consumption across code, metadata, patches, tests and exploits, reasoning, and traces, using a 1,522-paper corpus and 111 anchor papers.Source: §1 · p. 3

Findings

• Executable artifacts alone let successful runs directly evidence flaw reachability: 16/24 are authentically real and 23/24 have independently checked labels; of 41 deeply coded code-sample datasets, one uses authentically real-world data.Source: §1 · p. 3
• Leakage is unaddressed by 49/90 datasets where applicable, more than a quarter make no availability statement, and executable artifacts tie patches for lowest public availability at 12/24.Source: §1 · pp. 3–4
• Reasoning data appears only since 2024 and is mostly model-generated; traces have only three deep-coded contributions, and both artifact types remain among the least represented and directly studied.No quality attribute has been the primary subject of more than fourteen studies.Source: §1 · pp. 3–4

• The taxonomy separates what vulnerability data is, how good it is, and what it is used for, with item-level definitions for consistent, auditable coding.Source: §1 · p. 4
• The study deep-codes 111 anchor papers, backs affirmative rubric grades with verbatim spans, checks absences with targeted full-text scans, and provides an end-to-end reproducible pipeline except for inspectable but non-re-executable historical LLM screening.Source: §1 · p. 4
• The quality synthesis distinguishes how much each attribute is studied from how well datasets achieve it, showing that the six attributes constrain one another rather than varying freely.Source: §1 · p. 4
• An artifact-by-quality matrix shows where specific quality weaknesses occur across artifact types.Source: §1 · p. 4
• The study identifies seven structural limitations of existing data, including three unsupported failure modes, and proposes an agenda for evidence-rich vulnerability data.Source: §1 · p. 4

2.1 Vulnerability Analysis and Its Data

• Artificial vulnerabilities were easier to detect than real ones: benchmark rankings inverted, and per-category recall ranged from 0% to 100%, undermining headline accuracy.Synthetic suites offered labels reliable by construction, but less realistic data.Source: §2.1 · p. 4
• Commit-mined labels enabled larger supervised datasets but could mark every function touched by a fix as vulnerable; realistic evaluations then cut detector performance by more than half [11],.The commit-based approach turns expert judgment into a data-engineering problem, but not every changed function is necessarily vulnerable.Source: §2.1 · pp. 4–5
• LLM-era datasets add AI-generated code whose security must be judged and executable benchmarks of agent actions; undisclosed training corpora also make test-set contamination hard to measure [139], [34] [47] [160], [38].Source: §2.1 · p. 5

2.2 What Counts as Evidence: A Running Example

• Evidence ranges from cheaper, weaker artifacts to executable proofs: after processing roughly 2,500 sources and 448 CVEs, one benchmark retained only 23 functioning exploits, while 78.9% of CVEs lack public PoCs [157], [31].An exploit that runs is both the artifact and independently checkable evidence that its triggered behavior is reachable [184].Source: §2.2 · p. 5
• Executable artifacts (T4) are the only well-populated type whose majorities satisfy both realism and label evidence, yet they are among the least often shared.A run directly checks triggered behavior, while the realism result reflects authentic targets with deployment context retained.Source: §2.2 · p. 6

2.3 Related studies

• Unlike prior dataset reviews, this study spans six artifact types and six quality attributes, uses a fixed schema with verbatim support for affirmative grades, and reports study coverage separately from dataset achievement.Its scope runs from metadata through execution traces and organizes evidence by what each artifact establishes.Source: §2.3 · pp. 6–7
• Unlike reviews focused on general code quality or one malicious-code subdomain, this study examines vulnerability data across detection, patching, validation, and agent behavior [55], [176].It identifies a gap in reviews spanning the full artifact range and separately reporting how much each attribute is studied and how well datasets achieve it.Source: §2.3 · p. 7

3 Methodology

• The study uses a documented search, screening, two-tier reading, and evidence-backed coding pipeline; LLM-assisted exclusions were rechecked by authors.Harvesting, coding, and analysis re-execute from released scripts, although the LLM-assisted screening stage does not; its excluded records were re-screened by three authors.Source: §3 · p. 7

3.1 Literature Search

• The three-phase search combined topic queries, canonical dataset seeds and citation neighborhoods, and enrichment/backfilling, yielding 2,112 records after deduplication.Named-artifact seeding corrected an early bias toward post-2021 work.Source: §3.1 · p. 8

3.2 Screening: a data-centric filter

• Screening included papers that contribute vulnerability data or assess its quality, while excluding technique-only studies that merely consume existing datasets.The included corpus contains 813 preprints among 1,522 papers; the authors caution that venue skew cannot be cleanly separated from search bias and draw no conclusion from it.Source: §3.2 · pp. 8–9

3.3 Two-tier reading protocol

• The two tiers assign full-text quality analysis to 111 deep-coded papers and use title, abstract, and metadata to map attention across the remaining corpus.Consumption-task and technique judgments rely on reading, so the paper characterizes them for the anchor set rather than projecting them onto the full corpus.Source: §3.3 · p. 9

3.4 Deep-coding scheme

• Tier-1 coding records artifact, provenance, quality, consumption, scope, data-gap, and verbatim-evidence fields using closed enumerations for graded judgments.The scheme separately records which quality attributes a paper studies and how well datasets achieve them; study papers are scored on examined data, with specified attributes left not-assessed.Source: §3.4 · pp. 9–10
• Before large-scale coding, coders independently coded a shared paper set and compared results; ambiguities about study papers and context scope prompted added rules.Enumerated values were also extended twice when observed corpus values lacked faithful codes.Source: §3.4 · p. 10
• All 111 LLM-drafted records were author-verified; 80 received field-by-field source checks, while 31 were conflict-screened and flagged issues adjudicated by three authors.The conflict screen compared records against the codebook, their evidence spans, and an independent earlier coding.Source: §3.4 · p. 10
• Every affirmative rubric value required a verbatim supporting span, absence values were checked with targeted full-text scans, and numeric values were mechanically checked against sources.All 111 records carried 729 evidence spans in total.Source: §3.4 · p. 10

4 Taxonomy of Vulnerability Data

• The taxonomy separates artifact form, quality, and use, assigning each attribute to exactly one aspect and using operational closed vocabularies for graded attributes.Where a property resists grading, the authors record a reported figure instead.Source: §4 · p. 10
• The taxonomy folds vulnerability-production method into realism, keeps code authorship as provenance, and treats the evidence ladder and reproducibility as cross-cutting lenses rather than separate axes.Evaluation metric is excluded as a property of studies rather than data.Source: §4 · p. 10
• Its security-data focus spans mining, labeling, validation, and agent behavior, organizing analysis around the distinction between artifacts that assert a vulnerability and those that demonstrate it.This contrasts with general code-quality and data-quality frameworks, which do not make that artifact distinction.Source: §4 · p. 11

4.1 Derivation

• The authors derived the taxonomy in two corpus passes, first identifying candidate attributes and values, then generalizing them into a reusable set while keeping form, quality, and use separate.The second pass rejected evaluation metric, how-produced, and reproducibility as independent data attributes.Source: §4.1 · p. 11
• Coding added tool-derived labels and contamination-controlled as vocabulary values for recurring cases the original schema did not capture.Tool-derived labels lack human or execution checks; contamination-controlled datasets guard against pretraining exposure by construction, even without a train/test split.Source: §4.1 · p. 11

4.2 Aspect A: form, and why the types are ordered

• The six artifact types are ordered by evidence strength: metadata and code samples assert, patches localize a suspected flaw, and executable artifacts demonstrate reachable behavior.Reasoning adds explanation without strengthening demonstration, while traces observationally record what happened when a trigger ran and therefore sit apart from the demonstrative ordering.Source: §4.2 · p. 11

4.3 Aspect B: quality-item definitions

• Quality definitions distinguish realism and label evidence by evidential basis, while scale and diversity are recorded as reported figures rather than thresholded judgments.Verified labels received an independent execution or human-review check, but that does not certify the assigned CWE, root cause, or absence of other flaws.Source: §4.3 · pp. 12–13
• Leakage coding distinguishes split strategies and contamination controls from “not addressed” and “n/a,” while availability separates public, on-request, unavailable, and unreported datasets.“Not reported” was five times more common than an explicit refusal; it was not the most common value because most coded datasets were public.Source: §4.3 · p. 13

4.4 Aspect C: consumption

• Consuming task and technique are multi-valued and coded from what papers evaluate; technique categories are paradigms rather than model architectures.The categories track the kind of consumer a dataset was built for, a distinction the study says has shifted across three eras.Source: §4.4 · p. 13

4.5 Coding rules

• The coding rules assign one primary artifact type, use a benchmark’s novel contribution to break code-and-executable ties, and distinguish triggering inputs from execution records.Studies spanning many datasets are not assessed for realism, label evidence, or leakage because no single value is meaningful across them.Source: §4.5 · pp. 13–14

4.6 What the taxonomy does not capture

• The taxonomy is a measurement instrument rather than a causal model, and its single primary type can under-represent datasets that bundle several artifact types.Realism is composite and empirically coupled with label evidence, so realism figures should be read alongside their component attributes.Source: §4.6 · p. 14
• Five datasets using explanation as a consuming task were added after a corpus-wide screen for reasoning data, showing that an empty taxonomy cell can reflect search limits.The authors also note that provenance n/a and unlabeled labels were defined but never instantiated in the deep-coded set.Source: §4.6 · p. 14

5.1 Distribution by year and artifact type

• 429 papers were included in 2025, and 554 in 2026, the latter a partial-year lower bound.The corpus grew from 22, 35, and 68 papers in 2020–2022 to 117 in 2023 and 255 in 2024; the 2026 total covers a partial year. The paper therefore avoids inferring artifact-type trends from keyword tags whose recall varies over time.Source: §5.1 · p. 14, Figure 2 · p. 15
• 753 papers focus on code samples, the largest primary artifact type in the corpus, followed by 376 on tests/PoCs/exploits.The counts cover papers associated with artifact types, not datasets produced; primary types cover all 1,522 included papers.Source: §5.1 · p. 15

5.3 Study result: data artifacts (RQ1)

• 8 deep-coded reasoning datasets all appeared in 2024 or later, while the study found 3 deep-coded trace corpora.Corpus-wide, 94 papers carried a trace artifact versus 24 carrying reasoning data, but 82 of those 94 trace papers appeared in 2026; paper counts are not released-dataset counts, and reusable, well-documented data remains scarce.Source: §5.3 · p. 16
• 50 of 111 deep-coded papers use function granularity, the most common level, followed by 15 at repository granularity.Patches and diffs are coded at slice granularity.Source: §5.3 · p. 16

5.3.1 Metadata (T2).

• Only 5 deep-coded papers study metadata directly, and none of the 3 contributing metadata artifacts reaches real-world realism or verified labels.Metadata is widely inherited as labels attached to code, but receives little direct study.Source: §5.3.1 · p. 16
• Up to 20.7% F1 was lost to label noise in the worst case, while denoising recovered 10.4% on average.The reported errors arose from static-analysis false positives and tangled commits, which can distort measured performance.Source: §5.3.1 · p. 16

5.3.2 Code Samples (T1).

• 1 of 41 code-sample datasets has full-context, real-world vulnerabilities; the exception uses whole contracts, while other datasets slice functions or files.Smart-LLaMA-DPO’s Solidity contracts are at module granularity, so the deployed unit is the sample; function and file slices lose context.Source: §5.3.2 · p. 17
• 88% injection success on real-world code is reached by LLM prompting, but success varies sharply by CWE and over 65% of vulnerabilities in the largest real-world corpora require multi-line edits.Generators infer realism from improved detector performance rather than demonstrating exploitability; execution validation and context-preserving mining would improve label evidence and realism for different reasons.Source: §5.3.2 · p. 17

5.3.3 Reasoning (T5).

• Eight reasoning datasets are all from 2024 or later; seven are mined, semi-real, or synthetic, and only VADER reaches real-world realism with 174 curated cases [82] [83] [94] [164] [180].Reasoning is usually layered over existing code, so it inherits that code's realism.Source: §5.3.3 · p. 18
• Human ratings of generated explanations were consistently below LLM-assigned ratings in one study using eight domain PhD students.Across reasoning datasets, larger scale coincides with less direct human authorship: examples range from 174 expert-written cases to 53,693 model-generated explanations [94] [164] [180].Source: §5.3.3 · p. 18
• Reasoning datasets address leakage wherever it applies and seven of eight are public, but only one uses authentically real-world code and none checks reasoning ground truth with human review at scale.At scale, explanations are model-generated and checked by another model; the one expert-authored corpus stops at 174 cases.Source: §5.3.3 · p. 18

5.3.4 Patches and Fixes (T3).

• 5 of 14 patch datasets reach real-world realism and 7 of 14 have verified labels, below executable artifacts but above code samples on both measures.Patch-derived labels can overgeneralize: CVEfixes labels every function changed in a commit as vulnerable, a practice its authors acknowledge is “always not true” [11].Source: §5.3.4 · p. 18
• 4.3–6.0% of functionally correct patches were vulnerable without an attacker, rising to 40.7% when a developer-style suggestion was added.Passing tests alone is not a security oracle.Source: §5.3.4 · p. 19

5.3.5 Tests, PoCs, and Exploits (T4).

• 16 of 24 executable-artifact datasets meet the realism criterion and 23 of 24 meet the label-evidence criterion, the only major artifact type with majorities meeting both.Running an exploit makes triggered behavior directly observable, providing independently checkable evidence in the tested environment.Source: §5.3.5 · p. 19
• A prior study found no publicly available proof-of-concept, and 70.2% of CVE patches contained no test files [149], limiting the availability of executable data for automated testing.Across 164 projects, one effort yielded 23 Python CVEs with working PoCs amenable to automated testing—about 5% [157].Source: §5.3.5 · pp. 19–20
• K-REPRO reproduces over half of 100 Linux-kernel N-days, PBFuzz triggers 57 of 129 Magma CVEs, and ChainDelta generates exploits for 16 of 25 Ethereum vulnerabilities, yet hard cases remain.PBFuzz leaves 72 CVEs untriggered, including 17 unreached by prior approaches; ChainDelta finds four undisclosed vulnerabilities, while nine patched vulnerabilities resist it.Source: §5.3.5 · p. 20
• Execution-driven tests provide an objective oracle but cannot establish vulnerability absence, and executable datasets have the lowest major-type availability rate: 12 of 24 are public.Four corpus contamination-controlled values are T4 benchmarks, where memorization can be measured; static analysis can yield false positives without PoCs, while fuzzing can yield false negatives.Source: §5.3.5 · p. 20

5.3.6 Traces and Logs (T6).

• Three primary trace corpora survive full-text assessment, all from 2025 or later; screening-level counts overstate reusable trace releases.Of 94 trace-tagged papers, six initially appeared to release primary trace corpora, but full-text checks confirmed one additional primary corpus and identified other releases as benchmarks, venues, or protocols whose runs emit traces.Source: §5.3.6 · pp. 20–21
• 21% of agent trajectories contained insecure actions during ordinary operation across 12,000-plus actions and 93 real-world setup tasks, exposing risks invisible to code-centric datasets.Traces are cheap to produce but expensive to label: the first corpus required hand-labeling 500 steps, while a later released session corpus relies on model judges for security annotations.Source: §5.3.6 · p. 21

5.4 Study result: data quality (RQ2)—the core

• 49 of 90 applicable datasets leave leakage unaddressed, and no quality attribute receives findings in more than 14 papers among the 111 anchors.Label evidence is studied in 14 papers, leakage and realism in 8 each, diversity in 7, availability in 6, and scale in none; achievable scale appears constrained by labeling cost and coverage.Source: §5.4 · pp. 21–22

5.4.1 Realism.

• 20 of 41 code-sample datasets draw vulnerabilities from real projects or CVEs, but only 3 retain deployment-unit context and 2 do both; 15 executable datasets meet both realism and label-check criteria.Overall, 16 of 24 executable datasets are graded real-world and 23 of 24 have independently checked labels; just one code-sample dataset receives the stricter real-world grade.Source: §5.4.1 · p. 22
• 79.40% accuracy on synthetic code versus 10.03% on real-world code; patch-identification models lose up to 90% F1 in the wild [62].ReVeal’s greater-than-50% drop is attributed to context and distribution together, while Big-Vul’s authentic CVE provenance does not prevent context and distribution failures.Source: §5.4.1 · p. 22
• 29.5% of Python and 24.2% of JavaScript AI-generated snippets already merged into GitHub carry weaknesses spanning 43 CWE categories.The passage notes that much security evidence for generated code comes from controlled prompt scenarios rather than conditions resembling use.Source: §5.4.1 · p. 23

5.4.2 Label evidence.

• 51 of 88 engaged datasets are verified, but verification is far more common for executable artifacts (23 of 24) than code samples (14 of 36).Mined code and patch datasets most often use patch-derived labels (24), followed by tool-derived (11) and description-derived (2) labels; expert review is stronger but costly, while patch and scanner labels inherit errors.Source: §5.4.2 · p. 23
• Label errors cost up to 20.7% F1, while denoising recovers 10.4% on average; Mono finds 16.7% of evaluated CVEs undecidable from supplied static context [45].Mono also corrects ordinary mislabeling at a rate of 31.0%; unlike noisy labels, undecidable cases cannot be reliably resolved from the available static-code context alone.Source: §5.4.2 · p. 23

5.4.3 Scale.

• 686k VinJ samples are only 69% vulnerable, whereas VulnRepairEval retained 23 working-exploit instances after processing approximately 2,500 sources and 448 CVEs [157].Dataset size broadly follows label-oracle cost, so scale alone does not indicate label quality.Source: §5.4.3 · p. 24
• Augmentation gains plateaued past roughly 5k generated samples, after which additional synthetic data stopped helping and could reverse gains.The passage generalizes the mechanism—not the threshold—to data drawn from the same distribution under the same oracle, which may add little new information.Source: §5.4.3 · p. 24

5.4.4 Diversity.

• 45 of 93 contributed datasets are single-language and only 7 cover three or more; C/C++ appears in 27, the largest language group.Twenty-one datasets cover only C/C++, and the subsection characterizes coverage as a collection of single-language silos.Source: §5.4.4 · p. 24
• PrimeVul’s mined corpus has roughly 1 vulnerable function to 33 benign, illustrating class imbalance that is rarely reported.Seven papers study diversity, mostly as a secondary finding of realism or leakage studies.Source: §5.4.4 · p. 24
• Balanced or artificially generated datasets produced significantly overrated performance, while long-tailed CWE coverage and imbalance shape which weaknesses datasets represent.PrimeVul argues balanced classes misrepresent deployment base rates by more than an order of magnitude; PatchDB synthesizes control-flow variants to populate its long-tailed NVD-derived component.Source: §5.4.4 · p. 24

5.4.5 Leakage and contamination.

• 49 of 90 applicable datasets do not address leakage; counting random splits without deduplication, 54 of 90 report no effective leakage guard.A random split without deduplication is the highest-risk configuration because it neither detects nor separates duplicates.Source: §5.4.5 · p. 25
• Leakage studies report up to 68% train-test duplication for ReVeal and show that LLMs can collapse on post-cutoff zero-day sets [16, 116].Copilot reproduces original vulnerable code roughly 33% of the time, more often for older vulnerabilities; executable benchmarks increasingly use construction controls such as proprietary cases, mutation, and exclusion of public ready-to-run proofs.Source: §5.4.5 · pp. 25–26

5.4.6 Availability.

• Availability is unstated for 25 of 93 datasets; 62 are public and 5 explicitly withheld.Public availability is 12 of 24 executable-artifact datasets and 7 of 14 patch datasets, compared with 33 of 41 code-sample datasets.Source: §5.4.6 · p. 26
• Availability limits reproducibility as well as release: only 25.5% of reviewed deep-learning vulnerability detectors provided a public tool.Among reproducible tools, 14.3% were partially and 28.6% fully replicable on another real-world dataset; many vulnerabilities lack public PoCs and patches lack tests demonstrating the vulnerability [31] [149].Source: §5.4.6 · p. 26

5.4.7 The attributes are not independent.

• Dataset attributes form dependencies: affordable labeling constrains scale, while automatic-labeling granularity costs context and therefore realism.Labeling and context constraints also appear to restrict CWE diversity; leakage is more tractable because ordinary train-test protection largely depends on partitioning data already held, though pretraining contamination can require construction controls or audits.Source: §5.4.7 · pp. 26–27

5.5 Study result: consumption and evaluation (RQ3)

• Detection (60 datasets) and behavior/safety vetting (27) lead the anchor set’s tasks, followed by repair (25), validation (12), classification (11), localization (8), and explanation (5).Most datasets serve one task: 51 are single-task, compared with 27 tagged with two and 7 with three; these hand-coded results describe the purposive 111-paper anchor set, not literature-wide shares.Source: §5.5 · p. 27
• Validation is supported entirely by executable artifacts—12 datasets, 10 real-world—whereas localization, classification, and explanation have no executable backing.Detection draws most records from code samples and only 25% from executable artifacts; each of the three tasks without executable backing has only one or two real-world datasets.Source: §5.5 · p. 27
• Line-level localization depends on labels that are least verifiable: patch-derived labels show which lines a commit touched, not which lines were vulnerable.The paper identifies this as the tangled-commit problem and notes that such labels underpin training for line-level localization.Source: §5.5 · p. 27
• Prompting rises from 2 of 17 pre-2023 records to 37 of 67 in 2025–26, while agentic consumption grows from none to 21; deep learning falls from 10 of 17 to 11 of 67.Traditional program analysis remains broadly stable, from 4 of 17 to 13 of 67, and increasingly serves as an execution-based oracle within LLM-era systems.Source: §5.5 · p. 27
• Of 41 code-sample datasets, 25 are consumed through deep learning or fine-tuning and one by an agentic pipeline; none of 24 executable datasets is used for training.Executable datasets are instead used as benchmarks (17) and agentically (16); patch datasets straddle the boundary, with 7 trained on and 3 consumed agentically.Source: §5.5 · p. 28

5.6 Synthesis: the status quo at a glance

• Only executable artifacts are majority-realistic and independently checked (16/24 and 23/24), while code samples are weakest on realism (one real-world instance in 41) and partially verified (14/36).The matrix also reports that 49 of 90 datasets where leakage applies do not address it, and 67% are publicly available.Source: §5.6 · p. 29
• Traces remain limited to three deep-coded datasets, reasoning has grown to eight since 2024, and no quality attribute has been the primary object of more than fourteen papers.The authors identify reusable explanatory and observational artifacts, and direct study of vulnerability-data quality, as gaps.Source: §5.6 · p. 29

5.7 Trajectory: is the picture improving?

• Aggregate verified-label share rises over time, but code-sample verification declines from 3 of 8 through 2023 to 2 of 7 in 2026 as executable datasets grow from 25% to 37% of new datasets.Within artifact types, executable verification is near-total throughout; the anchor set is purposively selected, so the trend is not a population estimate.Source: §5.7 · p. 29, Figure 5 · p. 30
• Code-sample realism remains unchanged across the three anchor cohorts: exactly one dataset in each reaches the real-world grade.Effective leakage handling rises and then falls, public availability does not recover its earliest level, and the authors treat the trajectory as indicative because the anchor set is purposively selected.Source: §5.7 · p. 29

T2 Metadata

• The passage lists Nie et al. in 2023 with the sequence func MS Pat Ded Pub B.Source: T2 Metadata · p. 31
• Zhao et al. (2025) appears in the supplied passage with the entry func SY Desc none – D, retaining the listed wording and dash before D.Source: T2 Metadata · p. 31
• The cited entry lists ReVul-CoT [25] in 2025 and gives the associated labels in this order: MS, Desc, Prj, Pub, and D.Source: T2 Metadata · p. 31
• The supplied listing records Young et al. [176] in 2026 with the sequence – – – none Req S, preserving the entry’s three dashes and final terms.Source: T2 Metadata · p. 31
• The supplied bibliography entry identifies He et al. (Bridging) [55] as a 2026 work and, after the year, lists three dashes followed by Ded, Pub, and S.Source: T2 Metadata · p. 31

T1 Code samples

• The cited entries report sample granularity at the function level for Devign, file level for Nong et al. (Preliminary), statement level for D2A, and module level for VulnLLMEval.Source: T1 Code samples · p. 31
• The inventory spans 2019–2026, with Devign at its earliest end and 2026 datasets including Kharma et al. [70] and Jahromi et al. [63].The table’s columns identify the recorded dimensions as Gran, Real, Label, Leak, Avail, and Kind.Source: T1 Code samples · pp. 31–32
• Individual entries pair a granularity with coded values for realism, labels, leakage, availability, and kind; for example, Devign is listed as func / MS / Ver / Rnd / – / D.The header names the six fields, while the table does not expand the abbreviations in these passages.Source: T1 Code samples · p. 31

T5 Reasoning

• Smart-LLaMA (2024) and Yu et al. (Secure) [180] (2025) are mod/MS/Ver/Ded entries; both are public, while Yu’s availability is unreported.MS means mined/semi-real, Ver verified, and Ded deduplicated; Smart-LLaMA is kind both, while Yu et al. is kind dataset.Source: T5 Reasoning · p. 32, Table 9 · p. 33
• Lin et al. [83] (2025) is func/MS/Pat/Ded/Pub/D, whereas Vul-R2 (2025) is func/MS/Ver/Chr/Pub/B.Pat means patch-derived, Chr chronological, D dataset, and B both; both entries are mined/semi-real and publicly available.Source: T5 Reasoning · p. 32, Table 9 · p. 33
• VADER (2025) is the sole listed real-world entry, with verified labels, leakage marked none, public availability, and kind both.The table defines RW as real-world and none as leakage not addressed.Source: T5 Reasoning · p. 32, Table 9 · p. 33

T3 Patches & fixes

• CVEfixes [11], PatchDB, and VulRepair are listed for 2021–2022 at statement or function level.Their rows also show the tags MS, Pat or Ver, and D.Source: T3 Patches & fixes · p. 33
• The 2024 entries include function-level APPATCH [115] and statement-level Le et al. [75].Their rows list RW and MS, respectively, alongside Ver for both.Source: T3 Patches & fixes · p. 33
• The 2025 entries include Peng et al. at repository level and SecureFixAgent [43] and Mono [45] at statement level.Yuan et al. is listed at file level; the rows carry different abbreviated tags.Source: T3 Patches & fixes · p. 33
• For 2026, Irsan et al. [62] is listed at statement level and SPVR at function level.Their rows differ in the displayed tags, including RW and Ver for Irsan et al. and MS and Pat for SPVR.Source: T3 Patches & fixes · p. 33

T4 Tests, PoCs & exploits

• Ghera [96] is listed as mod/SY and Magma [53] as repo/IG; both have Ver labels, with leakage recorded as none and availability as Pub.Their kind is D in the table.Source: T4 Tests, PoCs & exploits · p. 33
• Other listed records span func, file, mod, and unspecified granularity, with both RW and IG/SY entries.Examples include Zhang et al. (func/RW), RedCode [47] (file/RW), and AgentDojo [34] (unspecified/SY).Source: T4 Tests, PoCs & exploits · p. 33
• CyberGym [160], PBFuzz [184], and SecureVibeBench [18] are all listed as repo/RW with Ver labels, but differ in leakage and availability entries.CyberGym is Chr/–, PBFuzz Con/Pub, and SecureVibeBench Con/–.Source: T4 Tests, PoCs & exploits · pp. 33–34
• The 2025–2026 records include verified datasets with varied leakage and availability entries, alongside records with missing or dash-coded fields.For example, VulnRepairEval [157] is repo/RW with Ver, none, and No; SecRepoBench is repo/RW with Pat and Con; and Qu et al. [124] is file/IG with Ver and Ded.Source: T4 Tests, PoCs & exploits · p. 34

T6 Traces & logs

• Kozak et al. [73] (2025) is recorded as RW, Ver none, B.Source: T6 Traces & logs · p. 34
• Kasu et al. [68] (2026) is recorded as SY, Tool n/a, Req D.Source: T6 Traces & logs · p. 34
• SWE-chat [10] (2026) is recorded as RW, Tool n/a, Pub B.Source: T6 Traces & logs · p. 34

6 Discussion and Future Directions

• Tests, PoCs, and exploits are the only well-populated artifact type where majorities meet both realism and label-evidence criteria, while code-sample datasets are weakest on realism.Only one of 41 deep-coded code-sample datasets uses authentic real-world data, and it qualifies because its samples are whole smart contracts rather than sliced functions.Source: §6 · p. 34
• Leakage reporting is absent in 49 of 90 applicable datasets, availability is unreported by more than a quarter, and traces and reasoning remain sparsely represented.Traces appear in three deep-coded datasets, reasoning has grown to eight since 2024, and no quality attribute has been the primary object of more than fourteen studies.Source: §6 · p. 34

6.1 Good practice, attribute by attribute

• The authors frame good practice as a target, an affordable fallback, and a reporting obligation for each quality attribute.They argue that target-only guidance would leave most dataset builders without actionable advice; Table 10 collects reporting obligations.Source: §6.1 · p. 35, Table 10 · p. 37
• For realism, preserve the vulnerability’s original context or mine at the coarsest affordable granularity; report provenance, sample granularity, and execution validation.For labels, the target is an independently checkable oracle; where none exists, prefer two independent weak signals and state the oracle used.Source: §6.1 · p. 36
• The 51 datasets called verified should report whether confirmation came from execution or manual review, since “verified” alone does not identify label provenance.The paper gives re-running a PoC as an example of a specific provenance statement.Source: §6.1 · p. 36
• Scale should match the chosen oracle’s capacity and be reported with a scaling curve, while diversity reporting should cover CWE, project, language, and class balance.The guidance also calls for stating dataset scope rather than presenting a narrow scope as general.Source: §6.1 · p. 36
• For leakage, report the split strategy or explain why none applies; for availability, provide a versioned link or a rebuildable construction pipeline and state release status.The discussion notes that 25 of 93 datasets make no availability statement, while five explicitly state one; executable artifacts are costly to rebuild when withheld.Source: §6.1 · pp. 36–37

6.2.1 Most vulnerability labels lack an affordable independent oracle.

• The field’s current data foundation imposes seven limitations, including the lack of a general-purpose, affordable independent oracle that establishes vulnerability across relevant categories.Existing labeling routes trade soundness against cost: static analyzers can be inaccurate, while expert manual labeling does not scale.Source: §6.2 · p. 37, §6.2.1 · p. 37
• For roughly one CVE in six, static-code context cannot reliably recover the intended label [45], while package hallucinations permit exact counts at 576,000-sample scale [139].The discussion also notes the lack of an accepted security oracle for generated code and the reliance on hand-built scanner-plus-inspection scenarios.Source: §6.2.1 · p. 38

6.2.2 The unit of data discards the evidence.

• 50 of 111 deep-coded papers use function slices that omit manifestation context; detectors can instead exploit names and known-pattern artifacts.Function slices cannot represent vulnerabilities dependent on interprocedural or environmental context.Source: §6.2.2 · p. 38

6.2.3 Seed-derived generators inherit their source’s coverage.

• Seed-derived generators inherit their source corpus’s class coverage, and single-statement injections cannot express multi-line changes required by over 65% of real-world vulnerabilities.The discussion argues that more compute does not overcome a generator’s restricted edit vocabulary unless additional knowledge or generation mechanisms expand it.Source: §6.2.3 · p. 38
• VinJ reaches 686k samples at 69% correctness, none validated by execution, while augmentation gains plateau past roughly 5k samples.Generator realism is typically argued indirectly through detector improvements rather than evidence that samples are exploitable; adversarial augmentation can improve robustness without evidence for rare classes.Source: §6.2.3 · p. 38

6.2.4 Reported performance does not measure capability.

• Over 36% F1 swings between balanced and imbalanced versions of the same real-world data, while detector rankings invert across benchmark suites and evaluation protocol can sharply alter reported performance [29].When security and functionality are evaluated jointly with multiple detectors, many published techniques degrade the base model by more than 50% [29].Source: §6.2.4 · p. 39

6.2.5 Coverage is siloed, and the silos are unrepresentative.

• C/C++ appears in 27 of 93 datasets and only 7 datasets span three or more languages; NVD-linked patch models lose up to 90% F1 in the wild [62].Widely used mined datasets therefore overrepresent publicly disclosed, well-linked vulnerabilities rather than vulnerabilities generally.Source: §6.2.5 · p. 39

6.2.6 Reproducibility remains limited.

• 25.5% of reviewed deep-learning detectors released a tool; among those assessed, 14.3% were partially and 28.6% fully replicable on another real-world dataset.Separately, 25 of 93 datasets make no availability statement, and only 12 of 24 executable-artifact datasets are publicly available.Source: §6.2.6 · p. 39

6.2.7 Three failure modes the corpus cannot support work on.

• Agents emit insecure actions in 21% of ordinary-operation trajectories [73], yet no labeled behavior corpus existed, while large reasoning datasets rely on model-generated annotations checked by another model.Building the reported trajectory detector required hand-labeling 500 steps [73]; the one expert-authored reasoning set contains 174 cases, leaving accounts without independent ground truth.Source: §6.2.7 · p. 40
• Poisoned code models can evade state-of-the-art defenses and yield vulnerable code 95% of the time with under 5% functional-benchmark loss; one poisoned retrieval sample yields 48% vulnerable code.Consumers also cannot distinguish mined from generated code or verify that distributed artifacts reached them untampered.Source: §6.2.7 · p. 40

6.3.1 Make executable validation a construction step.

• 78.9% of CVEs lack public PoCs and 70.2% of CVE patches lack tests, while one exploit benchmark retained 23 usable instances from 448 CVEs.The authors urge reporting validation yield, generating PoCs during dataset construction, and publishing failed reproductions.Source: §6.3.1 · pp. 40–41

6.3.2 Validate generated data by execution where it applies.

• 79.40% synthetic versus 10.03% real performance for the same neural editors illustrates the gap generated-data checks should expose.Generated samples should ship with a triggering input or be labeled unvalidated; generators should report edit span, since over 65% of real vulnerabilities require multi-line changes.Source: §6.3.2 · p. 41

6.3.3 Preserve context: move mining up the granularity ladder.

• Restoring context recovers 89% of inter-procedural vulnerabilities dropped by function-sliced pipelines, whose detectors can learn dataset artifacts instead of vulnerability causes.Repository-level benchmarks exist only at hundreds of instances; the open problem is training-scale mining that retains build configuration, cross-function flow, and valid labels.Source: §6.3.3 · p. 41

6.3.4 A security oracle for generated code.

• 61% of one agent’s solutions are functionally correct but only 10.5% secure, motivating a two-tier security oracle with executable checks and an auditable fallback.Tests cannot adjudicate issues such as weak cryptography, hardcoded credentials, or information leakage, and passing tests does not establish absence of vulnerabilities.Source: §6.3.4 · pp. 41–42

6.3.5 Provenance as a first-class field.

• No released artifact in the corpus carries sample-level, machine-readable provenance, leaving consumers unable to distinguish mined from model-written code.Provenance is distinct from integrity attestation; recording it would enable measuring whether training on AI-written code degrades security.Source: §6.3.5 · p. 42

6.3.6 Agent behavior as data.

• The primary agent-trace corpus contains three datasets, with three more recording traces secondarily; a trajectory-level detector corpus faces costly hand-labeling.The cited trajectory detector required hand-labeling 500 steps, and a larger safety-labeled corpus could support training detectors rather than prompting them.Source: §6.3.6 · p. 42

6.3.7 A reporting standard for vulnerability data.

• Leakage and availability are often unreported, malicious-prompt corpora used incompatible protocols, and only a quarter of studied detection tools were released.The authors propose a lightweight datasheet that makes four items mandatory rather than optional.Source: §6.3.7 · p. 42
• The proposed checklist covers split strategy or contamination guard, label provenance, code provenance, and availability.Table 10 presents the proposal as a checklist.Source: §6.3.7 · p. 42
• JavaVulBench offers a workable template with five split strategies and a per-model contamination audit [144].The marginal labeling cost is low.Source: §6.3.7 · p. 42
• A majority of applicable datasets still omit leakage reporting, although it requires a partitioning decision rather than new evidence and no oracle.The authors frame a reporting convention as an important constraint alongside resource cost.Source: §6.3.7 · pp. 42–43

Relative tractability

• A reporting standard requires no new technique or data and would make other attributes substantially more assessable; validation yield and edit span are inexpensive to report when pipelines already measure them.Recording provenance is also inexpensive where it is known.Source: Relative tractability · p. 43
• Repository-scale training data and trajectory-level safety corpora are expensive to build because they require unavailable infrastructure and substantial engineering and labeling effort.Source: Relative tractability · p. 43
• The ordering suggests reporting practice can narrow some immediate data-quality gaps, while other gaps still require new research and infrastructure.Source: Relative tractability · p. 43

7.1 Construct validity

• All six quality grades are author interpretations, made auditable by item-level definitions and 729 verbatim spans across 111 papers, but not tested for independent-coder reliability.Absence values were checked by targeted full-text scans and reflect reporting practice as well as underlying quality.Source: §7.1 · p. 43
• Assigning one primary artifact type per paper under-counts datasets that bundle several artifact types in rows beyond their primary type.The authors note this as a limitation of their classification.Source: §7.1 · p. 43

7.2 Internal validity

• Verification found one systematic fault—a pipeline default on provenance, corrected corpus-wide—and one record whose values came from a sentence describing a different dataset.Records were drafted by an LLM from full text and verified against the papers by the authors; 80 were checked field by field, while 31 from a second pipeline were screened against four conflict signals, adjudicated by three authors, and then checked field by field as well.Source: §7.2 · p. 43
• The executable-artifact majority finding survives two adverse record recodes, and the leakage finding survives eight.The authors distinguish this robustness check from measuring coding disagreement from a blank form.Source: §7.2 · p. 43

7.3 External validity

• The purposive 111-paper anchor sample characterizes field anchor artifacts, not a population mean; bias direction is unclear because it includes better-documented papers and documented failures.It includes foundational datasets, high-impact recent contributions across six artifact types, and every paper reporting a data-quality finding.Source: §7.3 · pp. 43–44
• 98.6% screening recall applies only within retrieved candidates; retrieval recall is unmeasured, with search caps, intermittent API availability, vocabulary bias, and English-only search as limitations.Named-artifact seeding, citation chasing, supplemental queries, and recall-oriented searching mitigate but do not eliminate retrieval limitations.Source: §7.3 · p. 44
• The corpus-wide type tagger agrees with manual types on 70.7% of papers carrying one, and its recall is nonstationary, but no compositional claim depends on it.Source: §7.3 · p. 44

7.4 Conclusion validity

• The trace matrix row rests on three deep-coded datasets, while a corpus-wide re-screen raised the reasoning row from one dataset to eight; percentages use 93 contributed datasets.Because one reclassification shifts a percentage by about a point and per-band counts are smaller, claims based on few records report fractions as well as percentages.Source: §7.4 · p. 44

8 Conclusion

• Executable artifacts are the only type with majorities meeting both realism and label-evidence criteria: 16 of 24 for realism and 23 of 24 for label evidence.The authors caution that part of the quality-profile association reflects the operational definitions of executability and realism rather than independent measurement.Source: §8 · p. 44
• Code samples, the largest category in both the corpus and anchor set, have just one of 41 datasets meeting the realism criterion.Source: §8 · p. 44
• Most applicable datasets report no effective leakage guard, a quarter make no availability statement, and artifacts for explaining verdicts or recording agent behavior remain scarce.Source: §8 · p. 44
• Previously identified data problems remain common in practice, with a gap between documented best practices and observed practice especially clear for low-cost measures such as chronological splitting.Chronological splitting requires a partitioning decision rather than additional labeling effort.Source: §8 · p. 44
• As consumers become autonomous and act on data rather than only presenting results to developers, the authors call for a shift from data that asserts vulnerabilities to data that demonstrates them.Source: §8 · p. 44

A Search and Screening Protocol

• The three harvesting phases ran from 8–17 July 2026, and the corpus was frozen for coding on 28 July.The harvesting and screening scripts were released so the figures could be recomputed.Source: A Search and Screening Protocol · p. 45

A.1 Phase 1 — topic harvesting

• Phase 1 issued 23 phrase queries across four tagged families to arXiv, DBLP, and, on a best-effort basis, Semantic Scholar.The four families were data, AI for security, security of AI-generated code, and agent behaviour.Source: §A.1 · p. 45

A.2 Phase 2 — named-artifact seeding and backfill

• 37 named artifacts were curated in Phase 2, adding 31 new records while six had already been found by topical search.The 19 named-artifact queries had no year floor, allowing foundational resources predating the review window to enter.Source: §A.2 · pp. 45–46

A.3 Phase 3 — supplemental retrieval and enrichment

• Phase 3 added ten conjunctive arXiv queries and six DBLP retries, then enriched records with publication metadata and abstracts and pursued backward citation chasing.Exact-phrase searching had returned few results for newer areas.Source: §A.3 · p. 46

A.4 De-duplication

• 2,240 records were retrieved across the three phases; title-key and DOI matching merged 128, leaving 2,112 candidates.Title keys lower-cased and collapsed non-alphanumeric runs; keys shorter than 15 characters were discarded, and duplicate records were merged.Source: §A.4 · p. 46

A.5 Screening dispositions

• 2,112 candidates received screening dispositions: 1,171 by automated rule, 668 by LLM-assisted screening, 236 by two-author independent screening, and 34 by manual seed curation.These sum to 2,109; three expert-recommended candidates were added after screening and coded directly.Source: §A.5 · p. 46
• The 668 LLM-assisted decisions came from three interactive passes, with no single prompt, model version, or temperature to report, so rerunning would not reproduce their verdicts.Each decision has a reason string, and excluded records remain available for individual auditing.Source: §A.5 · p. 46

B.1 The two verification routes

• 19 of 372 checked values (5.1%) were wrong, all due to provenance defaulted to mixed; the corpus-wide correction classified 15 as human and 4 as AI-generated.The second pass covered 31 records. Eighty records were checked field-by-field by one author; the 31 records verified by triage were screened against conflict signals and adjudicated by three authors.Source: §B.1 · p. 47
• All 332 rubric-graded values across 111 records had verbatim-span backing; numeric checks confirmed 231 scale and 155 diversity figures, and verification changed a field in 31 of 68 first-round records.That is a record-level change rate, not a measured per-value error rate; the 68 records contain 884 value slots.Source: §B.1 · p. 47

B.2 Agreement, and what triage changed

• Reviewers upheld 94 of 99 triage verdicts (94.9%), mean pairwise agreement was 96%, and the coded value was retained for 32 of 33 flags.Triage corrected one T1 dataset from injected/generated to synthetic because its samples were model-generated from question text, not real code with an inserted flaw.Source: §B.2 · p. 47

B.3 Tagger accuracy by type and period

• Patches had recall 1.00 and precision 0.80, while code samples scored 0.79 / 0.73; executable artifacts were recovered 55% of the time and reasoning data 12%.The low reasoning recovery motivated a re-screen that found six datasets missed by the tags.Source: §B.3 · p. 48
• The tagger recovered 0 of 3 true T4 papers through 2023, then 6 of 9 in 2024–25 and 6 of 10 in 2026, undermining trend measurement from a 2023 baseline.The change was a jump between the earliest period and the rest, not a steady climb.Source: §B.3 · p. 48

B.4 The re-screen of excluded records

• An independent re-screen of 808 exclusions returned 217 papers, expanding the corpus from 1,298 to 1,515; re-inclusions included 155 executable-artifact and 54 trace papers.The missed records were disproportionately executable and observational work; readers worked from title and abstract without seeing the original exclusion reason.Source: §B.4 · p. 48
• Applying anchor criteria to all 1,522 included papers yielded 111 records; full-text review added one post-training study and one primary trace corpus, while three trace-emitting releases were classified as evaluation records.None of the 217 re-inclusions qualified under T1a or T1b; one of ten T1c candidates qualified, and two of five apparent T1d trace corpora qualified.Source: §B.4 · p. 49

B.5 Estimating residual screening recall

• A second reader restored 7 of 122 sampled exclusions; size-weighting estimates a 4.5% residual false-exclusion rate versus the original screen’s 26.9%, implying roughly 21 eligible papers among 469 unread exclusions.The sample was allocated across all 41 exclusion categories, with a floor of two records per category.Source: §B.5 · p. 49
• 98.6% (97.5–99.8%) was the estimated screening recall within retrieved candidates, not retrieval recall; the corpus also inherits search-language concentration.The estimate says nothing about work the queries never returned.Source: §B.5 · p. 49

B.6 How much coder disagreement the conclusions absorb

• The executable-artifact finding survives two adverse recodes and reverses on the third; patch datasets would join after three specified regrades, while T4 holds until four.These thresholds are reported for types with at least ten datasets; the executable and patch rows contain 24 and 14 datasets, respectively.Source: §B.6 · pp. 49–50
• The leakage finding survives eight adverse recodes; 109 graded values are near-mechanical, while 180 depend on judgments about authenticity or independent checks.The executable and patch rates differ by 31 points, and the paper reports recode thresholds in record counts because records are the coding unit.Source: §B.6 · p. 49

C Released Materials

• Released materials include harvesting and analysis scripts, screening rules, the candidate table, the deep-coding schema and codebook, and coded records with evidence spans and verification worksheets.The release also includes re-screening worksheets covering 808 excluded records, second-reader validation, and checks reconciling text counts with the coded corpus.Source: C Released Materials · p. 50