PaperlayerView exact revision on arXiv

Source-linked AI summary

The Data Problem in Software Vulnerability Analysis: Artifacts, Quality, and Consumption

Yu Nong, Yao Du, Tianxiang Xu, Haipeng Cai

arXiv:2609.01503v1cs.SE

TL;DR

Software vulnerabilities remain consequential and persistent, while vulnerability-analysis data is not routinely examined as a first-class object. This study introduces a data-centric taxonomy and tiered reading protocol, finding that detectors lose more than half their accuracy on real-world code and that label accuracy in widely used corpora falls below 60%.

  • Problem

    Software vulnerabilities remain among the most consequential and persistent threats in computing, while the literature concentrates on some vulnerability-artifact types more than others.

  • Method

    The study uses a data-centric taxonomy of vulnerability data and a two-tier reading protocol for examining its findings and matrix.

  • Results

    Detectors lose more than half of their accuracy when moved to real-world code, while label accuracy in widely used corpora falls below 60%.

  • Takeaways & Limitations

    The findings connect vulnerability-analysis reliability with the realism and label quality of its training and evaluation data.

  • Takeaways & Limitations

    The study cannot cleanly separate how much of the observed skew is a property of the area from effects of faster publishing.

Abstract

from arXiv · show

Learning- and LLM-based software vulnerability analysis is only as trustworthy as the data it is trained and evaluated on, yet that data is rarely examined as a first-class object. We investigate the data behind vulnerability analysis through a dataset-centric taxonomy that separates what an artifact is (code, metadata, patches, tests/PoCs, reasoning, traces), how good it is (realism, label evidence, scale, diversity, leakage, availability), and what it is used for. From a systematically assembled corpus of 1522 papers covering 2016-2026 plus foundational earlier work we deep-code a tiered set of 111 anchor papers, backing every affirmative rubric-graded value with a verbatim span, and we report, per attribute, both how much it has been studied and how well datasets achieve it. The results trace an evidence ladder: executable artifacts are the only major type where 15 of the 24 datasets are both graded real-world and carry labels that received an independent check, while code-sample datasets-the largest category in both the auto-tagged corpus and the anchor set-are the least realistic: 20 of the 41 draw their vulnerabilities from authentic projects or CVEs, but only 3 keep the sample at the unit the code is deployed in, and only 2 do both-though these are coarse component tests, and just one code-sample dataset meets the codebook's stricter full-context real-world grade. Among these, leakage goes unaddressed by 49 of the 90 datasets where it applies, more than a quarter say nothing about availability, reasoning data has arrived only recently and is mostly model-generated, and primary trace corpora remain limited to three datasets, the total after a corpus-wide screen and a full-text check of every candidate it surfaced, with further datasets releasing traces secondarily behind benchmarks and harnesses.

1 Introduction

Software vulnerability-analysis data is rarely examined as a first-class object, despite evidence that weak realism, labels, and splits undermine trust. This study spans artifact types and quality attributes to characterize those weaknesses systematically.

  • Detectors lose more than half of their accuracy when evaluated on real-world code, while near-duplicate train/test pairs can reach 68%.These findings were previously reported for individual datasets or techniques but are presented here as a common data condition.
  • Label accuracy falls below 60% in widely used corpora, and roughly one CVE in six has a label unrecoverable from supplied static code alone.
  • Leakage is unaddressed by 49 of 90 applicable datasets, more than a quarter omit availability statements, and primary trace corpora number only three.Reasoning data appeared only since 2024 and is dominated by model-generated annotation.
  • The study contributes a data-centric taxonomy, an auditable corpus with 111 deep-coded anchor papers, and an artifact×quality matrix.It separates how much each quality attribute is studied from how well datasets achieve it.

2 Background and Related Work

Vulnerability-analysis data evolved from synthetic suites to patch-derived corpora and then to LLM- and agent-oriented artifacts. Across these eras, greater scale often came with weaker realism, label reliability, or contamination control.

  • Program-analysis era: Synthetic benchmarks provide labels by construction, but artificial vulnerabilities are easier to detect than real ones and can invert detector rankings.Per-category recall ranged from 0% to 100%, making a single headline accuracy figure misleading.
  • Deep-learning era: Security-fix commits enabled large supervised datasets, but touched functions may be mislabeled and detectors later lose more than half their performance on realistic data.
  • LLM-and-agent era: LLM and agent systems introduce AI-generated-code security corpora and executable benchmarks that measure actions against real or emulated systems.
  • LLM-and-agent era: Undisclosed training corpora make test-set pretraining contamination difficult to assess because unseen data can no longer be assumed.
  • Evidence ladder: The evidence ladder orders artifacts from assertions toward demonstrations: executable exploits provide independently checkable reachability evidence, whereas traces require interpretation.
  • Evidence ladder: Executable artifacts are the only well-populated type whose majorities satisfy both realism and label evidence, yet they are among the least shared.Their realism reflects retained deployment context, while a run checks triggered behavior directly.
  • Related work: This study differs from prior reviews by spanning six artifact types and six quality attributes through coded rather than narrative synthesis.

3 Methodology

The study combines a three-phase literature search, transparent data-centric screening, and two-tier reading. Deep coding supports quality and consumption findings, while corpus-wide metadata supports distributional attention patterns.

  • Research questions: The research questions cover artifact representation, data quality, and consumption across traditional analysis, deep learning, LLMs, and coding agents.
  • Study design: The pipeline uses a three-phase search, taxonomy-driven screening, and a two-tier reading protocol matched to each question’s evidence needs.
  • Study design: Every affirmative rubric-graded value is backed by a verbatim source span, while absence values receive targeted full-text checks.
  • Study design: Harvesting, coding, and analysis re-execute from released scripts, but the LLM-assisted screening of 668 ambiguous abstracts is not re-executable.Excluded records from that stage were re-examined by three authors, so final exclusions rest on a documented human pass.
  • Corpus construction: The search combines topical harvesting, named-artifact seeding, and citation backfill to reduce early bias toward post-2021 work.
  • Corpus construction: The assembled corpus contains 1,522 included records after deduplication, with 813 preprints and 52 refereed papers among the 111-paper deep-coded set.
  • Two-tier reading protocol: Tier 1 deep-codes 111 papers for quality findings and the matrix, whereas Tier 2 supplies corpus-scale attention patterns from titles, abstracts, and metadata.

4 Taxonomy of Vulnerability Data

The taxonomy separates what vulnerability data is, how good it is, and what it is used for. It orders artifact types by demonstrative evidence while treating reasoning and traces as complementary dimensions.

  • Design principles: The taxonomy has three non-overlapping aspects: Form, Quality, and Use, with each attribute assigned to exactly one aspect.
  • Design principles: Attributes are operationally defined with closed vocabularies so disagreements remain auditable; resistant properties are recorded as reported figures instead.
  • Design choices: The taxonomy excludes evaluation metrics, folds production method into realism, and treats reproducibility as a lens derived from other properties.
  • Artifact types: Metadata asserts a category, code samples assert vulnerability, reasoning explains why, patches localize a suspected flaw, and executable artifacts demonstrate reachability.
  • Artifact types: Traces are observational records of execution that presuppose a trigger and require interpretation, rather than constituting stronger demonstrations.
  • Artifact ordering: Reasoning adds explanatory content but remains an assertion and does not make a flaw more demonstrably real.
  • Artifact ordering: Artifact rows follow demonstrative strength, rising from metadata and code-sample assertions through patches to executable evidence.
  • Quality and use: Quality attributes include realism, label evidence, scale, diversity, leakage or contamination, and availability; Use records downstream analysis tasks.

C. Consumption (use)

The taxonomy separates artifact types, quality attributes, and consuming tasks, while distinguishing demonstrative strength from explanatory content and practical usefulness. Its coding conventions define how artifacts, evidence, realism, leakage, availability, and consumption are compared.

  • Evidence ladder: The evidence ladder ranks artifacts by what they establish, not by their usefulness for training or evaluation.Reasoning may be strongest for explanation, while traces are observational records; a large mined corpus can still be more useful for training than a single exploit.
  • Quality attributes: Realism distinguishes synthetic, injected/generated, mined/semi-real, and real-world artifacts according to origin, context, and label quality.Mined/semi-real data comes from real projects or CVEs but may be simplified or imperfectly labeled; real-world data retains authentic full context.
  • Evidence ladder: Executable artifacts provide independently checkable evidence because a successful run shows that the triggered behavior is reachable.The rubric treats execution as one route to independently checked labels, while patch evidence and descriptions provide weaker bases.
  • Quality attributes: Leakage coding distinguishes unanswered applicable risks from non-applicable risks, while availability separately records public access, restricted access, unavailability, or non-reporting.The taxonomy also records scale and diversity as reported figures rather than imposing cross-type thresholds.
  • Taxonomy and consumption: Artifact types are technique-agnostic, while consuming techniques record which paradigmatic consumer the dataset was built for.The study records consuming tasks and techniques from what each paper evaluates, not from what the data could theoretically support.
  • Coding conventions: The matrix uses one primary artifact type per dataset, so bundled code, patches, and tests are counted once and secondary contributions are under-represented.Study papers are coded by the quality of the data they analyze, while multi-dataset studies generally receive not-assessed values for realism, label evidence, and leakage.

5 Paper Attribution and Study Results

The corpus is recent and dominated by code samples, while reasoning and trace artifacts remain comparatively new or scarce. Across artifact types, the study finds persistent trade-offs between scale, realism, label evidence, validation, and context preservation.

  • Corpus distribution: 1,522 included papers are concentrated in 2023–2026, with 117 in 2023, 255 in 2024, 429 in 2025, and 554 in 2026.The 2026 count covers a partial year, while 2020–2022 contributed 22, 35, and 68 papers and earlier years contributed 42.
  • Corpus distribution: 753 papers concern code samples, followed by 376 tests/PoCs/exploits, 156 patches, 119 metadata papers, 94 traces, and 24 reasoning papers.These auto-tagged primary-artifact counts cover all 1,522 included papers.
  • Emerging artifacts: Eight deep-coded reasoning datasets all appeared in 2024 or later, while only three deep-coded datasets provide primary traces.Corpus-wide, 24 papers carry reasoning data and 94 carry traces, but these paper counts do not equal released dataset counts.
  • Code samples: Code samples are the least realistic major category: only one of 41 contributes authentic, full-context real-world vulnerability data.Most code-sample datasets slice functions or files from larger projects, discarding calling context, build configuration, or inter-procedural flow.
  • Code samples: SVEN prioritizes trustworthy labels across 1,606 programs and nine CWEs, whereas Big-Vul scales to 3,754 vulnerabilities across 348 projects with audited label accuracy below 60%.The comparison illustrates the field’s continuing trade-off between manually validated evidence and scalable mining.
  • Generation and validation: Generated-data methods improve scale but retain validation limits: Smart-LLaMA-DPO reports 69% vulnerable outputs, and VulScribeR sees detector gains plateau beyond roughly 5k generated samples.Prompting reaches 88% injection success on real-world code, but success varies sharply by CWE and many vulnerabilities require multi-line edits.
  • Reasoning data: Reasoning data is usually an annotation layer over existing code, with 7 of 8 deep-coded datasets classified as mined/semi-real or less realistic.Examples pair mined functions with structured explanations or chain-of-thought answers, so reasoning data generally inherits the source corpus’s limitations.

T2 Metadata

The metadata entries cover recent work from 2025–2026 and use varied artifact granularities, realism levels, labels, leakage handling, availability, and study kinds.

  • 2025–2026 entries include function- and model-level artifacts, with synthetic, mined/semi-real, and description-based records.
  • The listed records vary in leakage treatment, including deduplication, project-disjoint splits, and unreported handling.
  • Availability ranges from public release to on-request access or no reported release, while entries include datasets, studies, and both.

T1 Code samples

The code-sample records are predominantly function-level and include mined, injected/generated, synthetic, and real-world entries with varied labels and leakage practices.

  • Granularity: Most listed code-sample records use function-level granularity, alongside file-, statement-, model-, and repository-level entries.
  • Coverage: The entries span datasets, studies, and combined dataset-and-study contributions across 2020–2026.
  • Labels: Labels are commonly patch-derived or verified, with some tool-derived, description-derived, and unreported labels.
  • Availability: Availability is frequently public but is also withheld, on request, or not reported across the listed records.
  • Realism: The records combine mined/semi-real, injected/generated, synthetic, and real-world data sources.
  • Leakage: Leakage handling ranges across random, deduplicated, chronological, project-disjoint, contamination-controlled, and unaddressed settings.

T5 Reasoning

The reasoning-related records are recent and include function-, file-, and model-level artifacts with mined, synthetic, and withheld or public availability states.

  • Coverage: The listed reasoning records date from 2024–2025 and use function-, file-, and model-level granularity.
  • Realism: Their realism labels include mined/semi-real, synthetic, and real-world entries.
  • Quality attributes: The table distinguishes verified and tool-derived labels, deduplicated and unaddressed leakage handling, and public or withheld availability.

T3 Patches & fixes

The patch-and-fix records span statement-, function-, file-, and repository-level artifacts, with mined, real-world, synthetic, and injected/generated sources.

  • Granularity: The listed patch-and-fix records use statement, function, file, and repository granularity.
  • Labels: Labels are verified, patch-derived, tool-derived, or unreported across the records.
  • Leakage: Leakage treatment varies among random, chronological, deduplicated, project-disjoint, and unaddressed settings.
  • Availability and kind: Availability is public, withheld, or unreported, and the records include datasets, studies, and combined contributions.
  • Realism: Realism categories include mined/semi-real, real-world, synthetic, and injected/generated data.

T4 Tests, PoCs & exploits

The T4 section catalogs tests, PoCs, and exploits across papers, recording artifact granularity, realism, label evidence, leakage, availability, and dataset kind.

  • The catalog lists T4 datasets from 2017 through 2026, including Ghera, Magma, CyberGym, PBFuzz, and SecureVibeBench.
  • Entries distinguish repository, file, function, module, or unspecified granularity alongside realism and label-evidence grades.
  • The table records varied leakage and availability statuses, including Pub, No, Con, Ded, and unspecified values.

T6 Traces & logs

The T6 trace section contains three listed datasets, spanning 2024 to 2026 and differing in granularity, realism, label, leakage, availability, and kind.

  • Three trace-related entries are listed: Kozak et al. (2025), Kasu et al. (2026), and SWE-chat (2026).
  • The entries use unspecified, module, or unspecified granularity and report synthetic or real-world provenance, verified labels, and varied availability.

6 Discussion and Future Directions

The discussion finds that evidential strength generally rises with demonstrative strength, but major gaps remain in leakage reporting, availability, reasoning, traces, and data realism.

  • Availability: Availability runs counter to evidential strength: executable artifacts are difficult to build and share, with public availability of 12 of 24.
  • Evidence ladder: Tests, PoCs, and exploits are the only well-populated artifact type whose majorities meet both realism and label-evidence criteria.
  • Evidence ladder: Code-sample datasets are the largest category yet weakest on realism, with only one of 41 using authentically real-world data.
  • Recurring omissions: 49 of 90 applicable datasets omit train/test leakage reporting, while more than a quarter make no availability statement.
  • Underrepresented artifacts: Traces remain represented by three deep-coded datasets, reasoning has grown to eight since 2024, and no quality attribute was the primary object of more than 14 studies.
  • Limitations: The paper identifies seven limitations of the current data foundation, including context loss, inherited generator coverage, weak validation, leakage, and non-reproducible results.

Relative tractability.

The study distinguishes data-quality gaps that can be narrowed through reporting practice from those requiring new research and infrastructure. Validation yield, edit span, and provenance are relatively inexpensive to report, whereas repository-scale training data and trajectory-level safety corpora are costly.

  • Validation yield, edit span, and provenance are inexpensive to report when pipelines already measure or know them.
  • Repository-scale training data and trajectory-level safety corpora require infrastructure that does not yet exist.
  • Immediate data-quality gaps can be narrowed through reporting practice, while others require new research and infrastructure.

7 Threats to Validity

The validity threats concern interpretation, coverage, sampling, coding, and reproducibility. The authors make the coding auditable, but limitations remain in reliability, external validity, and rerun reproducibility.

  • Interpretation: Human judgments and author-defined quality attributes mean every graded value is an interpretation of source-paper descriptions.
  • Interpretation: 729 verbatim spans make affirmative coding decisions auditable, but independent-coder agreement was not established.
  • External validity: The purposive set of 111 anchor papers characterizes anchor artifacts but cannot estimate a population mean, with bias direction unidentified.
  • Coverage and coding: Screening recall within the retrieved candidate pool was 98.6% after re-screening excluded records.
  • Coverage and coding: Corpus-wide type tagging agrees with manual types on 70.7% of papers carrying one, and recall is not stationary.

8 Conclusion

The conclusion frames vulnerability datasets as an evidence ladder, from assertions to demonstrations, with quality varying across artifact types. Executable artifacts perform best on realism and label evidence, while code samples and explanatory or trace artifacts retain major gaps.

  • The study deep-codes 111 anchor papers from a systematically assembled corpus of 1,522 to analyze vulnerability data.
  • The association between evidence level and coded quality is clearest at the ladder’s ends, but partly follows from operational definitions of executability and realism.
  • Executable artifacts are the only sufficiently populated type whose majorities satisfy both realism and label evidence: 16 of 24 and 23 of 24.
  • Code samples are the largest category and achieve one of 41 on realism.
  • Most applicable datasets report no effective leakage guard, a quarter make no availability statement, and explanatory or agent-behaviour artifacts remain scarce.
  • The paper motivates shifting from data that asserts vulnerabilities toward data that demonstrates them as consumers become autonomous.

A Search and Screening Protocol

The search and screening protocol combines multi-source queries, named-artifact seeding, citation chasing, transparent screening, and auditable decision records. It produced 2,112 candidates before screening and retained reasons for every exclusion.

  • Search: Three harvesting phases used topical, named-artifact, supplemental, and citation-chasing searches across arXiv, DBLP, and Semantic Scholar.
  • Search: 37 named artifacts were curated, with 31 entering the candidate pool as new records.
  • Deduplication: 2,240 source records were merged into 2,112 candidates using normalized-title and DOI matching.
  • Screening: Every candidate received one disposition through automated rules, LLM-assisted screening, independent author screening, or manual seed curation.
  • Screening: LLM-assisted screening was not reproducible as a single run because it used three interactive passes without a single prompt, model version, or temperature.
  • Auditability: Per-record reason strings and retained exclusions make individual screening decisions auditable even though the pass cannot be rerun.

B.1 The two verification routes

The study combines full-text verification with targeted triage to check coded dataset attributes and correct systematic errors. These checks changed substantive records while supporting high agreement and complete span backing for rubric-graded values.

  • Full-text verification: 19 of 372 field-checks were wrong, and all 19 concerned provenance defaulted to mixed rather than the papers’ human or AI-generated descriptions.The errors affected 19 of the 31 records checked field by field.
  • Full-text verification: 332 rubric-graded values across 111 records were all backed by verbatim spans.Absence values were handled through targeted scans instead, while granularity, scale, and language coverage were read directly from papers.
  • Full-text verification: 31 of 68 records from the first reading round changed in at least one coded field after direct verification.Verification also found a scale and label borrowed from a different dataset’s related-work description.
  • Targeted triage: 94 of 99 individual triage verdicts upheld the coded value, with 96% agreement and pairwise Cohen’s 𝜅 values of 0.47, 0.65, and 0.65.At the flag level, the coded value was retained for 32 of 33 flags.
  • Targeted triage: Triage corrected one dataset from injected/generated to synthetic and another from real-world to mined/semi-real.The corrections followed the codebook’s treatment of model-generated samples and CVE-derived functions labeled from patches.
  • Corpus screening: The tagger agreed with manual artifact types on 70.7% of records, with recall 1.00 for patches, 0.79 for code samples, 0.55 for executable artifacts, and 0.12 for reasoning data.The low reasoning recall prompted a corpus-wide re-screen that found six missed datasets.
  • Corpus screening: The corpus-wide re-screen restored 217 papers, increasing the corpus from 1,298 to 1,515, with executable artifacts and traces disproportionately missed by the original screen.A later eligibility check restored seven additional papers, producing the final total of 1,522.
  • Robustness: The executable-artifact majority claim survived two adverse recodes but reversed on the third, whereas the leakage finding survived eight adverse recodes.Threshold comparisons were restricted to types with at least ten datasets because small rows are highly sensitive to individual recodes.
Loading 2609.01503v1…