PaperlayerView exact revision on arXiv

Source-linked AI summary

SecureCode: A Production-Grade Multi-Turn Dataset for Training Security-Aware Code Generation Models

Scott Thornton

arXiv:2512.18542v3cs.CRcs.AIcs.CLcs.LG

TL;DR

Security training data for AI coding assistants has not jointly covered web and AI/ML defenses in an instruction-tuning format. SecureCode provides an audited multi-turn dataset spanning both domains, and releases it with related tools and fine-tuned models as open-source artifacts.

  • Problem

    No public training dataset jointly addressed web security and AI/ML-specific defenses in a conversational format for instruction tuning, despite vulnerabilities in AI-generated code.

    Source: §1.1 · p. 2, §1.3 · p. 3, §9 · p. 26

  • Method

    SecureCode constructs a multi-turn dataset spanning web and AI/ML security and releases it alongside validation tools, training scripts, and fine-tuned models.

    Source: §9 · pp. 26–27

  • Results

    2,372 audited examples span 20 vulnerability categories across 12+ languages and 49+ frameworks, with the dataset and related artifacts released as open source.

    Source: §9 · pp. 26–27

  • Takeaways & Limitations

    The release provides researchers, practitioners, and educators with dataset and model artifacts for work on secure coding and AI coding assistants.

    Source: §9 · p. 27

    Quantitative comparisons of fine-tuned models with baselines were pending, limiting evidence for a direct dataset-to-outcome causal link.

    Source: §8.3 · p. 26

Abstract

from arXiv · show

AI coding assistants produce vulnerable code in 45\% of security-relevant scenarios~\cite{veracode2025}, yet no public training dataset teaches both traditional web security and AI/ML-specific defenses in a format suitable for instruction tuning. We present SecureCode, a production-grade dataset of 2,185 multi-turn security training examples spanning two domains: web application security (1,435 examples covering the OWASP Top 10 2021 across 11 languages and 9 frameworks, 100\% grounded in documented CVEs and security incidents) and AI/ML security (750 examples covering all 10 OWASP LLM Top 10 2025 categories across more than 40 frameworks, including LangChain, OpenAI, and Hugging Face). Every example follows a 4-turn conversational structure -- feature request; vulnerable and secure implementations with attack demonstrations; advanced probing; and defense-in-depth operational guidance -- designed for direct use in instruction tuning pipelines. Quality assurance combines automated structural validation with multi-agent review from seven specialist AI perspectives (more than 10{,}500 assessments) and an 8-phase remediation pipeline, producing a rubric-calibrated mean quality score of 93.8/100 ($\sigma = 0.93$) for the AI/ML component. Each example provides SIEM integration strategies, infrastructure hardening recommendations, and testing approaches using production frameworks. We release the unified dataset on Hugging Face with domain-specific loading configurations (web, aiml, default), alongside eight fine-tuned open-source models (3B--20B parameters, QLoRA), and an evaluation framework with four security-specific metrics. To our knowledge, SecureCode is the first public dataset that jointly provides OWASP Top 10 2021 web coverage and OWASP LLM Top 10 2025 AI/ML coverage in a unified conversational schema suitable for instruction tuning.

1 Introduction

SecureCode addresses gaps in security training data with a unified dataset covering web and AI/ML security in a conversational format. Its design includes vulnerable and secure code, attack demonstrations, and operational guidance.

1.1 The Security Crisis in AI-Generated Code

  • AI coding assistants produce vulnerable code in 45% of security-relevant implementations, and insecure patterns can propagate across projects.The introduction also notes that AI-generated vulnerabilities may enter production without the code-review scrutiny applied to human-written code.Source: §1.1 · p. 2

1.2 The Emerging AI/ML Security Gap

  • The OWASP LLM Top 10 2025 identifies AI-specific threats, while training data for coding assistants had not addressed these vulnerability classes.The introduction points to prompt injection, RAG poisoning, and other AI/ML risks as part of this emerging security gap.Source: §1.2 · p. 2

1.3 Why Existing Datasets Fall Short

  • Existing datasets range from large synthetic test suites to narrower code corpora, but often lack incident grounding and production-relevant context.Juliet is described as synthetic and detached from real-world incidents, while the audit of CWE-Sans found approximately 18% of examples referenced actual CVEs or documented breaches.Source: §1.3 · pp. 2–3
  • Existing datasets generally omit multi-turn developer interactions, operational defenses, and AI/ML vulnerability coverage.SecureCode's four turns include feature requests, vulnerable and secure implementations with attack demonstrations, advanced scenarios, and operational guidance.Source: §1.3 · p. 3, §1.4 · p. 4

1.4 SecureCode: A Unified Solution

  • SecureCode combines 2,372 examples across web and AI/ML security, with a shared four-turn format and quality assurance using seven specialist perspectives and an eight-phase remediation pipeline.The AI/ML component received a mean quality score of 93.8/100 after more than 10,500 assessments.Source: §1.4 · pp. 3–4

1.5 Contributions

  • The contributions include a unified instruction-tuning dataset, eight fine-tuned open-source models, a quality pipeline, schema unification, and an open-source release.The paper also describes an OWASP taxonomy-alignment method and a four-turn conversational structure.Source: §1.5 · pp. 4–5

2 Related Work

Prior secure-code datasets vary in scale and scope, while studies document security flaws in AI-generated code and a growing AI/ML attack surface. SecureCode is presented as training data aligned with both web and LLM security taxonomies.

2.1 Secure Coding Datasets

  • Existing secure-code datasets have varied scale and scope but lack combinations of incident grounding, broad application coverage, and conversational context for training AI coding assistants.The cited comparison includes synthetic suites, code-only examples, and large corpora focused on particular languages or vulnerability types.Source: §2.1 · p. 5

2.2 AI Code Generation Security Research

  • Studies reported CWE vulnerabilities in 45% of security-relevant AI-generated implementations and more privilege-escalation paths and architectural flaws than in manually written code.Apiiro (2025) reported 322% more privilege escalation paths and 153% more architectural design flaws compared to manuallywritten code [Apiiro, 2025].Source: §2.2 · p. 6

2.3 AI/ML Security Research

  • AI/ML security research documents prompt injection, data and model attacks, model extraction, supply-chain threats, and risks from agentic tool use.The paper notes that no prior dataset translated these findings into training data for AI coding assistants.Source: §2.3 · pp. 6–7

2.4 OWASP Standards

  • SecureCode aligns its web component with all OWASP Top 10 2021 categories and its AI/ML component with all ten OWASP LLM Top 10 2025 categories.The AI/ML component provides 75 examples per category, according to the passage.Source: §2.4 · p. 7

3 Dataset Design

SecureCode pairs vulnerable and secure implementations with attack explanations and production guidance in a four-turn workflow, then unifies web and AI/ML examples in a common training schema.

3.1 Design Principles

  • Examples ground claims in documented sources where available, pair vulnerable code with secure alternatives and layered defenses, and include operational guidance and authentic framework APIs.When no documented source exists, scenarios are labeled representative rather than presented as incidents.Source: §3.1 · pp. 7–8

3.2 Four-Turn Conversation Structure

  • The four turns move from a developer feature request to vulnerable and secure implementations, a deeper follow-up probe, and production security guidance.Turn 2 explains risks and attacks before presenting the secure implementation; Turn 4 covers logging, monitoring, detection, SIEM integration, and hardening.Source: §3.2 · p. 8, Figure 3 · p. 8

3.3 Web Security Domain

  • The web component contains 1,625 examples covering OWASP Top 10 2021, with examples spanning 11 programming languages and framework-native implementations.Framework examples include Express/NestJS, Laravel/Symfony, Spring Boot, Gin, Rails, and ASP.NET Core.Source: §3.3 · p. 9, Table 1 · p. 9

3.4 AI/ML Security Domain

  • The AI/ML component contains 747 examples covering all 10 OWASP LLM Top 10 2025 categories and spans more than 40 frameworks.Three categories have 74 examples each after the fix-correctness audit; the others have 75. Frameworks include LLM APIs, orchestration, vector databases, web, serving, and UI platforms.Source: §3.4 · pp. 9–10, Table 2 · p. 10

3.5 Schema Unification

  • The components’ different conversation schemas were normalized to {role, content}, while preserving domain-specific metadata and providing default, web, and aiml loading configurations.The configurations support training on the combined dataset or on either domain alone.Source: §3.5 · pp. 10–11

4 Dataset Construction

Construction combined incident-based web examples with generated AI/ML examples, systematic review and remediation, taxonomy and reference corrections, and unified release procedures. The audit also corrected web split contamination and checked AI/ML examples for near duplicates.

4.1 Web Security Examples

  • Web examples were built from mined incidents, synthetically generated vulnerable and secure implementations, and automated validation; remediation raised structural compliance from 47.2% to 100%.A framework expansion added 219 examples, taking the web component from 1,216 to 1,435 examples in this construction account.Source: §4.1 · pp. 11–12

4.2 AI/ML Security Examples

  • The AI/ML pipeline specified 75 scenarios per OWASP category, generated examples with Claude Sonnet 4.5, and validated structure and quality before batch review.The scenarios span 704 unique techniques across 750 files and more than 40 frameworks; failed immediate validation triggered regeneration.Source: §4.2 · pp. 12–13

4.3 Multi-Agent Quality Review

  • Seven specialist agents reviewed all 750 AI/ML examples across two batches, producing 10,500+ assessments; different perspectives caught distinct issues, and no single reviewer caught them all.Examples include missed defense layers, incorrect API signatures, and unreferenced claims.Source: §4.3 · p. 13

4.4 8-Phase Remediation Pipeline

  • The eight-phase remediation pipeline made 2,453+ individual file modifications across the AI/ML examples.Review findings drove the remediation pipeline.Source: §4.4 · p. 13, Table 4 · p. 14

4.5 OWASP 2025 Taxonomy Alignment

  • A mismatch between 2023 numbering and 2025 category names misplaced files; a staging rotation corrected 198 files, generated 67 examples, and archived 71 overflow files.The authors recommend pinning taxonomy versions, classifying by content, automating category validation, and budgeting for rotation rather than relabeling alone.Source: §4.5 · pp. 13–15, Figure 7 · p. 14

4.6 Reference Normalization

  • The AI/ML component’s 2,828 references were normalized from 30+ inconsistent labels to eight canonical types.The types include CVE, CWE, OWASP, research paper, vendor advisory, documentation, blog post, and tool.Source: §4.6 · p. 15

4.7 Dataset Unification

  • Unification converted web conversations to {role, content}, harmonized shared and domain-specific metadata, generated three dataset configurations, and verified no cross-domain content overlap.The build pipeline produces JSONL files for default, web, and aiml configurations.Source: §4.7 · p. 15

4.8 Data Splits and Leakage Prevention

  • 0% of the corrected web test set is contaminated by near-duplicates after family-based splitting; AI/ML examples sharing a technique have no near-duplicate pairs above Jaccard 0.8.The original web test set had 11.6% contamination despite CVE-aware splitting; corrected splits group near-duplicate families and stratify by category and language.Source: §4.8 · pp. 15–16

5 Quality Assessment

Automated checks and rubric scoring report high structural compliance and AI/ML quality, while the 2026 semantic audit found grounding, fix-correctness, and split-leakage problems that structural validation could not detect.

5.1 Web Security Quality

  • 100% of web examples passed structural checks after compliance rose from 47.2%, but those checks did not establish semantic accuracy.Remediation addressed 604 issues across five fix categories.Source: §5.1 · p. 16, Figure 8 · p. 16

5.2 AI/ML Security Quality

  • 93.8/100 was the AI/ML mean quality score (σ = 0.93), with 97.8% of files scoring 93–95.The five-tier rubric weights correctness, security, grounding, educational value, and production readiness.Source: §5.2 · p. 17, Table 6 · p. 17, Figure 9 · p. 17

5.3 Quality Score Computation and Calibration

  • All 750 AI/ML files met the production threshold of 92; score aggregation summed five dimensions while reviewer judgment, not agent scores, determined deductions.On a 50-file audit, 47 assigned scores matched within ±1 point; correction shifted the mean by less than 0.05 points.Source: §5.3 · pp. 18–19, Table 7 · p. 19

5.5 Reference Quality

  • The 750-example AI/ML generation set contained 2,828 structured references, averaging 3.8 per file; research papers and vendor advisories made up 63.4% combined.The audit cautions that many free-text incident narratives were representative scenarios, so the paper does not claim uniform Tier-2 grounding.Source: §5.5 · p. 20, Figure 11 · p. 20

5.6 Grounding and Fix-Correctness Audit (2026 Revision)

  • Of 1,379 web references checked, most failed verification: 802 misattributed CVEs were removed and 810 incident narratives corrected or relabeled.Twenty-one removed CVEs were replaced with the correct CVE; 378 narratives were rewritten to the real event and 432 relabeled as representative examples.Source: §5.6 · p. 21
  • Of 747 AI/ML examples, 613 incident-like narratives were corrected; after audit, roughly 18% cite a documented incident or verified CVE.Only 14 were tied to the real incident described; the remainder were labeled representative scenarios.Source: §5.6 · p. 21
  • The audit removed 28 defective examples and rebuilt family-grouped splits, reducing web test-set contamination from 11.6% to zero.The removed defects included mislabeled or still-exploitable fixes; near-duplicate families were kept within a single split.Source: §5.6 · p. 21

6 Fine-Tuned Models

The paper fine-tuned eight open-source code models on SecureCode using QLoRA, spanning five architecture families and parameter sizes from 3B to 20B.

6 Fine-Tuned Models

  • Eight open-source code models were fine-tuned with QLoRA 4-bit quantization and LoRA adapters on a single NVIDIA A100 40GB GPU.Training used the February 2026 release of 2,185 examples; retraining on the audited release was in progress.Source: §6 · p. 22
  • Table 10 lists the fine-tuned model collection, with all models hosted on Hugging Face.The supplied table caption provides no individual model results.Source: Table 10 · p. 22
  • The collection spans five architecture families and model sizes from 3B to 20B parameters.Families include Llama, Qwen, DeepSeek, Gemma, StarCoder2, and Granite.Source: §6 · p. 22

7 Evaluation

The evaluation is designed to test whether SecureCode fine-tuning shifts security-relevant code generation, using held-out and external prompts, four security metrics, and planned ablations. Model training was still in progress at submission.

7.1 Evaluation Design

  • The evaluation holds out 219 examples and adds 100 external security prompts to test generalization beyond training scenarios.The held-out set is category-stratified, with CVE-aware web splitting and near-duplicate checks; external prompts cover 50 web and 50 AI/ML scenarios.Source: §7.1 · p. 23

7.2 Security-Specific Metrics

  • Four metrics assess vulnerability patterns, expected security controls, defense-in-depth layers, and retention of operational security guidance across turns.Vulnerability-pattern presence is lower-is-better; control inclusion is the fraction of expected controls present, and multi-turn retention checks Turn 4 guidance.Source: §7.2 · p. 23

7.3 Evaluation Protocol

  • Eight fine-tuned models and their base counterparts are evaluated with reproducible greedy decoding, automated scoring, and a 50-response human-validation subset per model.Results are planned by domain and for cross-domain generalization.Source: §7.3 · pp. 23–24

7.4 Planned Ablations

  • Three planned ablations compare unified versus domain-only data, four-turn versus two-turn training, and dataset sizes from 25% to 100%.Training was in progress at submission; full evaluation and ablation results were deferred to a companion report.Source: §7.4 · p. 24

8 Discussion

SecureCode combines web and AI/ML security coverage in one conversational dataset, with findings on review and remediation alongside limitations in coverage and pending model evaluation.

8.1 Key Findings

  • The unified dataset addresses the gap between web and AI/ML security by covering both attack surfaces under one schema.Seven specialist perspectives and more than 10,500 assessments contributed to review; the paper also identifies content-based classification and multi-phase remediation as key practices.Source: §8.1 · p. 24

8.2 Dataset Family Architecture

  • The unified Hugging Face dataset is the recommended entry point, with default, web, and aiml configurations for filtering its 2,372 examples.Figure 12 compares domain composition and severity, and highlights complementary language and framework coverage.Source: §8.2 · p. 25, Figure 12 · p. 25

8.3 Limitations

  • AI/ML examples are 90.7% Python, limiting coverage for Go, Rust, and C# AI deployments; web examples cover 11 languages but concentrate on popular web languages.AI/ML framework APIs may also require updates as frameworks evolve, and LLM-generated code may miss subtle biases or edge cases.Source: §8.3 · p. 25
  • The dataset covers application-layer security, not model-training infrastructure, GPU-cluster hardening, or hardware-level attacks.AI/ML quality scores are assigned during generation and review rather than by independent external evaluators; the web component has no quality scores.Source: §8.3 · pp. 25–26
  • Quantitative comparisons of fine-tuned models against baselines were pending, limiting the paper's ability to demonstrate a direct dataset-to-outcome causal link.The evaluation framework and ablation designs were specified, but results were deferred to a companion report.Source: §8.3 · p. 26
  • The dataset includes vulnerable code and attack explanations, with defensive framing and secure alternatives, but the paper acknowledges potential dual-use risk.Mitigations include grounding attack patterns in published research and CVEs and a license that constrains redistribution.Source: §8.3 · p. 26
  • Not all code examples were executed in live environments; framework API correctness was checked against documentation, and held-out evaluation includes execution-based validation.The paper notes that static review may miss subtle runtime errors.Source: §8.3 · p. 26

8.4 Future Work

  • Future work includes controlled fine-tuning evaluation, a standardized benchmark, expanded agentic AI and language coverage, multi-file examples, and continuous updates.Source: §8.4 · p. 26

9 Conclusion

SecureCode addresses the need for assistants that understand both classical and AI/ML-specific vulnerabilities with a unified training dataset. Its audited scope, conversational design, quality pipeline, and open-source release are presented as production-grade resources.

9 Conclusion

  • 45% of AI-generated code in security-relevant contexts contains vulnerabilities, while AI/ML applications face threats traditional security training does not address.The conclusion says organizations need assistants that understand both classical and AI-specific vulnerability classes.Source: §9 · p. 26
  • SecureCode provides a unified, production-grade dataset of 2,372 audited examples spanning 20 vulnerability categories across 12+ languages and 49+ frameworks.It covers the OWASP Top 10 2021 for web security and the OWASP LLM Top 10 2025 for AI/ML security.Source: §9 · pp. 26–27
  • Its four-turn conversational structure, multi-agent quality pipeline, and eight-phase remediation process are designed to support security training.The paper also describes eight fine-tuned open-source models ranging from 3B to 20B parameters.Source: §9 · p. 27
  • SecureCode, component datasets, validation tools, training scripts, and fine-tuned models are released as open-source artifacts.The stated intended uses include research reproducibility, improving AI coding assistants, and teaching secure coding through real-world incidents.Source: §9 · p. 27

Availability

The dataset and fine-tuned models are released alongside training scripts, under distinct licensing terms. Web and AI/ML examples share a unified conversation format while retaining domain-specific metadata.

Availability

  • The unified Hugging Face release provides separate web-security and AI/ML-security datasets, plus a collection of eight fine-tuned models and training scripts in GitHub repositories.The dataset is released under CC BY-NC-SA 4.0; fine-tuned models use their respective base-model licenses.Source: Availability · p. 27
  • CC BY-NC-SA 4.0 permits non-commercial research, education, and internal corporate security training, while commercial integration into paid products requires contacting the authors.The stated rationale is to constrain commercial offensive-toolkit use, keep derivatives open under equivalent terms, and support provenance through attribution.Source: Availability · p. 27

A Unified Schema

  • Web and AI/ML examples share a unified conversation format, with domain-specific metadata preserved in each example.The simplified examples show fields such as security category, subcategory, severity, language, and vulnerability references.Source: A Unified Schema · p. 29

B CWE Coverage (AI/ML Component)

The AI/ML component maps to 92 unique CWE identifiers.

B CWE Coverage (AI/ML Component)

  • The AI/ML component maps to 92 unique CWE identifiers.The passage introduces a list of the ten most frequent identifiers but does not provide their values in this request's evidence.Source: B CWE Coverage (AI/ML Component) · p. 29

C Framework Coverage (AI/ML Component)

Framework usage is summarized approximately across the 750 AI/ML examples, with examples sometimes using multiple frameworks.

C Framework Coverage (AI/ML Component)

  • Approximate framework usage is reported across 750 AI/ML examples, and many examples use multiple frameworks.The supplied passages do not include framework-specific counts.Source: C Framework Coverage (AI/ML Component) · p. 29
  • The accompanying frequency list gives CWE-200 Information Exposure as 103 and CWE-345 Insufficient Verification of Data Authenticity as 57.It also lists CWE-74 and CWE-94 at 42 each, CWE-400 at 39, CWE-506 at 29, CWE-269 at 27, CWE-502 and CWE-79 at 22 each, and CWE-770 at 19.Source: C Framework Coverage (AI/ML Component) · p. 30
Loading 2512.18542v3…