PaperlayerView exact revision on arXiv

Source-linked AI summary

AuraForge: Scaling Security Supervision for Training Coding Agents

Danqing Wang, Songwen Zhao, Harsh Sharma, Jierui Wang, Andre Vicente Duarte, Ivan Bercovich, Lei Li

arXiv:2610.00850v1cs.CRcs.CLcs.CY

TL;DR

Coding agents can meet explicit feature requests while violating implicit security requirements, and scaling executable security supervision from real repositories remains challenging. AuraForge synthesizes attack-oriented tests and constructs tasks with language support and retrieval safeguards; training with synthesized tests yields larger functional and security improvements than training with human-written tests.

  • Problem

    Coding agents can satisfy requested functionality while producing vulnerable implementations, and reliable executable security supervision is difficult to obtain at scale from real-world repositories.

    Source: §1 · p. 2

  • Method

    AuraForge synthesizes and validates attacker-oriented security tests, constructs language-extensible executable tasks, and safeguards environments against reference retrieval.

    Source: §1 · p. 2, §3.2 · p. 4, §3.2.1 · p. 4

  • Results

    Averaged over both benchmarks, AuraGym improves FuncPass and SecPass by 19.7 and 6.2 points, compared with 14.9 and 4.4 points for AuraGymh.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    AuraForge provides more diverse and reliable security supervision for training secure coding agents than human-written tests alone.

    Source: §1 · p. 2, §6 · p. 8

    The study focuses on repository-level feature implementation with implicit security requirements.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Coding agents are now proficient enough to generate complex software applications from a single prompt. As their capabilities have grown, human oversight has increasingly shifted from line-by-line code review toward hands-off evaluation of outcomes. However, recent studies have shown that such a transition exposes a critical risk: functional correctness alone does not guarantee a secure implementation. Despite growing attention to code security, training safer coding agents remains challenging because reliable security supervision is difficult to obtain at scale from real-world repositories. We introduce AuraForge to synthesize and validate executable security tests for training secure coding agents. Our approach combines attack-oriented test synthesis, language-extensible task construction, and safeguards against reward hacking. Using AuraForge, we construct AuraGym, a multi-language and multi-CWE executable training gym: 679 executable feature-implementation tasks from 344 real-world repositories across Python, JavaScript, and TypeScript, covering 177 CWE categories. On the subset with human-written security tests, AuraForge produces about 3 times as many test cases on average and reduces the false-positive rate by 83.23%, allowing alternative secure implementations to receive correct supervision. Training Qwen3.5-4B with synthesized security tests gains larger improvements than human-written security tests (average 19.7 FuncPass and 6.2 SecPass vs. 14.9 FuncPass and 4.4 SecPass) on three languages. These results demonstrate that AuraForge provides more diverse and reliable security supervision to train secure coding agents.

1 Introduction

Coding agents can satisfy explicit feature requests while violating implicit security requirements, and executable security supervision is difficult to scale from real repositories. AuraForge addresses this by synthesizing security tests and constructing executable tasks with safeguards against reward hacking.

Motivation

  • Training safer agents requires security supervision at scale because functional correctness does not ensure security, and human review cannot reliably catch implicit security violations.Source: §1 · p. 2

Challenges

  • Useful security oracles must detect the target vulnerability, accept secure alternatives, and execute reliably, while task masking preserves needed interfaces and tests the agent’s modified source.Source: §1 · p. 2

Main results

  • 19.7 and 6.2 percentage-point gains in functional and security pass@1 with synthesized tests exceed the 14.9 and 4.4-point gains with human-written tests across three languages.The comparison uses Qwen3.5-4B trained with mini-swe-agent.Source: §1 · p. 2

Contributions

  • AuraGym contains 679 executable coding tasks from 344 repositories across Python, JavaScript, and TypeScript, covering 177 CWE categories.Source: §1 · p. 2

2 Related Work

Prior executable coding-agent datasets mainly provide functional supervision, while security benchmarks evaluate security across several task settings. AuraForge positions its contribution as scalable security-test supervision for training, distinct from evaluation-focused benchmarks and vulnerability-lifecycle tasks.

Executable Training Data for Coding Agents

  • Executable training datasets primarily supervise functional issue resolution, while AuraForge focuses on security tests as training signals for secure coding agents.Source: §2 · pp. 2–3

Secure Coding Benchmark

  • Security benchmarks pair functionality checks with exploits or analysis across backend generation, repository edits, and feature implementation; AuraForge instead targets scalable training supervision for real-world repositories.Source: §2 · p. 3

Vulnerability Discovery, Exploitation, and Repair

  • Vulnerability-lifecycle benchmarks evaluate discovery, exploit generation, or repair, whereas AuraForge addresses the data bottleneck by producing security tests as training rewards.Source: §2 · p. 3

3 AuraForge: Forging Executable Security Tasks at Scale

AuraForge formulates repository-level feature implementation with implicit security requirements as an executable training task. It combines attacker-oriented test synthesis, language adapters, execution validation, and safeguards against retrieving reference implementations.

3.1 Problem Formulation

  • Agents receive a repository with target functionality removed and must meet explicit feature requests and implicit security requirements; passing tests is evidence only within their scope.Source: §3.1 · pp. 3–4

3.2 Synthesizing Reliable Security Tests

  • Given a repository, feature request, functional tests, environment, and construction metadata, AuraForge synthesizes security tests from the attacker’s perspective and validates their reliability and discrimination.Source: §3.2 · p. 4

3.2.1 Attack-Oriented Synthesis

  • Synthesis identifies security invariants and attack variants, then asserts that each attack’s observable effect is prevented rather than requiring a particular fix location or mechanism.Source: §3.2.1 · p. 4

3.2.2 Execution-based validation.

  • Test suites avoid external dependencies and order effects, and are independently run in fresh environments; acceptance requires error-free runs that distinguish vulnerable from fixed implementations.Source: §3.2.2 · p. 4

3.3 Language-Extensible Construction

  • A shared, language-agnostic core delegates source masking, environment setup, and outcome parsing to adapters, so new languages need adapter components rather than core-workflow changes.Source: §3.3 · pp. 4–5, Figure 2 · p. 5

3.4 Preserving Environment Integrity

  • Retrieval threatens task validity: it occurred in 16% of 5,174 runs, so AuraForge removes repository history and local implementation copies and blocks network and package-manager access.Source: §3.4 · p. 5

4 AuraGym: A Multi-Language, Multi-CWE Security Gym

AuraGym scales repository-derived security tasks across languages and vulnerability categories. Compared with the human-written-test subset, its synthesized tests provide denser security supervision.

Dataset construction

  • 679 instances from 344 repositories span 177 CWE categories, with an additional repository-disjoint validation set for the new TypeScript/JavaScript SusVibestsjs.Source: §4 · p. 6

Comparison with existing datasets

  • AuraGym has the most instances among the compared secure-coding benchmarks and covers over 3× as many repositories and 2× as many CWE categories as SusVibes.Source: §4 · p. 6

Comparison with human-written tests

  • 3× as many security tests per instance on average are provided by AuraGym compared with AuraGymh, across Python, JavaScript, and TypeScript.Source: §4 · p. 6, Figure 4 · p. 6

5 Experiments and Analysis

The experiments compare synthesized security tests with human-written tests, then evaluate how training on AuraGym affects functional and security performance. Synthesized tests retain comparable vulnerability recall with fewer false positives, and synthesized supervision produces larger training gains than human-written supervision.

5 Experiments and Analysis

  • The study uses agent trajectories to compare test verdicts and fine-tunes Qwen3.5-4B on functionally correct, secure trajectories.Trajectory collection used two models and generic-reminder or task-specific-hint prompts; evaluation reports FuncPass and SecPass on SusVibes and TS/JS.Source: §5 · p. 6, §5.2 · p. 8

5.1 RQ1: Synthesized Tests Match Human Recall, 83% Fewer False Positives

  • Synthesized tests reject 98.6% of vulnerable solutions versus 98.3% for human-written tests, with recall within two percentage points in every language.Among synthesized-suite misses, 11 of 25 were one-point tests and 14 of 25 were vacuous; none resulted from an untested code path.Source: §5.1 · p. 7, Figure 5 · p. 7, Table 2 · p. 7
  • Synthesized suites reject 2.8% of secure solutions versus 16.7% for human-written suites, an 83% reduction in false positives.Most human-suite false positives were fix-bound: 68 versus 4 for synthesized suites.Source: §5.1 · p. 7, Figure 5 · p. 7, Table 2 · p. 7

5.2 RQ2: Synthesized Signals Outperform Human-Written Tests for Training

  • Across both benchmarks, AuraGym synthesized supervision improves FuncPass and SecPass by 19.7 and 6.2 points on average, versus 14.9 and 4.4 points for AuraGymh trained with human-written security tests.Source: §5.2 · p. 8

6 Conclusion

AuraForge turns real-world vulnerability fixes into executable training tasks with attack-oriented tests, language-extensible construction, and safeguards against reward hacking. The conclusion reports a 679-task, three-language AuraGym spanning 177 CWE categories, while the appendix describes task construction and validation details.

6 Conclusion

  • AuraForge produces 679 tasks across three languages and 177 CWE categories; its tests match human-written recall while reducing false positives by 83%.Training on AuraGym improves functional and security performance beyond human-written supervision.Source: §6 · p. 8

AI use statement

  • LLMs assisted with writing refinement and figures and tables; authors reviewed all assisted content and take responsibility for the final paper.They were not used to formulate mathematical claims, develop or write proofs, or translate research materials.Source: AI use statement · p. 9

Reproducibility statement

  • The reproducibility materials point to Section 3 and the appendix for task construction, security-test synthesis, environment safeguards, data sources, filtering, and fine-tuning details.Experiments specify the functional and security evaluations used to assess supervision quality and agent performance.Source: Reproducibility statement · p. 9

A Language Adaptation and Task-Construction Details

  • The shared construction workflow adapts source transformation, environment preparation, and test execution to language and repository requirements, illustrated with JavaScript and TypeScript.Language-specific masking accounts for declarations, exports, and type annotations while removing implementation bodies and initializers.Source: A Language Adaptation and… · p. 11, §A.1 · p. 11

A.1 Patch Separation and Feature Masking

  • Patch separation uses file paths and, for ambiguous edits, LLM classification; masking removes the target implementation while preserving surrounding interfaces and relevant security protections.A shared verifier checks the generated request and rejects masks that introduce unrelated changes.Source: §A.1 · p. 11

A.2 Runtime, Dependency, and Build Configuration

  • Environment construction determines runtime, dependencies, and commands from repository configuration, then rebuilds generated outputs to keep tests aligned with modified source.Stale artifacts are removed before validation when tests consume generated outputs.Source: §A.2 · p. 11

A.3 Test Execution and Result Normalization

  • Repository-specific adapters run native test workflows and normalize results, distinguishing completed test outcomes from execution errors or incomplete runs.An error printed by a test alone does not establish test failure.Source: §A.3 · p. 11

A.4 Execution-Based Task Validation; A.5 JavaScript and TypeScript Collection and Filtering

  • Task validation compares fixed and vulnerable implementations and checks whether functional tests detect feature removal; JS/TS candidate filtering yields 863 candidates for construction.The final pool comprises 249 MoreFixes and 614 GHSA candidates, with filtering rules described for each source.Source: §A.4 · p. 11, §A.5 · pp. 11–12

A.6 TypeScript/JavaScript Training and Test Split

The TS/JS split is repository-disjoint and emphasizes recent vulnerabilities, but it is not strictly chronological. The security-test procedure derives attack properties from public records, generates and validates tests in the task environment, and drops suites that fail reliability requirements.

A.6 TypeScript/JavaScript Training and Test Split

  • The 82-instance test set is the proven minimum satisfying repository disjointness and coverage of all 72 labels, but the split is not strictly chronological.It emphasizes recent vulnerabilities, yet training also includes records from 2026.Source: §A.6 · p. 12

B Security-Test Synthesis Details

  • Stage 1 derives a structured security property from public vulnerability records, specifying attack variants and observable effects while excluding implementation-specific defenses.The property also identifies security-irrelevant differences that tests must not assert.Source: B Security-Test Synthesis Details · p. 13
  • Stage 2 writes tests in the vulnerable repository environment to cover attack variants and assert observable effects rather than a particular defense.The historical fix is used only to check that tests fail on the vulnerable state and pass on the fixed state.Source: B Security-Test Synthesis Details · p. 13
  • Tests must avoid nondeterminism and infrastructure errors; suites that cannot meet these reliability rules are dropped.Acceptance also requires completed runs on historical vulnerable and fixed implementations, with at least one test failing on vulnerable and passing on fixed.Source: B Security-Test Synthesis Details · p. 13

C Environment Integrity: Retrieval Audit and Sanitization

AuraForge audits whether an agent retrieved the masked reference implementation and sanitizes task images that expose it. Images are revalidated after removal, and tasks that still expose the fix are excluded.

C Environment Integrity: Retrieval Audit and Sanitization

  • Retrieval detection combines a high-recall scan of tool calls with an isolated language-model review of flagged trajectories to determine whether the reference was obtained and how.The first route by which reference content enters a trajectory defines its retrieval channel.Source: C Environment Integrity: Retrieval… · pp. 13–14, Table 6 · p. 13
  • Because images are built from the fixed repository state before the work tree is rolled back and masked, fixed code can remain in generated outputs, installed copies, caches, or compiled artifacts.Sanitization removes the project’s own build output or installed copy, but leaves copies belonging to dependencies.Source: C Environment Integrity: Retrieval… · pp. 13–14
  • After sanitization, tasks are revalidated for security-test behavior, functional baseline, and unchanged test command; tasks whose images still expose the fix are excluded.An artifact stays if removing it would prevent tests from running.Source: C Environment Integrity: Retrieval… · p. 14

D More Details on Existing Agentic Coding Datasets

Existing coding datasets and benchmarks span general software engineering, backend generation, and repository-based security evaluation, but their task constructions and vulnerability coverage differ.

D More Details on Existing Agentic Coding Datasets

  • Dataset counts are not directly comparable because tasks, repository counts, and CWE annotations reflect different construction and annotation choices.The paper distinguishes tasks from interaction trajectories and repositories from vulnerability categories.Source: D More Details on Existing Agentic… · p. 14
  • General SWE resources provide executable issue-resolution tasks or synthetic bugs, with functional feedback rather than supervision organized around vulnerability categories.Examples include SWE-Gym, SWE-smith, SWE-rebench, and multilingual SWE-rebench V2.Source: D More Details on Existing Agentic… · p. 14
  • Backend-generation benchmarks evaluate functionality and security across framework-instantiated scenarios, with CWE reporting and task counts that include framework-specific versions.AutoBaxBench excludes CWE-400 from its main evaluation.Source: D More Details on Existing Agentic… · p. 14
  • Repository-based security benchmarks range from C/C++ secure completion and implementation to feature tasks with implicit security requirements.SusVibes evaluates feature implementation where agents must meet functional requirements while avoiding vulnerabilities absent from the request.Source: D More Details on Existing Agentic… · p. 14

E AuraGym Trajectory Collection

AuraGym trajectories are collected with secure solutions, using security hints during collection but removing them before training. The appendix also describes the synthesized-versus-human test comparison and its task statistics.

E AuraGym Trajectory Collection

  • Security hints based on each instance’s CWE or CVE guide collection toward secure implementations, then are removed so training prompts retain the original feature request.The original request is denoted d.Source: E AuraGym Trajectory Collection · p. 14
  • Command filtering blocks online search and code retrieval, and only trajectories passing both functional and security tests are retained.Blocked operations include git clone, git remote, and package downloads; mini-swe-agent avoids scaffold packages in the solving container.Source: E AuraGym Trajectory Collection · p. 15
  • 3,753 Muse trajectories and 2,218 DeepSeek v4.1 Flash trajectories remain after excluding empty patches and hacked behaviors.Fine-tuning uses only trajectories passing functional and security tests; the reported setup uses 64K context, batch size 8, 3 epochs, and peak learning rate 1e-5.Source: E AuraGym Trajectory Collection · p. 15

F Comparing Synthesized and Human-Written Security Tests

  • The comparison appendix covers solution collection, label verification, error categories, vulnerability-family splits, and comparison limits.Table 7 reports functional tests, security tests, and target files as per-instance averages; mixed JavaScript/TypeScript repositories count under both languages.Source: F Comparing Synthesized and… · p. 15, Table 7 · p. 15

F.1 Tasks and Solutions

The comparison evaluates both test suites on the same functionally passing solutions, with exclusions driven mainly by repository functional-test flakiness.

F.1 Tasks and Solutions

  • Of 3,190 collected trajectories on 434 tasks, 2,533 solutions on 351 tasks entered comparison after both suites completed and repository functional tests passed under both gradings.The same patch on the same task counts once.Source: §F.1 · p. 15
  • The largest exclusion was functional-suite flakiness: 351 trajectories failed the repository’s functional suite under one of the two gradings.The comparison also dropped 37 recorded passes contradicted by the runner logs.Source: §F.1 · p. 15

F.2 Labels

Synthesized tests have a substantially lower pooled false-positive rate than human-written tests, while adjudication and audits expose shared and suite-specific failure patterns. The evidence also varies by vulnerability family and includes five illustrative disagreements.

F.2 Labels

  • The suites agree on 2,271 of 2,450 solutions, and the shared verdict becomes the label, leaving errors missed by both suites unobservable.Recall is therefore measured against what at least one suite catches.Source: §F.2 · p. 15
  • A suite-blinded judge produced 190 rulings on 88 tasks after attacking solutions in-container and running both suites; hand verification read 13 medium- and 20 high-confidence rulings, overturning two.All rates in Section 5.1 use verified labels.Source: §F.2 · p. 16
  • 16.7% [12.0, 22.0] vs. 2.8% [1.0, 5.0] pooled false-positive rate for human and synthesized suites, respectively; recall is 98.3% [97.2, 99.2] vs. 98.6% [97.7, 99.4].The rates use 2,450 solutions, of which 635 were labelled secure and 1,815 vulnerable.Source: §F.2 · p. 16

F.3 Why Each Suite Was Wrong

  • Fix-bound assertions, one-point tests, and mock-based vacuous checks share an implementation-specific root cause, while scope errors affect both suites.Nine of the synthesized suites’ 14 vacuous checks occur in one repository; timing and memory thresholds account for 9 of 18 synthesized false positives and fail the historical fix on some runs.Source: §F.3 · p. 17

F.4 Assertion Audit

  • 87% vs. 48% of tasks have synthesized vs. human suites asserting absent attack effects; on 82 shared effect-asserting tasks, secure-solution FPR is 3.1% vs. 12.7%.The audit classifies suites by whether tests assert that an attack’s effect is absent or reproduce the historical fix’s outcome.Source: §F.4 · p. 17, Table 10 · p. 19

F.5 By Vulnerability Family

  • 13 of 25 synthesized misses cluster in path traversal and SSRF, while both suites miss resource-exhaustion cases; family counts are not rates to rank.The table caption notes that most secure-side denominators are under 100.Source: §F.5 · p. 17, Table 11 · p. 19

F.6 Case Studies

  • Five adjudicated cases illustrate three human false positives and two human false negatives across Python, TypeScript, and JavaScript.The cases present vulnerability requirements, suite code, agent solutions, and execution evidence.Source: §F.6 · pp. 17, 19

F.6.1 Keras: a model file that runs code on load (CVE-2025-49655, Python)

The section examines how synthesized security tests specify required protections across argument positions and nesting depths without depending on the historical fix’s exact implementation or wording. The defu example shows that testing only an advisory’s attack position can miss prototype pollution through defaults, and notes limits in evaluating suite agreement and disagreements.

F.6.1 Keras: a model file that runs code on load (CVE-2025-49655, Python)

  • Two Keras solutions prevent payload execution in safe mode but fail the human test’s exact exception-message assertion; the synthesized suite checks whether the payload ran and passes both.The secure solutions refuse the layer with different wording or use a restricted unpickler, showing that safe behavior need not match the historical fix’s surface.Source: §F.6.1 · pp. 19–20

F.6.2 markdownify: a heading tag that allocates gigabytes (CVE-2025-46656, Python)

  • All four markdownify solutions keep out-of-range heading tags from producing huge output, but fail the human test requiring <h10> to render as <h6>; the synthesized suite passes them.The security property is bounded output regardless of whether an out-of-range heading is clamped, dropped, or emitted as text.Source: §F.6.2 · pp. 20–21

F.6.3 NocoBase: SQL injection through record keys (CVE-2026-41640, TypeScript)

  • All four NocoBase solutions block crafted keys from exposing other-table data, yet fail human tests that require the historical SQL text; the synthesized effect test passes them.The solutions parameterize values or delegate query construction to the ORM, while the synthesized test checks whether a crafted key can retrieve a secret.Source: §F.6.3 · pp. 21–22

F.6.4 Sentry: account takeover through single sign-on (CVE-2025-22146, Python)

  • All four Sentry solutions pass the human test’s anonymous-attack scenario but allow a logged-in attacker to link the victim’s identity, which the synthesized suite detects.Secure confirmation requires the session user to own the account identified by the SSO email, not merely to be authenticated.Source: §F.6.4 · pp. 22–23

F.6.5 defu: prototype pollution through the defaults argument (CVE-2026-35209, JavaScript)

  • For defu, no argument at any position or nesting depth should make the returned object carry a property the caller did not provide.The vulnerability arises because Object.assign({}, defaults) can trigger prototype pollution; the described fix uses { ...defaults } to copy the key as an own property.Source: §F.6.5 · p. 23
  • The synthesized suite exposed a defaults-position bypass: s1 passed the human suite but failed 8 of 37 synthesized tests, with the prototype replaced.The 37 tests varied where a proto-carrying object entered the API; unlike the human suite’s first-argument example, the synthesized tests also covered defaults and checked Object.prototype.Source: §F.6.5 · pp. 23–24

F.7 Limits

  • Evaluation limits include treating suite agreement as truth, leaving 35 disagreements unadjudicated, and having small secure-side denominators for most language-family groups.Two agents supplied solutions; a third was proposed to test solver dependence. No planted known-answer cases were run on these sets, though the judge prompt and model had been calibrated on 17 separate cases, getting 16 correct.Source: §F.7 · p. 24
Loading 2610.00850v1…