Source-linked AI summary
CyberFactory: Scaling Cyber Security Capabilities with Instances from the Wild
Jian Yang, Haau-Sing Li, Shawn Guo, Zixi Zhao, Yibo Tan, Jiajun Wu, Aishan Liu, Xianglong Liu, Tianyu Zheng, Bryan Dai, Chengran Yang
TL;DR
CyberFactory targets the lack of unified, reproducible open-source methods and scalable agentic data for training cybersecurity models. It converts vulnerability artifacts into verified multi-task instances and uses a reusable analysis skill to guide trajectories whose procedure is internalized by OpenAegis. Under a one-hour CyberGym budget, OpenAegis reaches 52.4% Pass@1, improving 22.8 points over Qwen 3.5 and exceeding evaluated larger baselines.
Problem
Existing open-source cybersecurity efforts are fragmented, lack reproducible training solutions and scalable agentic data, and require strong domain priors for scaling rollouts.
Method
CyberFactory connects data construction, trajectory synthesis, and model training across vulnerability detection, patch generation, and CyberQA using executable instances and skill-guided agentic supervision.
Results
52.4% Pass@1: OpenAegis improves 22.8 points over Qwen 3.5 and exceeds GLM 5.2 and Kimi K2.7 under the same one-hour CyberGym scaffold.
Takeaways & Limitations
OpenAegis internalizes the prior-guided vulnerability-analysis procedure without requiring the skill at inference time.
Takeaways & Limitations
The evaluation is bounded by available CVE artifacts, benchmark coverage, and a fixed one-hour execution budget, with some targets still benefiting from manual input construction.
Abstract
from arXiv · showhide
As large language models (LLMs) continue to advance in coding capabilities, their potential in cybersecurity has drawn increasing research attention, with closed-source LLMs (e.g., Mythos) delivering advanced cybersecurity capabilities. However, existing open-source efforts remain limited: frontier open-weight models do not provide reproducible cybersecurity training solutions, open-source training solutions focus on isolated tasks and lack scalable agentic data, and scaling agentic rollouts requires strong domain priors. In this work, we introduce \textbf{CyberFactory}, a unified open-source framework that connects data construction, trajectory synthesis, and model training across proof-of-concept (PoC) generation, vulnerability patching, and cybersecurity question answering (CyberQA). CyberFactory transforms public vulnerability artifacts, including CVEs from the wild, into executable and verifiable task instances. It further uses a reusable vulnerability-analysis skill to guide the teacher through source inspection, problem solving with domain prior, and evidence-based validation. The resulting supervision is agentic: the model interacts with tools and target environments and revises its solutions according to execution feedback. Using these trajectories, we train and release \modelname\footnote{\emph{Aegis} is, in Greek mythology, the protective shield of Zeus and Athena; the name reflects the model's defensive, security-oriented purpose.}, which internalizes the skill-guided procedure without requiring the skill at inference time. On CyberGym, \modelname reaches 52.4% Pass@1 under a one-hour budget, improving over its Qwen~3.5 base model by +22.8 points and outperforming the evaluated general-purpose backbones under the same scaffold.
1 INTRODUCTION
CyberFactory addresses fragmented and non-reproducible open-source cybersecurity training by connecting vulnerability evidence, agentic trajectory synthesis, and model training. It trains OpenAegis to internalize a reusable vulnerability-analysis workflow, achieving strong CyberGym performance under a one-hour budget.
- Open-source cybersecurity efforts provide isolated capabilities, lack scalable agentic training data, and require strong domain priors for rollout scaling.
- CyberFactory connects data construction, trajectory synthesis, and model training across vulnerability detection, patch generation, and CyberQA.The framework is presented as a unified, open-source training recipe.
- CyberFactory converts public vulnerability artifacts into executable, verifiable task instances and uses a reusable vulnerability-analysis skill to guide tool-interactive trajectory synthesis.The skill combines source inspection, domain-guided problem solving, evidence-based validation, and execution feedback.
- 52.4% Pass@1: OpenAegis improves over its Qwen 3.5 base model by 22.8 percentage points under the one-hour CyberGym budget.
- OpenAegis internalizes the skill-induced workflow without requiring the skill at inference time and outperforms substantially larger baselines under the same scaffold.
2 RELATED WORK
Prior work develops cybersecurity benchmarks, reproducible vulnerability datasets, executable software-agent environments, vulnerability repair systems, and cybersecurity QA benchmarks. CyberFactory connects these previously separate forms of supervision with executable PoC construction in one training pipeline.
- Benchmarking cybersecurity capabilities of LLMs: Cybersecurity benchmarks progress from professional CTF tasks toward executable evaluations spanning vulnerability discovery, PoC generation, and patch generation.
- Reproducible vulnerability datasets: Reproducible vulnerability datasets link CVEs to vulnerable code, fixes, reachability information, and triggering oracles.
- LLMs for real-world software tasks: Software-task research studies repository editing, interleaved reasoning and environment actions, agent-computer interfaces, and executable tool environments.
- Vulnerability repair and cybersecurity question answering: Neural vulnerability repair and cybersecurity QA are typically developed as separate supervision and evaluation settings.
- Vulnerability repair and cybersecurity question answering: CyberFactory connects vulnerability repair, cybersecurity QA, and executable PoC construction within a single multi-task training pipeline.
3 METHODOLOGY
CyberFactory synthesizes multi-task cybersecurity data from executable vulnerability instances, trusted evidence, and skill-guided agentic trajectories. Its workflow validates outputs against build differences, source evidence, and task-specific checks while transferring the analysis procedure into OpenAegis.
- The methodology spans vulnerability detection, vulnerability fixing, and cybersecurity question answering within an end-to-end workflow.
- PoC construction: A PoC is validated differentially by triggering the target behavior in the pre-patch build but not in the post-patch build.
- PoC construction: PoC instances draw from ARVO, OSS-Fuzz, and wild CVEs, with wild CVEs presenting the greatest instance-creation difficulty.
- Instance descriptions: Vulnerability descriptions combine fix-commit quality classification with vulnerability-specific evidence and the corresponding fix commit.
- Question answering: QA samples derive answers from execution results, structural facts, or authoritative reports, then undergo traceability, leakage, and uniqueness checks.
- Skill-guided trajectory synthesis: The reusable vulnerability-analysis skill guides target inspection, exploratory testing, evidence validation, and revision, while supervised fine-tuning transfers the procedure into model parameters.
4 EXPERIMENTS
CyberGym evaluates vulnerability reproduction with a differential oracle under a shared one-hour budget, and OpenAegis achieves the strongest reported performance while avoiding inference-time skill guidance. Additional analyses examine context management, skill effects, and behavioral internalization.
- 4.1 TASK AND EVALUATION PROTOCOL: A task is solved when a candidate input triggers the target sanitizer failure on the pre-patch build but not the post-patch build.This differential criterion provides a programmatic signal that excludes incidental crashes.
- 4.3 BASELINES: All evaluated models use the same scaffold, tools, oracle, submission logic, and one-hour wall-clock budget, without the vulnerability-analysis skill at evaluation time.The skill is used only to synthesize OpenAegis training trajectories.
- 4.4 MAIN RESULTS: 52.4% Pass@1: OpenAegis solves the highest fraction of CyberGym tasks under the one-hour per-task budget.Pass@1 is the fraction of tasks whose submitted input satisfies the differential oracle.
- 4.4 MAIN RESULTS: OpenAegis improves Pass@1 over Qwen 3.5 from 29.6% to 52.4%, an absolute gain of 22.8 points on the same backbone.The comparison isolates the effect of specialization relative to the initialization checkpoint.
- 4.4 MAIN RESULTS: OpenAegis exceeds GLM 5.2 by 9.1 points and Kimi K2.7 by 0.7 points despite using fewer total and active parameters.The comparison uses the same evaluation scaffold, tools, oracle, and wall-clock budget.
- 4.4 MAIN RESULTS: The experiments separately analyze context management, skill-guided data synthesis, and whether the model learned the intended security workflow.The aggregate result alone does not identify which part of CyberFactory produces the gain.
5 ANALYSIS
The analysis examines how context management and skill-guided synthesis affect cybersecurity performance and behavior. It finds that skill-guided trajectories are internalized, producing more systematic validation, consolidated tool use, and selective submission without inference-time skill access.
- 5.1 LONG-HORIZON CONTEXT COMPACTION: 54.8% overall Pass@1 is achieved by compact execution, exceeding full-history execution and simple truncation under the same scaffold and time budget.For tasks requiring more than 40 tool interactions, Pass@1 increases by 8.1% and context-exhaustion failures decrease by 11.3%.
- 5.2 EFFECT OF THE VULNERABILITY-ANALYSIS SKILL: Skill-guided GLM 5.2 achieves higher Pass@1 despite receiving one quarter of the run time, indicating higher synthesis throughput rather than equal-compute improvement.The comparison uses one 60-minute attempt without the skill versus five independent 15-minute skill-guided attempts.
- 5.2 EFFECT OF THE VULNERABILITY-ANALYSIS SKILL: The skill shifts the teacher toward domain-guided exploration and evidence-based validation, with slightly less compound commands.The reported trajectory shift attributes the gain to selecting a scalable procedure rather than packing more work into each command.
- 5.2 EFFECT OF THE VULNERABILITY-ANALYSIS SKILL: OpenAegis reproduces the teacher’s directional workflow shift after supervised fine-tuning without receiving the skill at inference time.Both explicit skill use and fine-tuning substantially increase exploration and validation calls.
- 5.2 EFFECT OF THE VULNERABILITY-ANALYSIS SKILL: OpenAegis consolidates tool actions, shifting routine inspection from read calls to shell calls and increasing calls with 6–10 operations.Read calls decrease from 28.4% to 7.3%, shell calls rise from 70.1% to 89.9%, and calls with 6–10 operations rise from 4.3% to 30.8%.
- 5.2 EFFECT OF THE VULNERABILITY-ANALYSIS SKILL: Training also induces stronger instrumentation and verification and more selective submission, while action consolidation is interpreted as broader learned behavior rather than a direct skill fingerprint.OpenAegis compiles with AddressSanitizer more often, checks sanitizer output more frequently, and submits exactly once more often.
6 CONCLUSION
CyberFactory converts real-world CVE artifacts into reproducible, executable supervision for training OpenAegis across multiple cybersecurity tasks. OpenAegis reaches 52.4% Pass@1 on CyberGym, while the evaluation remains bounded by artifact availability, benchmark coverage, and a fixed one-hour budget.
- 6 CONCLUSION: CyberFactory transforms fragmented real-world CVE artifacts into executable and verifiable supervision spanning PoC generation, vulnerability patching, and CyberQA.The framework uses a vulnerability-analysis skill covering source inspection, domain-guided exploration, evidence-based validation, and iterative refinement.
- 6 CONCLUSION: 52.4% Pass@1 is achieved by OpenAegis under a one-hour CyberGym budget, improving 22.8 points over its Qwen 3.5 base model and exceeding the evaluated larger baselines.The reported comparisons are 9.1 points above GLM 5.2 and 0.7 points above Kimi K2.7.
- 6 CONCLUSION: Trajectory analysis links the aggregate gain to internalized prior-guided exploration, systematic instrumentation and validation, and fewer tool calls without inference-time skill access.The conclusion states that verifiable trajectories can teach both task outcomes and coherent security-analysis procedures.
- 6 CONCLUSION: The evaluation is bounded by available CVE artifacts, benchmark coverage, and a fixed one-hour execution budget, with some targets still benefiting from manual input construction.The paper presents CyberFactory as a step toward reproducible capability development rather than a complete solution.