PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agents

Pengfei He, Ash Fox, Lesly Miculicich, Stefan Friedli, Daniel Fabian, Burak Gokturk, Jiliang Tang, Chen-Yu Lee, Tomas Pfister, Long T. Le

arXiv:2602.02164v2cs.LGcs.CR

TL;DR

Existing LLM-based cybersecurity approaches have limited interaction, weak execution grounding, and little experience reuse for automatic vulnerability discovery and exploitation. Co-RedTeam addresses this gap with a security-aware multi-agent workflow that coordinates evidence-backed discovery and iterative, execution-grounded exploitation while retaining long-term memory. It consistently outperforms baselines, achieving over 60% exploitation success and over 10% absolute improvement in detection tasks.

  • Problem

    Existing approaches struggle to automate vulnerability discovery and exploitation because they provide limited interaction, weak execution grounding, and insufficient experience reuse.

  • Method

    Co-RedTeam coordinates code-aware discovery and iterative exploitation using security-domain knowledge, execution feedback, and long-term memory.

  • Results

    Over 60% exploitation success and over 10% absolute improvement in detection tasks are reported across challenging cybersecurity benchmarks.

  • Takeaways & Limitations

    Execution feedback, structured interaction, and memory are important components for robust and generalizable cybersecurity agents.

  • Takeaways & Limitations

    The task assumes access to a target codebase and an associated execution environment for vulnerability analysis and reproduction.

Abstract

from arXiv · show

Large language models (LLMs) have shown promise in assisting cybersecurity tasks, yet existing approaches struggle with automatic vulnerability discovery and exploitation due to limited interaction, weak execution grounding, and a lack of experience reuse. We propose Co-RedTeam, a security-aware multi-agent framework designed to mirror real-world red-teaming workflows by integrating security-domain knowledge, code-aware analysis, execution-grounded iterative reasoning, and long-term memory. Co-RedTeam decomposes vulnerability analysis into coordinated discovery and exploitation stages, enabling agents to plan, execute, validate, and refine actions based on real execution feedback while learning from prior trajectories. Extensive evaluations on challenging security benchmarks demonstrate that Co-RedTeam consistently outperforms strong baselines across diverse backbone models, achieving over 60% success rate in vulnerability exploitation and over 10% absolute improvement in vulnerability detection. Ablation and iteration studies further confirm the critical role of execution feedback, structured interaction, and memory for building robust and generalizable cybersecurity agents.

1. Introduction

Red teaming is important for proactively identifying and validating software vulnerabilities, but existing LLM-based approaches struggle with complex, multi-step security workflows. Co-RedTeam addresses these limitations through coordinated multi-agent reasoning, execution-grounded validation, and experience reuse.

  • Red teaming helps organizations assess security posture, validate defenses, and reduce potential losses from software vulnerabilities.
  • Existing LLM-based security approaches struggle with limited interaction and reliable handling of complex vulnerability discovery and exploitation workflows.
  • Co-RedTeam integrates security grounding, code-aware analysis, execution-driven reasoning, and experience accumulation to support realistic vulnerability analysis.
  • Co-RedTeam evaluates candidate vulnerabilities with code-level evidence and iteratively refines exploitation strategies using real execution feedback in isolated environments.
  • Over 60% exploitation success and 20% detection accuracy are reported across challenging cybersecurity benchmarks.

2. Related works

Research on LLM-based cybersecurity agents spans single-model, single-agent, and multi-agent approaches. Co-RedTeam is positioned within this shift toward structured agentic systems for complex security tasks.

  • LLM research increasingly targets vulnerability detection, exploitation, and repair using code-capable models.
  • Agentic systems structure LLMs as autonomous agents that interact with tools, environments, and intermediate feedback.
  • Co-RedTeam’s overview combines collaborative vulnerability discovery with iterative exploitation and shared long-term memory.
  • Single-agent security workflows iteratively analyze codebases and refine vulnerability hypotheses but remain limited on complex, multi-step tasks.

3. Co-RedTeam

Co-RedTeam orchestrates vulnerability discovery and exploitation as coordinated stages over codebases and execution environments. Its agents generate evidence-backed hypotheses, validate actions, learn from feedback, and produce reproducible exploitation evidence.

  • System overview: Co-RedTeam takes a target codebase and optional vulnerability description, then performs collaborative discovery followed by execution-driven exploitation.
  • System overview: The system maintains shared long-term memory across both stages to accumulate experience and support continual improvement over time.
  • Problem setup: The task requires code-level vulnerability identification and execution-based reproduction of impact within an associated execution environment.
  • Orchestration: The orchestrator assigns role-specific tools, including code browsing and security documentation for discovery and sandboxed execution interfaces for exploitation.
  • Stage I: Vulnerability Discovery: Stage I uses Analysis and Critique agents to inspect code, construct evidence chains, review hypotheses, and produce validated candidates with assessed risk levels.
  • Stage II: Iterative Exploitation: Stage II treats exploitation as structured search, using explicit plans, validation gates, isolated execution, and evaluation feedback to revise actions.
  • Stage II: Iterative Exploitation: Execution-grounded iterative reasoning enables adaptive vulnerability validation beyond static or single-shot approaches.

4. Experiments

The experiments evaluate Co-RedTeam on multiple challenging cybersecurity benchmarks against single-model and agent-based baselines. Results emphasize strong overall performance while using isolated, reproducible setups and benchmark-specific success-rate metrics.

  • Experimental setup: The evaluation covers CyBench, BountyBench, and CyberGym, spanning CTF tasks, real-world offensive and defensive tasks, and executable proof-of-concept reproduction.
  • Experimental setup: Baselines include vanilla models, OpenHands, C-Agent, VulTrail, and RepoAudit, representing single-model, generic-agent, execution-feedback, and specialized multi-agent approaches.
  • Experimental setup: The setup uses a shared backbone LLM, isolated Docker containers, retrieved vulnerability documentation, and initialized long-term memory for safety and reproducibility.
  • Experimental setup: Stage I allows 3 refinement iterations, while Stage II defaults to a 20-iteration exploitation cap whose impact is examined separately.
  • Evaluation: Success rates for vulnerability detection and exploitation are the primary metrics, with CyBench and CyberGym focused exclusively on exploitation.
  • Main results: Co-RedTeam consistently achieves the strongest performance across the evaluated benchmarks and backbone models.
  • Main results: CyBench experiments omit RepoAudit and VulTrail because those static-analysis baselines lack execution capabilities required to capture flags.

Co-RedTeam

Co-RedTeam’s performance depends on closed-loop planning, execution feedback, and memory across vulnerability exploitation tasks. Iterative execution improves results, with stronger backbones converging faster and reaching higher peaks.

  • Main results: 63.7% ASR on CyBench, 65.0% exploit success and 20.0% detection accuracy on BountyBench, and 37.3% ASR on CyberGym are reported with Gemini-3-Pro.These results represent gains over the strongest baselines across the evaluated benchmarks.
  • Ablation studies: Long-term memory, code browsing, and vulnerability documentation each contribute to performance, while their removal degrades vulnerability analysis and long-horizon exploitation.The results attribute strong performance to the integration of planning, execution, validation, critique, and memory rather than one component alone.
  • Iteration analysis: Both Gemini-2.5-Pro and Gemini-3-Pro improve as exploitation iterations increase, but Gemini-3-Pro peaks around 13 iterations while Gemini-2.5-Pro continues improving until approximately 17.After their respective peaks, both models saturate, indicating diminishing returns from additional iterations.
  • Iteration analysis: Gemini-3-Pro reaches a higher peak success rate and exploits execution feedback more efficiently than Gemini-2.5-Pro.The comparison links stronger backbone models with faster convergence and higher final performance in the iteration study.

5. Analysis

The analysis examines Co-RedTeam through memory, detection reliability, latency, and backbone efficiency. Long-term memory supports both immediate gains from prior knowledge and sustained improvement through experience, while the system remains faster than complex agent baselines in the reported comparison.

  • Memory analysis: Warm-started memory configurations outperform cold-start variants early, while evolving configurations improve over time and static or memory-free settings plateau.Tasks are processed sequentially and evaluated with a moving-average success rate over a window of 100 tasks.
  • Memory analysis: Warm Start (Evolving) achieves the strongest performance by combining rapid early gains with continued long-term improvement.The result highlights complementary benefits from curated prior knowledge and continual learning.
  • Vulnerability discovery: 14.3% precision on BountyBench is roughly 5x higher than C-Agent, indicating more reliable vulnerability discovery for Co-RedTeam.The analysis reports improved precision and recall relative to all baselines using Gemini-2.5-Pro.
  • Latency analysis: 198.7s versus 219.6s for OpenHands on BountyBench illustrates Co-RedTeam’s lower reported runtime despite its multi-turn architecture.Co-RedTeam is reported to have lower runtimes than both OpenHands and C-Agent across all three benchmarks.
  • Latency analysis: Gemini-3-Pro reduces latency by approximately 10-15% across the board relative to Gemini-2.5-Pro.The reported transition improves speed while supporting more advanced iterative detection strategies.

6. Conclusion

Co-RedTeam integrates security knowledge, code-aware analysis, execution-grounded reasoning, and long-term memory for automated vulnerability discovery and exploitation. It achieves over 60% exploitation success and over 10% absolute improvement in detection tasks.

  • The framework targets both automatic vulnerability discovery and exploitation.
  • Co-RedTeam integrates security-domain knowledge, code-aware analysis, execution-grounded iterative reasoning, and long-term memory.
  • Over 60% exploitation success and over 10% absolute improvement in detection tasks validate the framework’s effectiveness.

A. Details of Co-RedTeam

The detailed framework materials describe agent prompts, vulnerability documentation, code browsing and execution tools, and memory items.

  • The framework details prompts for each agent.
  • It documents code browsing and execution tools used by the agents.
  • The materials include examples of memory items.

A.1. Agent setups

Co-RedTeam structures vulnerability discovery and exploitation through specialized agents, explicit evidence schemas, iterative planning, execution feedback, and critique. The setup emphasizes grounded analysis, validated outputs, and corrective refinement rather than unsupported guesses.

  • Structured Interaction: Strict schemas constrain agent inputs and outputs, including an exploit-output-schema with strategy, log, loop status, and action steps.
  • Analysis Agent: The analysis agent proposes vulnerabilities from code while aiming to remain creative but grounded in evidence.
  • Discovery: Vulnerability analysis requires source, sink, and context evidence before generating structured vulnerability records.
  • Critique Agent: The critique agent reviews each proposal’s description, evidence, and risk rationale, assigning status, feedback, and an estimated risk level.
  • Grounding: The workflow uses code-browser tools, security-knowledge tools, and memory retrieval to ground analysis and exploitation planning.
  • Planner Agent: The planner updates research plans from execution results, marks steps done or blocked, and inserts corrective steps after failures.

A.4. Example of memory items

Co-RedTeam’s memory examples encode reusable security strategies and technical patterns for assessing and testing authentication weaknesses. The SSO example organizes mechanism identification, attack-vector selection, and execution-based verification.

  • The memory system includes an SSO Assessment Strategy for assessing and exploiting Single-Sign-On configurations.
  • The SSO strategy identifies the authentication mechanism before selecting a corresponding attack vector.
  • The workflow verifies suspected SSO weaknesses by attempting manipulation and checking for unauthorized access.
  • A stored SAML signature-bypass pattern links SAML usage and XML assertions to hypotheses about weak signature verification.
  • The SAML signature-stripping example tests whether a service provider accepts assertions after the signature block is removed.

A.5. Examples of detection and exploitation tasks

The evaluation separates vulnerability discovery from exploitation. Discovery uses only target code to produce vulnerabilities and reproduction steps, while exploitation adds a vulnerability description and requires triggering it.

  • Task design: The study focuses on two tasks: vulnerability detection and exploitation.These tasks are evaluated in a unified format across several benchmarks.
  • Detection tasks: Detection agents receive only the target codebase and must identify potential vulnerabilities while generating reproduction steps such as proofs of concept.
  • Exploitation tasks: Exploitation agents receive the target codebase and a specified vulnerability, then attempt to trigger or reproduce it.
  • Unified benchmark format: The original benchmark tasks are converted into a unified format for different benchmarks, with examples spanning detection and exploitation.
  • Exploitation tasks: Exploitation examples include reversing a compression algorithm to retrieve a hidden flag and analyzing a binary challenge.

B. Case studies

The case studies show Co-RedTeam producing execution-backed outcomes on both vulnerability detection and reverse-engineering exploitation tasks. Successful examples connect source analysis or binary inspection to concrete evidence and reproduction scripts.

  • Stash exploitation: A reverse-engineering challenge involving the stash binary was marked unattempted and unsolved despite the report’s overall success status.
  • Stash exploitation: The stash workflow identified a 64-bit ELF executable, used strings to find the flag, and created a shell script to print it.
  • AgentScope detection: A security audit confirmed a path-traversal vulnerability in AgentScope’s /api/file endpoint and successfully read the system hosts file.
  • AgentScope detection: The AgentScope case traced the flaw to an unsanitized path parameter passed directly to open(), then validated it with a curl request.
  • Iteration analysis: Figure 4 plots CyBench success rate against maximum detection iterations to illustrate the effect of multi-turn discussions.

C. Additional experiments

Additional experiments test whether Co-RedTeam generalizes across backbone models and whether iterative detection discussions improve performance. Across the evaluated model families and benchmarks, Co-RedTeam is reported as the strongest approach, while additional discussion rounds improve detection before gains plateau.

  • Backbone generalization: Across GPT, Claude, and open-source backbones, Co-RedTeam achieves the best performance on CyBench, BountyBench, and CyberGym.
  • Backbone generalization: Co-RedTeam’s gains remain robust across model families, including smaller and open-source models, unlike execution-aware baselines whose performance varies and degrades on weaker backbones.
  • Detection iterations: Gemini-3-pro rises from 5% to a 20% detection success-rate peak by the third iteration, while Gemini-2.5-pro rises from 2.5% and plateaus at 15% after four iterations.
  • Detection iterations: The reported plateau suggests that multi-turn discussions help detection, but marginal gains diminish after 3–4 refinement rounds.
Loading 2602.02164v2…