PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthesizing Multi-Agent Harnesses for Vulnerability Discovery

Hanzhi Liu, Chaofan Shou, Xiaonan Liu, Hongbo Wen, Yanju Chen, Ryan Jingyang Fang, Yu Feng

arXiv:2604.20801v1cs.CR

TL;DR

Vulnerability-discovery harnesses are hand-designed or narrowly optimized and often receive only coarse pass/fail feedback. AgentFlow jointly searches harness components with a typed graph DSL and uses runtime diagnostics to guide revisions, reaching 84.3% on TerminalBench-2 and finding ten previously unknown Chrome zero-days. Its scope includes source-available targets with configurable build instrumentation and static harness topologies.

  • Problem

    Existing harness optimizers search narrow design spaces and use coarse feedback that does not explain why vulnerability-discovery trials fail.

  • Method

    AgentFlow uses a typed graph DSL spanning agents, topology, message schemas, tools, and coordination, plus a feedback-driven loop using coverage, sanitizer output, and other runtime signals.

  • Results

    AgentFlow reaches 84.3% on TerminalBench-2 with Claude Opus 4.6 and discovers ten previously unknown Chrome zero-days with Kimi K2.5, including two Critical sandbox escapes.

  • Takeaways & Limitations

    Harness design materially affects agent-system performance, and runtime-guided joint edits can produce strong benchmark results and vendor-confirmed vulnerability discoveries.

  • Takeaways & Limitations

    Prior optimizers are limited by narrow search scope, while AgentFlow restricts candidates to statically analyzable topologies without runtime agent spawning or within-execution topology changes.

Abstract

from arXiv · show

LLM agents have begun to find real security vulnerabilities that human auditors and automated fuzzers missed for decades, in source-available targets where the analyst can build and instrument the code. In practice the work is split among several agents, wired together by a harness: the program that fixes which roles exist, how they pass information, which tools each may call, and how retries are coordinated. When the language model is held fixed, changing only the harness can still change success rates by several-fold on public agent benchmarks, yet most harnesses are written by hand; recent harness optimizers each search only a narrow slice of the design space and rely on coarse pass/fail feedback that gives no diagnostic signal about why a trial failed. AgentFlow addresses both limitations with a typed graph DSL whose search space jointly covers agent roles, prompts, tools, communication topology, and coordination protocol, paired with a feedback-driven outer loop that reads runtime signals from the target program itself to diagnose which part of the harness caused the failure and rewrite it accordingly. We evaluate AgentFlow on TerminalBench-2 with Claude Opus 4.6 and on Google Chrome with Kimi K2.5. AgentFlow reaches 84.3% on TerminalBench-2, the highest score in the public leaderboard snapshot we evaluate against, and discovers ten previously unknown zero-day vulnerabilities in Google Chrome, including two Critical sandbox-escape vulnerabilities (CVE-2026-5280 and CVE-2026-6297).

1 Introduction

AgentFlow targets two limitations in multi-agent harness optimization: existing methods search narrow design spaces and rely on coarse feedback. It uses a typed graph DSL and runtime diagnostics to jointly rewrite harness components, achieving leading benchmark performance and finding Chrome zero-days.

  • Motivation: LLM agents have found real vulnerabilities missed by human auditors and automated fuzzers, motivating improved multi-agent vulnerability-discovery systems.Reported examples include SQLite memory corruption, an OpenBSD denial-of-service flaw, website exploitation, and CTF-level exploits.
  • Motivation: A single agent struggles with large outputs, competing subtasks, lost-in-the-middle effects, and sequential commitment to one hypothesis.These limitations motivate splitting source analysis, input crafting, and crash triage across specialized agents.
  • Harness Design: With Claude Opus 4.6 fixed, public TerminalBench-2 harnesses span 20% to 80% pass rates, showing that orchestration, prompts, tools, coordination, and feedback channels matter.The harness determines agent roles, prompts, tools, communication routing, and execution protocol; poor designs can waste substantial campaign compute.
  • Limitations of Prior Work: Existing optimizers search only narrow slices of harness design space, preventing cross-component edits such as adding an agent while rewiring communication and prompts.Meta-Harness remains single-agent, ADAS fixes the communication graph, AFlow uses predefined operators, and MaAS fixes its communication protocol.
  • AgentFlow: AgentFlow represents agents, communication, schemas, tools, and coordination as editable fields in a typed graph DSL, with structural checks rejecting malformed candidates before expensive evaluation.The type system verifies variable resolution, declared endpoints, and graph connectivity; about 20% of proposer outputs are rejected before inference.
  • AgentFlow: AgentFlow replaces binary outcomes with target-runtime diagnostics including test verdicts, coverage, sanitizer reports, action traces, and prior trials to localize failures.Coverage can show whether vulnerable code was reached, while sanitizer output distinguishes memory-safety errors from benign crashes.
  • Results: 84.3% on TerminalBench-2 was achieved with Claude Opus 4.6, while Chrome evaluation with Kimi K2.5 found ten previously unknown zero-days, including two Critical sandbox escapes.The reported CVEs are CVE-2026-5280 and CVE-2026-6297, and the Chrome findings were vendor-confirmed.

2 Motivating Example

The libheif example shows AgentFlow improving a vulnerability-finding harness through successive feedback-driven edits. It progresses from malformed input, to reaching the target function, to targeting the buggy branch with verification and retries.

  • Target: The libheif vulnerability is a heap-buffer-overflow caused by copying 16-bit alpha data from an 8-bit plane, requiring both format validity and an 8-bit alpha channel.The vulnerable color-conversion logic is associated with CVE-2020-23109.
  • Iteration 1: Iteration 1 fails before the vulnerable code runs because a single generic agent produces an input rejected by an early format check.The program’s stderr identifies the rejection even though the agent’s trace claims it tested the parser.
  • Iteration 2: Iteration 2 adds an analyzer and crafter; the resulting file passes format validation and reaches color conversion but misses the 8-bit-alpha branch.Line coverage identifies that the relevant function ran while the buggy branch remained unvisited.
  • Iteration 3: Iteration 3 adds a verifier, rewrites its prompt to start from a real file and retry, and uses AddressSanitizer to expose the memory error.The verifier is downstream of the crafter, shifting effort from constructing a valid container toward exercising the vulnerable behavior.
  • Overall Progress: Across three iterations, AgentFlow adds agents, rewires communication, revises prompts, and routes new runtime signals to the agents that need them.The example illustrates harness improvement as coordinated edits rather than isolated prompt changes.

3 Background

The paper frames vulnerability discovery as a sparse-reward sequential task in which a harness coordinates tool-using agents and runtime feedback disambiguates failed trials. The setting assumes source access and build infrastructure sufficient to expose structured target outputs.

  • Agent and Harness: An agent is an LLM with a system prompt and tools that repeatedly selects actions, observes results, and continues until completion or context exhaustion.A harness coordinates one or more such agents into a pipeline.
  • Runtime Feedback: Runtime feedback includes test verdicts, stdout/stderr, line and branch coverage, and sanitizer reports.Coverage and sanitizer channels require instrumented builds, while verdicts and textual output are generally available without them.
  • Runtime Feedback: Coverage reveals whether inputs reached potentially vulnerable code, while sanitizer reports reveal memory errors even when no visible crash occurs.These signals provide more diagnostic information than a simple pass/fail result.
  • Vulnerability Discovery as an Agent Task: Vulnerability discovery is a sparse-reward sequential decision problem because most trials fail and identical verdicts can reflect very different execution states.A trial may fail before reaching the vulnerable function or after reaching it without taking the relevant error-handling branch.
  • Operational Setting: The operational setting assumes a security analyst has source code and build infrastructure available to configure target feedback channels.Targets can emit verdict and stdout/stderr data, optionally augmented with coverage and sanitizer instrumentation.

4 Problem Formalization

AgentFlow models multi-agent harnesses as typed graph programs whose editable fields cover roles, topology, schemas, tools, and coordination. Its well-formedness rules make these programs executable while supporting fan-out, feedback routing, and retries.

  • 4.1 Multi-Agent Harnesses: A harness H is a multi-agent pipeline decomposed into agent set A, communication topology G, message schemas Σ, tool bindings Φ, and coordination protocol Ψ.
  • 4.1 Multi-Agent Harnesses: Prior systems search restricted subsets of harness components, whereas AgentFlow ranges over all five components in one typed grammar.
  • 4.2 A Typed DSL for Harnesses: AgentFlow programs are labelled directed graphs whose nodes represent agents and whose edges represent dataflow or retry links.
  • 4.2 A Typed DSL for Harnesses: Nodes encode role labels, prompt templates, model identifiers, and tool sets, allowing different agents to receive disjoint capabilities.
  • 4.2 A Typed DSL for Harnesses: Fan-out clones an agent into independent parallel copies, while a downstream template referencing the family output performs the aggregation without a separate join operator.
  • 4.2 A Typed DSL for Harnesses: Edges enforce data dependencies or guarded ok/fail branches, allowing ordinary successors and retry back-edges to coexist.
  • 4.2 A Typed DSL for Harnesses: Feedback channels expose test output, stdout/stderr, coverage, and sanitizer reports to whichever agents reference them in their templates.
  • 4.2 A Typed DSL for Harnesses: Well-formedness requires resolvable template variables, used declared edges, and graph connectivity; the optimizer rejects invalid edits before execution.

5 AgentFlow Framework

HarnessOpt searches the typed AgentFlow program space through iterative proposal, execution, scoring, and diagnosis. Runtime traces and target feedback guide edits to agents, communication, prompts, tools, and coordination.

  • 5 AgentFlow Framework: The optimization objective selects the well-formed harness H maximizing the domain-specific score over task set D.
  • 5 AgentFlow Framework: Algorithm 1 supplies model M, task set D, feedback Ω, score S, and step budget K, then repeats proposal, execution, scoring, and diagnosis.
  • 5 AgentFlow Framework: Each proposal must pass well-formedness validation before dispatch, while ill-typed programs are rejected and re-proposed.
  • 5 AgentFlow Framework: Execution collects per-agent traces and structured feedback for every task, including verdicts, stdout/stderr, coverage, and sanitizer reports when available.
  • 5 AgentFlow Framework: A single rewrite can add or remove agents, rewire edges, alter templates or feedback bindings, change tools, or convert sequential execution into fan-out and merge.
  • 5 AgentFlow Framework: The score is task pass rate for TerminalBench-2 and distinct AddressSanitizer crash signatures for Chrome.
  • 5 AgentFlow Framework: Coverage localizes unreached code regions and sanitizer output distinguishes genuine vulnerabilities from benign crashes or false positives.
  • 5 AgentFlow Framework: Diagnosis identifies the bottleneck agent, intended behavior, actual execution, and corrective harness edit using traces and runtime signals.

6 Implementation

The implementation uses one HarnessOpt loop across TerminalBench-2 and Chrome, with target-specific feedback, reproducibility controls, validation, and bounded retries. On TerminalBench-2, AgentFlow achieves 84.3% in the reported leaderboard snapshot.

  • 6 Implementation: The same HarnessOpt loop is instantiated across both targets, with TerminalBench-2 using Claude Opus 4.6 and Chrome using Kimi K2.5.
  • 6 Implementation: The leaderboard comparison includes ForgeCode at 81.4%, Capy at 77.7%, and Meta-Harness at 76.4%.
  • 6 Implementation: Figure 5 reports synthesis trajectory and leaderboard comparison on 89 TerminalBench-2 tasks using Claude Opus 4.6.
  • 6 Implementation: Feedback bundles expose target-provided channels, with TerminalBench-2 wiring the first two and Chrome wiring all four.
  • 6 Implementation: The implementation uses provider-default sampling and tool schemas, with Anthropic prompt caching enabled for Claude runs at a 71.2% cache-hit rate.
  • 6 Implementation: Approximately 20% of proposer outputs are rejected by the well-formedness check, and each iteration permits at most two retries.

7 Evaluation

AgentFlow is evaluated across TerminalBench-2 and Google Chrome, combining broad harness search with controlled ablations and a real-world vulnerability campaign. It reaches 84.3% on TerminalBench-2 and discovers ten previously unknown Chrome zero-days, including two Critical sandbox escapes.

  • Evaluation protocol: The TerminalBench-2 evaluation uses one task-agnostic AgentFlow program across 89 tasks, with shared model, wall-clock budget, and aggregate-score gating.The protocol is designed to isolate harness differences and retain edits only when they improve the shared program across task categories.
  • RQ1: Effectiveness on TerminalBench-2: 84.3% of TerminalBench-2 tasks are passed, the highest score in the evaluated public leaderboard snapshot.AgentFlow passes 75 of 89 tasks, 2.9 percentage points above ForgeCode and 7.9 points above Meta-Harness.
  • RQ1: Effectiveness on TerminalBench-2: The synthesis trajectory rises from 35.2% to 84.3% through infrastructure, specialization, and ensemble phases.The phases apply tool, coordination, agent, retry, prompt, and topology edits, collectively touching all five formalization components.
  • RQ2: Ablation Study: Prompt-edit ablation causes the largest performance drop at −32.5 pp, compared with −7.9 pp for structural edits and −12.4 pp for tool edits.The ablation is interpreted as evidence that prompt edits carry most optimization signal, while structural and tool edits add further gains.
  • RQ3: Real-World Impact: The Chrome campaign is reported as a real-world capability case study rather than a compute-matched comparison against alternative systems.It uses a synthesis loop on a codebase spanning over 35 million lines and Kimi K2.5 rather than Claude Opus 4.6.
  • RQ3: Real-World Impact: Ten previously unknown Chrome zero-days are discovered with Kimi K2.5, including two Critical sandbox-escape vulnerabilities.CVE-2026-5280 and CVE-2026-6297 are Critical use-after-free vulnerabilities enabling sandbox escape to host code execution; all ten findings were accepted and confirmed by Google.

8 Related Work

Prior harness optimizers expose only limited portions of the five-component harness, while multi-agent frameworks leave topology to application code and self-improving agents rely mainly on self-reported feedback. Table 3 formalizes these uneven search interfaces.

  • Prior optimizer coverage: Meta-Harness partially mutates a single agent but fixes team cardinality at |A|=1, preventing edits that introduce additional roles or inter-agent coordination.Its searchable slice includes Σ, Φ, and part of A, but not multi-agent structure.
  • Prior optimizer coverage: AFlow searches workflow graphs using predefined operators, while MaAS samples agents through a hand-coded cascade with coordination patterns outside that cascade fixed.Their searchable dimensions remain constrained by fixed libraries or protocols.
  • Frameworks and self-improvement: AutoGen, MetaGPT, CAMEL, and ChatDev provide cooperative runtimes, but their application code—not an optimizer—specifies the agent topology.Users can rewrite graphs manually, but task feedback does not drive topology search.
  • Frameworks and self-improvement: Reflexion, Self-Refine, and Tree of Thoughts use agent self-reports or internal states, whereas AgentFlow additionally reads structural signals from the target program.These signals include test outcomes, program output, and available coverage information.
  • Security-oriented systems: Coverage-guided fuzzers provide instrumentation that AgentFlow reuses as one runtime feedback channel, while several LLM security systems keep the surrounding harness fixed.AgentFlow combines instrumentation with broader harness search rather than treating the harness as fixed.
  • Prior optimizer coverage: Prior optimizers differ in which of the five harness components—A, G, Σ, Φ, and Ψ—they expose as search variables.Table 3 distinguishes first-class, indirect, and fixed components.

9 Conclusion

AgentFlow synthesizes multi-agent harnesses with a unified typed graph DSL, feedback-driven optimization, and structural validation. It achieves 84.3% on TerminalBench-2 and finds ten previously unknown Chrome zero-days, including two Critical sandbox-escape CVEs.

  • Conclusion: AgentFlow combines a typed graph DSL over five harness dimensions with a feedback-driven outer loop and cheap structural validation.The system targets automated synthesis of multi-agent harnesses.
  • Conclusion: 84.3% on TerminalBench-2 is AgentFlow’s score, the highest among Claude Opus 4.6 entries on the public leaderboard.The comparison is limited to the public leaderboard entries reported in the paper.
  • Conclusion: Ten previously unknown zero-day vulnerabilities were discovered in Chrome, including two Critical sandbox-escape CVEs: CVE-2026-5280 and CVE-2026-6297.The Chrome campaign used Kimi K2.5.

A Open Science

The paper releases the AgentFlow harness optimizer and the materials used to run its TerminalBench-2 optimization rounds, while the Chrome-specific configuration remains withheld.

  • Open Science: The released artifact includes the harness optimizer source, example pipelines, templates, and the CLI used for every optimization round in Section 7.The artifact covers the AgentFlow runtime, diagnoser and proposer prompts, archive manager, and proposal-validation pipeline.

B Ethical Considerations

The reported Chrome vulnerabilities were disclosed to Google and accepted through its reward program, but the release withholds exploit-enabling materials and the Chrome-specific configuration.

  • Disclosure: All ten previously unknown vulnerabilities were disclosed to the affected vendor before paper submission and accepted by Google’s Chrome Vulnerability Reward Program.The filings occurred in Q1 2026, with some public patch metadata spanning releases dated 2026-03-18, 2026-03-31, and 2026-04-15.
  • Release boundaries: The released artifact excludes per-target proof-of-concept inputs, exploit primitives, crashing inputs, and trigger conditions for the ten Chrome vulnerabilities.The paper states that it applies the same restraint in the body of the paper.
  • Release boundaries: The Chrome-specific AgentFlow configuration is held back, while the released per-iteration prompts and programs correspond to the TerminalBench-2 leaderboard run.This creates a deliberate boundary between reproducibility materials for the benchmark and materials for the Chrome campaign.
Loading 2604.20801v1…