Source-linked AI summary
No, Your Tools Are Not Enough: It's Time to Bring Agents to the Hunt
Ati Priya Bajaj, Xiang Mei, Tiffany Bao, Yan Shoshitaishvili, Ruoyu Wang, Adam Doupé
TL;DR
Vulnerability discovery still depends on human expertise and intervention, while few public accounts show end-to-end agentic workflows across diverse domains. The paper presents iterative workflows using AI agents across varied programs and reports that structured analysis, dynamic verification, and suitable program representations support discovery. Its scope is vulnerability discovery and program analysis, not exploit generation or weaponization.
Problem
Existing vulnerability discovery approaches still depend on human expertise for domain knowledge and intervention throughout the process.
Source: §1 · p. 1Method
The authors use agentic workflows with general-purpose LLMs for vulnerability discovery across diverse real-world programs.
Source: §1 · p. 1Results
Agents are most effective in multi-step structured workflows; well-scoped analysis with dynamic verification can reduce false positives, and reverse-engineering tools and program representations can enhance performance.
Source: §1 · pp. 1–2Takeaways & Limitations
The authors report that researchers can systematically use readily available general-purpose models for vulnerability discovery across diverse targets.
Source: §1 · p. 1The work addresses vulnerability discovery and program analysis, not exploit generation or weaponization.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Modern software systems are large and complex, demanding scalable and autonomous techniques for vulnerability discovery. Existing approaches still depend on human expertise for domain knowledge, and on human intervention throughout the process of vulnerability discovery. AI agents offer us an opportunity to make vulnerability discovery more adaptive, scalable, and autonomous. Agents bring the ability to make continuous decisions and adapt, and general knowledge that can be augmented. In this paper, we present our experience using AI agents to find vulnerabilities across diverse programs such as Windows drivers, the Linux kernel, XNU, firmware, and web applications. We use an iterative agentic workflow and find that it generalizes across these targets. We discuss tasks that agents help with, which were previously challenging, and some future research directions we consider important for advancing autonomous vulnerability discovery.
1 INTRODUCTION
The introduction motivates scalable vulnerability discovery as manual discovery becomes intractable, then presents AI-agent workflows for real-world targets as a practical response to the limited public evidence for end-to-end, cross-domain agentic workflows. It reports that structured, multi-step analysis and dynamic verification reduce false positives, while reverse-engineering tools and program representations affect agent performance.
1 INTRODUCTION
- Manual vulnerability discovery is intractable at software-system scale, motivating decades of work toward more scalable and autonomous techniques.The introduction notes DARPA’s Cyber Grand Challenge and subsequent combinations of techniques to address individual limitations.Source: §1 · p. 1
- General-purpose LLMs supported agentic vulnerability discovery across operating system kernels, firmware, Windows drivers, and web applications, despite lacking the security specialization of Mythos and Fable 5.The authors also describe how researchers can systematically use these readily available models in practice.Source: §1 · p. 1
- Well-scoped analysis with dynamic verification can significantly reduce false-positive reports, and reverse-engineering tools and program representations can enhance agent performance.The authors find agents most effective in multi-step structured workflows rather than one-shot vulnerability searches.Source: §1 · pp. 1–2
- The paper focuses on vulnerability discovery and program analysis, not exploit generation or weaponization.The authors explicitly state that their workflows are not designed to weaponize vulnerabilities.Source: §1 · p. 2
2 BACKGROUND
The background surveys static, dynamic, machine-learning-based, and agentic approaches to vulnerability discovery, distinguishing analysis techniques from adaptive agents that can act and refine their strategies.
2 BACKGROUND
- The background introduces the techniques used for vulnerability discovery before surveying their distinct approaches.The section’s overview encompasses static analysis, dynamic analysis, machine-learning methods, and agents.Source: §2 · p. 2, §2.1 · p. 2, §2.2 · p. 2
2.1 Static Vulnerability Discovery
- Static analysis reasons about code without executing it, using program representations to track data flow, state transitions, or relationships between components.It is an over-approximation of program behavior.Source: §2.1 · p. 2
2.2 Dynamic Vulnerability Discovery
- Dynamic analysis observes program behavior through execution, including fuzzing and symbolic execution, which trade scalability against path coverage or precision.Static and dynamic analysis can be combined in hybrid approaches; emulation is useful for embedded firmware and closed-source programs.Source: §2.2 · p. 2
2.3 Machine-Learning-Based Vulnerability Discovery
- Earlier machine-learning approaches trained on program representations to predict vulnerable code, classify vulnerability types, or rank potentially vulnerable functions.LLMs later enabled richer generated outputs and have been used with analysis tools for tasks such as filtering false positives or generating seeds.Source: §2.3 · p. 2
2.4 Agentic Vulnerability Discovery
- An AI agent uses an LLM to reason, plan, act with tools and memory, and adapt its strategy based on feedback.Agent objectives can include finding, reproducing, or validating vulnerabilities.Source: §2.4 · p. 2
3 WORKFLOW FOR AGENTIC VULNERABILITY DISCOVERY
The workflow organizes agentic vulnerability discovery into five phases, from understanding and narrowing the target to verifying candidates and applying the findings downstream.
3 WORKFLOW FOR AGENTIC VULNERABILITY DISCOVERY
- Reconnaissance identifies entry points, components, interactions, and dependencies; vulnerability scoping then prioritizes relevant classes and target components.For user-controlled input, scoping may focus on parsing and downstream data use; reused libraries may be checked for versions with known vulnerabilities.Source: §3 · p. 2
- After narrowing the scope, agents can hypothesize vulnerabilities from prior understanding, optionally combining their hypotheses with tool outputs or generating inputs and scripts to reach vulnerable code.Potential inputs or scripts can then be tested against the target or assessed by another agent.Source: §3 · p. 3
- Verification can discard or refine false positives, while successfully verified vulnerabilities can be reported, analyzed for root cause, patched, or combined to amplify impact.The workflow’s figure presents a high-level view of the agentic discovery process.Source: §3 · p. 3, Figure 1 · p. 3
4 A TOUR OF THE EVIDENCE
The section presents findings and experiences from agentic vulnerability discovery across diverse program targets.
4 A TOUR OF THE EVIDENCE
- The section reports findings and experience using agentic workflows across a diverse set of program targets.Source: §4 · p. 3
4.1 Linux Kernel
In a heavily tested Linux kernel, LLM-assisted methods surfaced many crashes, including a race condition that survived years of fuzzing and review. The workflow linked code paths, guided fuzzing, and triaged the resulting crash, while hardened-kernel exploitation still required expert intervention.
4.1 Linux Kernel
- LLM-assisted methods found around 200 unique Linux kernel crashes and more than 20 CVE-assigned bugs; CVE-2026-31419 won a $10,000 kernelCTF bounty.Source: §4.1 · p. 3
- Thread 2’s tail movement let Thread 1 reuse a freed skb; the fix replaced the racy predicate with an index comparison against a pre-loop slave-count snapshot.Source: Figure 2 · p. 3, §4.1 · p. 3
- The bug survived over four years because two threads had to interleave across data and configuration paths in a tight window that gave no positive coverage signal.Source: §4.1 · p. 3
- An AI agent linked functions across the data and configuration paths, then directed syzkaller to focus and mutate the related corpus more often.Source: §4.1 · p. 4
- After a crash without a stable reproducer, the agent hypothesized the race, wrote a proof-of-concept, and completed triage in several hours without manual effort.Source: §4.1 · p. 4
- A human expert supplied the netem-delay reclaim insight and selected suitable primitives; only then did the agent finish exploitation on hardened Linux.Source: §4.1 · p. 4
4.2 XNU
The workflow found vulnerabilities in open-source XNU, including an unprivileged heap overwrite, and also surfaced under-submission bugs in closed-source drivers and complex protocols. XNU underpins Apple operating systems including iOS, iPadOS, WatchOS, visionOS, and tvOS.
4.2 XNU
- The XNU workflow prompted across target modules and bug classes, then used an XNU KASAN kernel to validate candidates; findings included fixed bugs and reported zero-days.The open-source XNU release lags behind the deployed version.Source: §4.2 · p. 4
- An inverted size check in recvmsg_x reuses an undersized uio vector, causing an unprivileged process to trigger a heap overwrite.The private syscall is intended for Apple frameworks but remains accessible to unprivileged user code.Source: §4.2 · pp. 4–5
- The recvmsg_x bug was introduced in macOS 14.0 and fixed in macOS 26.4; fuzzing would need a specialized multi-buffer network harness.The fuzzer would need to control buffer sizes and the number and size of received datagrams.Source: §4.2 · p. 5
4.2.2 Under Submission Bugs.
- One under-submission finding was a reproducible use-after-free in a closed-source driver analyzed with IDA decompilation through MCP.In an accidental run without IDA, the agent used objdump to disassemble the driver and still found issues.Source: §4.2.2 · p. 5
- A remotely triggerable use-after-free in a complex protocol required a custom implementation to drive XNU into the vulnerable state, and had been present for over a decade.The authors say this was unlikely to be found by fuzzing and report the finding as CVE-2026-28982.Source: §4.2.2 · p. 5
4.3 Windows Drivers
The Windows-driver workflow combines agentic vulnerability hypotheses from decompilation with dynamic symbolic execution (DSE) verification. It found multiple vulnerabilities in Windows 10 and 11 drivers, while human triage remained necessary because most generated hypotheses were false positives.
4.3 Windows Drivers
- Multiple vulnerabilities were identified in first-party and third-party Windows 10 and 11 drivers, mostly through agentic discovery with static analysis and without fuzzing.The paper withholds details because triage and responsible disclosure were incomplete.Source: §4.3 · p. 5
4.3.1 Hypothesis Generation using Decompilation.
- Agents inspect decompiled drivers to locate user-input paths, build high-level feature models, and generate hypotheses from focus locations and reachable-function closures.High-level feature reports led to more abstract hypotheses; without them, agents often overfocused on code patterns and ignored functionality.Source: §4.3.1 · p. 5
- Decompilation quality matters: recovering pointer and struct types reduces false hypotheses, while meaningful variable and function names improve agent-generated hypotheses.The authors added full-binary type inference and Windows kernel-specific type and function definitions, then used LLM-assisted renaming.Source: §4.3.1 · pp. 5–6
4.3.2 Hypothesis Verification using DSE.
- Most generated vulnerability hypotheses are false positives, so human analysts still need to triage them and benefit from quickly filtering unsupported candidates.DSE can eliminate hypotheses that overlook checks such as bounds checks in a callee.Source: §4.3.2 · p. 6
- DSE verification checks whether a hypothesized code path is reachable and its vulnerability condition can hold, with generated scripts limited to three function-call layers.Agents identify relevant dynamic memory values; generated triggering inputs rarely work, but when they do, they directly yield high-likelihood findings.Source: §4.3.2 · p. 6
4.4 Firmware
The firmware workflow uses agents to connect firmware binaries to upstream projects, propose known and novel issues, and validate candidates iteratively with human help. The authors found that open-source anchors and validation improved the grounding of findings, while context limits and hardware-dependent checks constrained the process.
4.4.1 Hypothesis Generation.
- Firmware analysis starts by identifying likely upstream components, checking whether known flaws persist in frozen or lightly modified forks, and mapping novel source-based hypotheses back to reachable firmware code.Candidates require concrete validation steps.Source: §4.4.1 · p. 6
4.4.2 Hypothesis Validation.
- Validation often required human assistance, including commands, tools, purchased equipment, or physical operation of Bluetooth devices.Researchers supplied harnesses through emulation or physical hardware, while agents iterated on hypotheses using observed behavior.Source: §4.4.2 · p. 6
4.4.3 AI Agent Effectiveness Impression.
- During iterative BLE validation, observed traffic showed that a suspected encryption flaw was unsupported because the relevant code was inactive, and the agent revised its finding.The initial report had treated blank ER/IR keys as evidence of a BLE encryption vulnerability.Source: §4.4.3 · p. 6
- Directly asking agents to find firmware vulnerabilities without an open-source anchor or validation loop produced ungrounded, frequently spurious guesses.Without binary reverse-engineering tools such as IDA Pro MCP, agents were less effective, in part due to context limits.Source: §4.4.3 · p. 6
- Specialized domain subagents helped keep reasoning focused amid context limits, although agents were not consistently effective at maintaining coherent documentation over time.The paper describes separate firmware-component skills, such as Bluetooth and GPS security roles.Source: §4.4.3 · pp. 6–7
4.5 Web Applications
Agents reported vulnerabilities across web applications, with reports checked through PoCs and manual review. The experience also exposed false positives from test-only conditions and misread application-specific business logic.
4.5 Web Applications
- Agents reported more than 200 vulnerabilities across approximately 25 web applications, each with a PoC; agentic triage left half for manual review.PoCs were run against agent-set-up local deployments.Source: §4.5 · p. 7
- Verification combined PoC execution, code and log review, optional manual reproduction, and sometimes patching followed by a rerun.Reports generally included affected files and lines, a PoC, and a potential patch or mitigation recommendation.Source: §4.5 · p. 7
- Dynamic PoC verification still left false positives when agents missed production-versus-test conditions, such as required environment variables.The cited workflow exposed credentials only when triggered by a developer with access to the organization’s GitHub credentials; refining scope and mirroring production can mitigate such cases.Source: §4.5 · p. 7
- Agents also misread target-specific business logic: an OAuth access finding was spurious because both users already shared access to the data.The authors describe these as reports that do not hold under a target’s specific context.Source: §4.5 · p. 7
- Agents are suited to application-specific logic vulnerabilities and can sometimes chain vulnerabilities into an end-to-end attack.They form and verify custom hypotheses, a task the authors note is often manual.Source: §4.5 · p. 7
4.6 Agentically-Reimplemented Code
The SafeLibs study uses agentically reimplemented Rust libraries as a controlled setting for searching for re-emerged bugs. A single detection pass found multiple reimplemented vulnerabilities per library on average, despite guidance seeded with prior bugs.
4.6 Agentically-Reimplemented Code
- The setting is motivated by evidence that software reimplementations can inherit bug types found in their originals, while agent-generated code is also vulnerable.The authors use this as a controlled dataset for evaluating agentic reimplementation bug hunting.Source: §4.6 · p. 7
- A single agentic vulnerability-detection pass identified multiple reimplemented vulnerabilities per library on average, despite being seeded with prior-bug specifics.SafeLibs also provides reimplementation agents information on prior non-memory CVEs and instructions to avoid those bugs.Source: §4.6 · p. 7
- SafeLibs reimplemented 24 load-bearing C libraries in Rust, aiming for memory-safer, drop-in-compatible replacements.Examples include libpng and libgcrypt.Source: §4.6 · p. 7
- The authors report analogous bug susceptibility in agentic reimplementations and human reimplementations.They note that this area merits further exploration as agentic reimplementation becomes more common.Source: §4.6 · p. 7
5 WHAT AGENTS ENABLE
The paper argues that agents can address recurring bottlenecks in vulnerability discovery by performing reconnaissance, forming target-specific hypotheses, and refining them. Their potential spans input generation, harnessing, logic-bug detection, and adaptive multistep work.
5 WHAT AGENTS ENABLE
- Automated discovery remains bottlenecked by the expertise and effort needed to write rules, harnesses, specifications, or oracles, especially for application-specific logic bugs.Some such violations do not produce sanitizer-detectable crashes.Source: §5 · p. 8
- Agents perform reconnaissance and refine target-specific hypotheses, unlike classification-based machine learning or a single interactive LLM.The authors frame this as enabling scalability and autonomy.Source: §5 · p. 8
- Agents can infer code structure and refine inputs using execution feedback to reach target functions, avoiding reliance on predefined corpora or handwritten grammars.The passage contrasts this with limits from fuzzing inputs and solver constraints in symbolic execution.Source: §5 · p. 8
- Agents can generate harnesses across a codebase, addressing the manual API and component-interaction knowledge that leaves projects under-harnessed.Harnesses are fuzzers’ entry points for exploring code.Source: §5 · p. 8
- Agents can create application-specific oracles or invariants that make logic violations observable when sanitizers cannot detect deviations from intended behavior.Memory-corruption bugs may be detected by sanitizers, whereas logic bugs require annotations or custom sanitizers for automatic detection.Source: §5 · p. 8
- Agentic discovery supports adaptive decisions across iterative steps; agents can also help set up firmware emulation and automate crash triage and deduplication.Firmware analysis otherwise requires manual identification of architecture, peripherals, and open-source libraries.Source: §5 · p. 8
6 WHAT NOW
The paper identifies verification, measurement, observability, supply-chain risk, cost, guardrails, and dual use as challenges for agentic vulnerability research. It highlights dynamic verification’s value while noting the distinct burden of validating firmware.
6 WHAT NOW
- Dynamic verification has filtered false positives in the authors’ experience, but firmware validation requires costly rehosting for emulation or slow, limited-availability physical hardware.The authors identify emulation as a promising area for agent contribution.Source: §6 · p. 8
- The authors call for measuring where agents outperform existing methods, complement them, or fail as model capabilities advance.Tasks such as grammar-based fuzzing, structured input generation, and automatic harnessing have become more approachable.Source: §6 · p. 8
- Agent decisions and discarded hypotheses can be difficult to inspect, motivating methods that expose decision-making and intermediate results.The authors note that outdated repository material, saved memory, or lost context can shape decisions.Source: §6 · pp. 8–9
- Agent workflows raise supply-chain and cost concerns, while guardrails may disrupt defensive research and shared agent capabilities pose dual-use questions.The paper calls for inspecting externally introduced components, improving token efficiency, clarifying safeguards, and studying how defenders can stay ahead.Source: §6 · p. 9