PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Based Agents for Software and Systems Security: Approaches, Applications, and Assessment

Jingjing Nie, Jiawei Guo, Krishna Meda, Haipeng Cai

arXiv:2608.28490v1cs.CRcs.AI

TL;DR

Security-agent research lacks a systematic, comparable account of how agents are built, applied, and assessed, despite the risks of delegating procedural security workflows. This paper reviews the peer-reviewed literature across those dimensions and finds that agents are commonly built for action, while bounded authority, auditable behavior, and realistic, reproducible, safe evaluation remain underdeveloped.

  • Problem

    Security-agent studies vary in agent definitions, application risk, and assessment practices, making it difficult to determine what has been built and whether results are safe and comparable.

  • Method

    The paper systematically reviews peer-reviewed LLM-based security-agent research and organizes it by approach, application, and assessment.

  • Results

    91% adopt a modular pipeline, 88% place the backbone LLM in a planner-and-actor role, and 95% support fully autonomous operation, while bounded authority and auditable behavior remain less developed.

  • Takeaways & Limitations

    The survey points toward evidence-centered architecture, risk-aware autonomy, realistic and reproducible benchmarks, trajectory-level assessment, and accountable human-agent collaboration.

  • Takeaways & Limitations

    Current assessment lacks benchmark ecosystems that are simultaneously realistic, reproducible, and safe.

Abstract

from arXiv · show

Software and systems security workflows are typically procedural: analysts inspect heterogeneous artifacts, form hypotheses, invoke tools, interpret outputs, and revise plans. Large language model (LLM)-based agents, which can plan, use tools, retain state, and revise actions across multi-step workflows, are being rapidly adopted to automate this work. Given the consequences of delegating security decisions to autonomous systems, understanding how such agents are built, used, and assessed is crucial. Yet to this date, there remains a lack of systematic understanding of what has been done and how far we are in this field: the term "agent" is applied inconsistently, applications differ sharply in risk, and assessment protocols are often incomparable. To gain a comprehensive and coherent view of this area hence inform relevant future research, this paper provides a systematic literature review of the (1) technical approaches, including agent architecture, perception, memory, reasoning and planning, action space, orchestration, and self-improvement, (2) applications, with respect to the security tasks served, and (3) assessment, including the datasets, outcome and trajectory metrics, safety measures, and baselines considered, over the peer-reviewed literature spanning the emergence of this area (2023--2026). Our synthesis reveals a field that has built agents able to act but not yet agents whose authority is bounded or whose behavior is auditable. In addition to knowledge systematization, we also extend our insights into the limitations of and challenges faced by current approach, application, and assessment designs, which shed light on potentially promising future research directions.

1 Introduction

Security workflows require agents to reason across heterogeneous artifacts, use tools, and revise multi-step actions, but the literature remains difficult to compare because agent definitions, application risks, and assessments vary. This survey organizes 100 peer-reviewed papers across agent approaches, security applications, and assessment practices, finding that agents can act but lack bounded authority and auditable behavior.

  • Motivation: Security tasks require inspecting heterogeneous artifacts, forming hypotheses, invoking tools, interpreting outputs, and revising plans.Examples include repository auditing, penetration testing, vulnerability repair, and SOC evidence correlation.
  • Challenges: “Agent” is used inconsistently, ranging from a single tool-using LLM to multi-agent systems with planners, critics, memory, retrieval, and policy layers.This variation makes reported system designs difficult to interpret consistently.
  • Challenges: Security applications differ sharply in risk, from alert summarization to exploiting assets, patching production code, or changing access-control policies.Consequently, the safety profile of an agent depends strongly on the task it performs.
  • Challenges: Final task success does not reveal whether agents used evidence faithfully, stayed within scope, avoided hallucinations, respected permissions, or controlled time and cost.The survey therefore treats trajectory and safety properties as important complements to outcome metrics.
  • Survey contribution: The survey reviews 100 peer-reviewed papers from January 2023 through March 2026 across Approach, Application, and Assessment.These dimensions cover agent construction, security tasks, and datasets, metrics, safety measures, baselines, and protocols.
  • Key finding: The synthesis finds agents able to act, but not yet agents whose authority is bounded or whose behavior is auditable.The authors extend this observation into challenges involving evidence-centered architecture, risk-aware autonomy, benchmarks, trajectory assessment, and human-agent collaboration.

2 Background and Related Work

LLM-based agents extend language models into systems that perceive artifacts, reason over state, plan actions, invoke tools, and learn from feedback. In software and systems security, they support tasks spanning vulnerability discovery, offensive security, defensive monitoring, low-level artifact analysis, and repair, while prior surveys have not unified agent construction, applications, and assessment at this granularity.

  • LLM-based agents: An LLM-based agent operates in an observe-reason-act loop rather than returning a single response to one prompt.The loop observes state, selects and executes an action, then reasons over the result before continuing.
  • Agent architecture: Agent designs may use one model for the full task loop or multiple specialized agents connected through communication, shared memory, or delegation.The taxonomy distinguishes these architectural choices from standalone LLM usage.
  • Perception and memory: Memory preserves state through working context, external short-term traces, or persistent stores such as vector databases and knowledge graphs.This continuity lets agents remember scanned assets, failed hypotheses, and supporting evidence.
  • Perception and memory: Perception transforms source code, binaries, traces, logs, packets, and alerts into reasoning-ready context through prompts, RAG, summarization, or semantic matching.These mechanisms mediate how heterogeneous security artifacts reach the language model.
  • Reasoning and planning: Reasoning and planning include ReAct, explicit chain-of-thought, reflection, dynamic planning, hierarchical decomposition, and failure backtracking.These paradigms determine how agents select and revise subsequent actions.
  • Applications: Security applications span vulnerability discovery, penetration testing and exploit development, defensive monitoring, binary and malware analysis, and repair or hardening.The tasks range from proactive analysis to reactive response and defensive modification.
  • Related work: The survey uniquely couples agent construction, security applications, and assessment, including perception, memory, action space, orchestration, self-improvement, datasets, metrics, safety, and baselines.Prior surveys did not characterize these dimensions together at the stated granularity.

3 Overview of Survey Methodology

The survey uses a structured methodology organized around three research questions covering agent approach, security application, and assessment. It searches and screens peer-reviewed work, derives a taxonomy, maps papers to it, and analyzes cross-paper patterns, limitations, and future directions.

  • Research questions: RQ1 asks how security agents are constructed and operated across architecture, perception, memory, planning, action space, orchestration, and self-improvement.This question defines the survey’s Approach dimension.
  • Research questions: RQ2 asks which software and systems security tasks agents address and how effort is distributed across the security lifecycle.This question defines the Application dimension.
  • Research questions: RQ3 asks how agent behavior is assessed through datasets, outcome and trajectory metrics, safety measures, baselines, and experimental protocols.This question defines the Assessment dimension.
  • Methodology: The authors search peer-reviewed studies from January 2023 through March 2026 using keyword search, snowballing, and manual eligibility filtering.The process produced the final survey corpus.
  • Methodology: The search and screening process produced a final set of 100 papers for taxonomy attribution and analysis.The authors then mapped selected papers to taxonomy items and analyzed patterns, limitations, challenges, and future directions.

4 Literature Search

The literature search combines broad database queries, manual screening, forward and backward snowballing, and explicit inclusion and exclusion criteria. These stages expand and then filter the candidate pool to a final corpus of 100 papers.

  • Search scope: The search targets papers that use or develop LLM-based agents for software or systems security tasks, excluding work focused primarily on securing agents or LLMs.This establishes the survey’s instrument and domain scope.
  • Keyword search: Queries combine agent-related terms with security-task terms such as vulnerability detection, penetration testing, exploit analysis, fuzzing, and intrusion detection.The keyword groups were searched across Google Scholar, ACM DL, IEEE Xplore, Scopus, and DBLP.
  • Initial screening: The initial keyword search returned 108 candidate papers, which were manually inspected and filtered for relevance.Papers clearly outside the target scope were removed during this stage.
  • Snowballing: Forward and backward snowballing from 48 seed papers expanded the candidate pool to 144 papers.Backward snowballing followed references, while forward snowballing followed later citing papers.
  • Eligibility criteria: Papers were included only if they used an LLM-based agent for software or systems security and were published between January 2023 and March 2026.Short, non-English, and exclusively preprint studies were excluded, alongside papers failing the scope criteria.
  • Final corpus: After applying the eligibility criteria, the final corpus contained 100 papers whose designs, applications, datasets, metrics, findings, limitations, and future work were recorded.This corpus forms the basis for the survey analysis.

5 Survey Taxonomy Derivation

The survey derives a reusable taxonomy by coding recurring concepts in selected papers, then consolidating them into three aspects: Approach, Application, and Assessment. These aspects cover agent design, security tasks, and assessment data, metrics, safety, and baselines.

  • The taxonomy is organized around Approach, Application, and Assessment.These high-level aspects structure the survey’s characterization of LLM-based agents for software and systems security.
  • The authors first identify candidate attributes and items by reading and coding selected papers, then generalize and consolidate them into a reusable taxonomy.Paper-specific terminology is normalized through iterative merging of similar terms and resolution of overlapping categories.
  • Approach: Approach characterizes seven technical attributes, including architecture, perception, memory, reasoning and planning, action space, workflow orchestration, and self-improvement.Together, these attributes treat an agent as an operational system rather than a single prompt-response model.
  • Application: Application organizes security tasks into domains including vulnerability detection, exploit development, penetration testing, repair, fuzzing, malware analysis, reverse engineering, and defensive operations.The domains are organized by task objective so they remain applicable across different architectures, datasets, and tools.
  • Assessment: Assessment distinguishes dataset characteristics and four metric families: outcome-oriented, safety-oriented, process-oriented, and comparative measurements.Process and trajectory metrics include tool calls, hallucinations, cost, iterations, artifacts, and explanations, while comparative metrics capture baselines such as humans and non-agentic systems.

6 Paper Attribution

The survey maps the collected literature to its taxonomy and reports publication, application, assessment, and architectural patterns. The dataset contains 100 papers from 2023 through March 2026 across 73 venues, with diverse agent architectures and increasing use of collaboration.

  • Publication venues: 100 papers published between 2023 and March 2026 span 73 distinct publication venues.
  • Publication venues: The 15 venues represented by at least two papers account for 42 papers, while 58 remaining venues contribute one paper each.
  • Publication venues: Five security venues collectively account for 16% of surveyed papers, with TrustCom contributing 5 and USENIX Security 4.
  • Agent architecture: Surveyed agents use single- or multi-agent architectures with varied internal structures and backbone-LLM roles.The attribution framework records only capabilities explicitly described in each paper.
  • Agent architecture: Multi-agent systems increasingly use collaborative, hierarchical, or peer-to-peer structures for complex security tasks.Examples include task-specialized agents connected through a RAG-augmented pipeline and equal-status agents resolving disagreements through debate or voting.
  • Agent architecture: Reinforcement-learning multi-agent designs achieve coverage gains of 12–27% over single-agent baselines.

6.2.5 Agent action space.

Agents use diverse tool-mediated action spaces spanning static and dynamic analysis, execution environments, orchestration, and human-facing review. The surveyed evidence emphasizes that tool use supports concrete validation, while process and safety assessment remain uneven.

  • Action-space categories: Action spaces span software-analysis utilities, security platforms, multi-agent coordination mechanisms, and human-facing review steps.The survey summarizes these categories in Table 8.
  • Static analysis: Static-analysis actions first filter likely relevant code regions, while data-flow evidence and validation assess whether suspected bugs are feasible along program paths.RepoAudit uses demand-driven data-flow evidence and validation for this purpose.
  • Formal validation: PropertyGPT combines generation with compilation, static analysis, ranking, and a prover to determine whether smart-contract properties are usable for formal verification.Formal-verification tools check generated or retrieved properties against precise semantic requirements.
  • Binary analysis: Binary-analysis agents inspect functions and control-flow structures and validate candidate bugs through analysis interfaces rather than unconstrained text generation.ClearAgent illustrates how tool-mediated binary understanding changes the LLM’s role.
  • Dynamic analysis: Dynamic tools let agents generate or mutate inputs, inspect execution states, run risky behavior in sandboxes, and observe network traffic and protocol behavior.These tools support fuzzing, debugging, controlled execution, and packet or flow analysis.
  • Reported outcomes: Application results include 78.4% precision at roughly $2.54 per project for vulnerability auditing and over 90% repair of 178 real-world vulnerabilities.Other reported outcomes include 34 zero-day vulnerabilities across 20 applications, 14 real-world zero-day exploits, and 4.3× improvement over a single-agent baseline.

6.4.2 Dataset.

The survey characterizes datasets by origin, security task, and cross-cutting properties such as realism, label quality, scale, and modality. Real-world data improves ecological validity, whereas synthetic and semi-synthetic sources improve control, repeatability, and safety, creating comparability and scale–realism trade-offs.

  • Characterization axes: Dataset characterization records origin, exercised security task, and properties including realism, label quality, scale, modality, and accessibility.The taxonomy treats these as integral attributes for mapping papers to assessment datasets.
  • Real-world sources: Real-world datasets reflect repository layout, service state, execution dependencies, and operational constraints that affect agent trajectories.BountyBench covers detection, exploitation, and patching across 25 real-world systems; CVE-Bench reproduces vulnerable web applications in sandboxes.
  • Competition-derived sources: Competition-derived datasets provide defined success conditions, reusable tasks, and writeups but may reward puzzle-solving behavior unlike production intrusion.ClearAgent is assessed on binary challenges from NYU CTF Bench.
  • Synthetic and controlled sources: Synthetic and controlled environments provide tight control over vulnerabilities, attack paths, services, state transitions, and agent-security tasks.Their advantage is measurability, although they may underrepresent real-world messiness.
  • Security-task coverage: Datasets span vulnerability detection, penetration testing, smart-contract analysis, repair, defense, fuzzing, malware analysis, and SOC operations.The task axis distinguishes assessments using similarly realistic artifacts for different capabilities.
  • Assessment trade-offs: Real-world sources improve ecological validity, whereas synthetic and semi-synthetic datasets improve control, repeatability, and safety.Results may therefore remain non-comparable even within the same application area.
  • Corpus distribution: 36% of papers use fully real-world datasets and 47% use semi-synthetic datasets, while large datasets are relatively rare and concentrated in vulnerability detection and smart-contract analysis.The survey identifies a recurring trade-off in which scale is exchanged for realism and measurement depth.

7.1 Agentic Approach to Software/Systems Security

LLM-based security agents have shifted from standalone text generation toward modular systems that plan, act, observe, and revise. However, architecture and workflow design are more mature than uncertainty handling, safety controls, human governance, memory management, and durable self-improvement.

  • Architecture and orchestration: 91% of surveyed papers use modular pipelines, but this modularity is often operational rather than principled.Most systems separate planning, execution, and related roles without making failure modes sufficiently auditable.
  • Perception and memory: 60% use in-context working memory and 57% use long-term or persistent memory, but explicit memory writes appear in only 7%.Security-specific stores include tool documentation (49%), execution traces and fuzzing logs (33%), vulnerability knowledge bases (25%), and repositories or patch histories (23%).
  • Perception and memory: 100% of systems ingest LLM-readable inputs directly, while retrieval-augmented perception appears in 30% and summarisation/compression in 35%.Perception also uses tool-mediated observation (68%), environment observations (66%), structured data (60%), and system logs (55%).
  • Workflow governance: 95% of papers report fully autonomous workflows, while human-on-the-loop, human-in-the-loop, and interactive workflows appear in 8%, 5%, and 3%, respectively.These systems may affect real systems, generate exploits, patch code, or interpret incidents, yet only a small fraction structurally encode human review or approval.
  • Self-improvement: Learning from feedback within a session appears in 32% of papers, but storing successful trajectories appears in 3% and updating fine-tuning data in 2%.Most agents adapt locally through tool feedback or iterative prompting rather than accumulating durable experience across tasks.

7.2 Applications of Agentic Approaches in Software/Systems Security

LLM-based security agents now span a broad range of tasks, but applications remain concentrated in vulnerability-centered and offensive workflows. Lifecycle integration, operational realism, specialized domains, and application-level safety remain limited.

  • Application landscape: Agents address diverse software and systems security tasks, including repository auditing, executable CVE exploitation, bug-bounty workflows, and patch generation.Examples include RepoAudit, CVE-Bench, BountyBench, and PatchAgent.
  • Application landscape: Current applications concentrate on vulnerability detection, penetration testing, exploitability assessment, and repair, while defensive, low-level, and governance-heavy tasks remain less mature.The surveyed landscape is especially shaped by finding, exploiting, or fixing vulnerabilities.
  • Lifecycle coverage: Many systems address isolated lifecycle stages but do not connect discovery, exploitation, remediation, validation, reporting, and monitoring into one workflow.BountyBench explicitly separates detect, exploit, and patch tasks, but lifecycle-spanning designs remain uncommon.
  • Risk and maturity: Offensive workflows such as penetration testing, red-teaming, CTF solving, exploit generation, and web vulnerability scanning are better represented than post-exploitation or autonomous defense.Post-exploitation tasks are less developed, while defensive systems often remain in simulation or controlled-alert settings.
  • Domain coverage: Agents are strongest with natural language, source code, or familiar command output, and weaker when tasks require specialized representations or stateful execution semantics.Binary analysis and reverse engineering require disassembly, decompiled code, debugger state, calling conventions, and low-level execution artifacts.
  • Operational constraints: Application success depends on workflow integration, production realism, and explicit authorization boundaries because controlled benchmarks omit authentication, noisy telemetry, legal limits, and human approval chains.Dual-use capabilities require clear boundaries, yet application-level risk management is not consistently treated as part of task definition.

7.3 Assessment of Agentic Approaches in Security Applications

Assessment practice has moved beyond single outcome scores, but coverage remains uneven across effectiveness, trajectories, safety, datasets, and baselines. The field still lacks consistent ways to evaluate agent behavior, realism, reproducibility, and cost.

  • Assessment coverage: Task-level success, classification, time, and resource metrics are relatively common, while tool-call quality, safety, reproducibility, and human utility remain sparse.The survey identifies uneven use across task effectiveness, trajectories, safety and reliability, baselines, and datasets.
  • Safety and reliability: Consistency appears in 18% of papers, attack success rate in 10%, defense success in 8%, prompt-injection robustness in 6%, scope containment in 5%, and overhead in 3%.The most common safety and reliability family appears in fewer than one fifth of the corpus.
  • Datasets and ground truth: 50% of papers use purpose-built programs or environments, 28% production systems, 23% curated public artifacts, and 17% threat feeds; 36% use fully real-world datasets and 47% semi-synthetic datasets.Only 16% use explicitly expert- or reference-verified labels, and open-ended tasks require stronger oracles than surface fluency.
  • Baselines and comparability: 36% of papers compare with non-AI tools, 28% with other agent systems, 27% with prior ML/DL methods, 26% include ablations, 12% compare with humans, and 9% with plain LLMs.Direct human and plain-LLM baselines remain uncommon.
  • Reproducibility and cost: Cost/token efficiency appears in 32% of papers, but safety overhead appears in only 3%, and measurement conventions remain inconsistent for repository-scale and continuous security pipelines.The survey recommends multidimensional profiles, released traces, safety assessment, stronger baselines, and explicit reporting of cost and variance.

7.4 Cross-Cutting Future Research Directions

Future research should co-design agent architecture, security applications, and assessment protocols around evidence, authority, risk, and operational context. The agenda emphasizes lifecycle-aware systems, realistic safe benchmarks, human governance, and reusable engineering principles.

  • Co-design: Agent approach, target application, and assessment protocol should be designed together because architecture determines feasible tasks, context determines relevant risks, and assessment determines trustworthiness and reproducibility.The survey frames these three perspectives as interdependent rather than independent research tracks.
  • Domain-specific design: Different security applications require different authority boundaries, evidence models, success conditions, and failure-handling mechanisms rather than only different prompts and tools.The survey specifically contrasts penetration testing, repository auditing, SOC triage, and patch generation.
  • Evidence management: Security agents should maintain explicit evidence graphs or ledgers recording each claim’s source, time, producer, confidence, validation status, and downstream uses.Benchmarks should evaluate whether supporting evidence is complete, faithful, and minimally sufficient, not only whether final answers are correct.
  • Governance and safety: Risk-aware autonomy levels should permit more autonomy for low-risk tasks, require human-on-the-loop review for medium-risk tasks, and constrain high-risk tasks such as exploit execution.Autonomy levels should be reported in system designs and benchmark setups, then stress-tested against adversarial inputs.
  • Benchmark ecosystems: Benchmark ecosystems should be realistic, reproducible, and safe, using executable instrumented environments with task metadata, allowed actions, evidence expectations, validation oracles, and baselines.Existing studies often optimize only part of this realism–reproducibility–safety triangle.
  • Human and resource factors: Future assessments should include human-centered measures and explicit resource reporting covering token usage, wall-clock time, tool-call count, infrastructure, memory growth, and run variance.High-impact systems should preserve human governance over reasoning and authority while improving decisions, reducing workload, and preserving accountability.

8 Threats to Validity

The survey’s validity is constrained by unstable terminology, selective literature coverage, subjective taxonomy and attribution decisions, and limited comparability across reported results. The authors mitigate these threats through broader searches, manual screening, explicit coding, full-text review, and pattern-focused analysis.

  • Literature coverage: Inconsistent terminology may cause relevant systems to be missed or non-agentic systems to be included, because papers use overlapping labels and apply “agent” inconsistently.The search combined multiple keywords, manual screening, and snowballing to recover papers that keyword search alone might miss.
  • Scope decisions: Restricting the corpus to peer-reviewed papers from January 2023 through March 2026 improves consistency and inspectability but may exclude emerging systems, industry reports, repositories, and recent preprints.The results represent a structured view of the selected literature rather than an exhaustive inventory of deployed or publicly released systems.
  • Taxonomy derivation: The manually derived taxonomy reflects recurring corpus distinctions, but alternative researchers could choose different abstraction levels or groupings.The authors treat it as an evolving synthesis that may need attributes for memory governance, supply-chain risk, tool permissions, and deployment monitoring.
  • Paper attribution: Multi-label paper attribution requires judgment because individual systems can combine retrieval, planning, execution, verification, reporting, multiple tasks, and multiple assessment properties.The authors allowed multiple attributions and reviewed full texts, methods, appendices, and reported measures before retaining blank metric rows.
  • Result interpretation: Results across papers are often not directly comparable because differences may reflect models, tools, environments, context, attempts, budgets, or changing benchmark versions rather than architecture alone.The analysis therefore emphasizes recurring patterns instead of ranking individual systems, though industrial practice and unpublished negative results may be underrepresented.

9 Conclusion

The survey finds that LLM-based security agents have rapidly become modular, tool-mediated, and often autonomous, but remain less developed in bounded authority, inspection, and safety assessment. Applications and evaluations are uneven across the security lifecycle, motivating co-design of architectures, applications, and assessments around process quality, safety, cost, and human oversight.

  • The survey covers 100 peer-reviewed papers published between January 2023 and March 2026, organized around Approach, Application, and Assessment.
  • 91% of agents adopt modular pipelines, 88% place the backbone LLM in planner-and-actor roles, and 95% support fully autonomous operation.
  • Explicit critic or verifier roles appear in 15% of systems, guardrail layers in 13%, and human review points in 13%.
  • Applications concentrate on the vulnerability lifecycle and offensive assessment, while post-access phases, low-level artifact analysis, and governance tasks remain comparatively undeveloped.
  • Future progress requires co-designing architectures, applications, and assessments to address evidence, risk, authority boundaries, error recovery, human communication, safety, cost, and oversight.
Loading 2608.28490v1…