PaperlayerView exact revision on arXiv

Source-linked AI summary

SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response

Lehan Wang, Boli Chen, Ruixue Ding, Pengjun Xie, Jinwei Huang, Zhendong Liu, Shuo Wang, Tao Lei, Xin Ouyang, Xiaomeng Li

arXiv:2607.26791v1cs.CRcs.AIcs.CL

TL;DR

Existing cybersecurity benchmarks largely assess agents before compromise, leaving post-compromise incident response underexplored. SecRespond benchmarks this workflow across compromised hosts and finds that agents detect alerted problems but struggle with proactive investigation and complete remediation.

  • Problem

    Existing cybersecurity benchmarks emphasize pre-compromise, clean environments, leaving evidence about agents’ post-compromise incident-response capabilities limited.

  • Method

    SecRespond evaluates agents analyzing compromised-host disk snapshots and security-product findings to produce forensic reports and remediation plans across 10 diverse cyber ranges.

  • Results

    Agents reliably uncover alerted problems but struggle with proactive forensic investigation and complete remediation; the strongest model achieves 72.4% averaged over detection and planning.

  • Takeaways & Limitations

    Current agents show imbalanced incident-response capabilities and remain limited in supporting comprehensive post-compromise response.

  • Takeaways & Limitations

    The evaluation uses controlled experimental environments and synthetic artifacts rather than production systems.

Abstract

from arXiv · show

Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities. However, existing cybersecurity benchmarks focus on pre-compromise settings where agents are placed in a clean and idealized environment before an attack occurs. This leaves the post-compromise setting underexplored. To address this gap, we introduce SecRespond, the first benchmark for evaluating LLM agents on the post-compromise incident-response workflow. Given a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks reported by a host security product, agents are required to produce forensic reports on intrusions, baseline risks, and vulnerability risks, together with a remediation plan. We instantiate this task across 10 cyber ranges, each constructed from a distinct compromised cloud host, spanning 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. We evaluate 23 frontier LLMs on the OpenCode agent harness. Experimental results show that although current agents can reliably uncover the problems exposed by alerts, they struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans, with no model achieving complete detection and remediation on any single range. This reveals a fundamental bottleneck in building agents for real-world incident response. The benchmark is publicly available at https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond.

1 Introduction

SecRespond addresses the gap between clean-environment security benchmarks and real-world post-compromise incident response, where agents must investigate compromised hosts and respond to alerts and intrusion artifacts. The benchmark evaluates whether LLM agents can detect incidents, assess risks, and complete remediation across diverse cloud-host scenarios.

  • Motivation: LLM agents with host-artifact and CLI access are increasingly expected to assist with real-world security operations, making their security capabilities important to assess.Their iterative reasoning and autonomous tool-use abilities support complex professional tasks, including cybersecurity.
  • Motivation: Existing security benchmarks primarily evaluate agents before compromise, in clean environments focused on finding, exploiting, or patching weaknesses.Defensive benchmarks also simplify incident response by reasoning over isolated alerts or system logs.
  • Motivation: Post-compromise evaluation tests whether agents can discover problems on compromised disks and respond effectively to live alerts and intrusion artifacts.This capability is presented as a prerequisite for deploying agent assistance in production security operations.
  • Benchmark: SecRespond is the first benchmark for post-compromise incident response, providing compromised-host disk snapshots alongside alerts, vulnerability scans, and baseline checks.Agents must investigate the host, reconstruct events, and produce progress and incident-response reports.
  • Findings: Across 23 LLMs evaluated with OpenCode, agents reliably surface alerted problems but struggle with proactive forensic investigation, complete remediation, and balanced incident-response capabilities.Each task was independently judged by three strong proprietary LLMs to mitigate individual evaluator bias.
  • Benchmark: The benchmark contains 10 cloud-host cyber ranges spanning 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems.Its evaluation framework further uses 280 expert-designed checkpoints mapped to 52 capability items in a five-dimensional taxonomy.

2 Related Work

Prior agent benchmarks span coding, shell execution, GUI interaction, and cybersecurity evaluation, but cybersecurity benchmarks often emphasize knowledge probing or deterministic verification. SecRespond addresses open-ended post-compromise investigation, requiring evaluation methods suited to report-based outputs.

  • General agent benchmarks: Existing agent benchmarks evaluate coding, long-horizon software tasks, interactive shell execution, web navigation, and desktop environments.SWE-Bench tests GitHub issue resolution; SWE-Bench-Pro extends this to multi-file, substantial-code problems; Terminal-Bench targets terminal execution; WebArena and OSWorld evaluate web and desktop interaction.
  • Cybersecurity benchmarks: Cybersecurity evaluations cover offensive and defensive settings, but early benchmarks often probe security knowledge or constrain agents to containerized tasks.The supplied passages distinguish curated multiple-choice knowledge assessments from agent evaluations conducted inside Docker containers.
  • Evaluation protocols: Cybersecurity benchmarks predominantly use deterministic verification, including multiple-choice accuracy, CTF flag matching, or execution-output verification.These protocols rely on fixed ground truth and are designed for structured outcomes rather than open-ended investigation reports.
  • Evaluation protocols: Open-ended incident-response reports require evaluation methods beyond conventional deterministic checks, motivating an LLM-as-a-Judge approach that decomposes outputs into verifiable units.The passage states that fixed-ground-truth protocols cannot be directly applied to this benchmark’s report-based task.

3 SECRESPOND Benchmark

SecRespond benchmarks post-compromise incident response by asking agents to investigate compromised-host snapshots and security analytics, then produce forensic findings and remediation plans. It operationalizes this evaluation through diverse cyber ranges, a five-dimensional capability taxonomy, and fine-grained scored checkpoints.

  • Benchmark task: Agents investigate forensic disk snapshots and host-security analytics to reconstruct intrusions and produce reports on intrusion, baseline, vulnerability, and remediation risks.The snapshot contains naturally generated artifacts such as logs, configuration changes, and residual files, while analytics include real-time alerts and scan findings.
  • Cyber-range construction: 10 cyber ranges span 4 entry points, 5 operating systems, and 21 ATT&CK techniques, reproducing complete multi-stage attacks from initial access through impact.Each range uses a frozen, reproducible, read-only forensic disk snapshot from a fully compromised cloud host.
  • Cyber-range construction: Each range combines blueprint, instance, and checklist layers, enforcing real vulnerabilities, network-delivered attacks, and naturally produced intrusion traces.The construction pipeline contains 12 stages and includes expert auditing and validation of attack-chain plausibility and results.
  • Capability taxonomy: The CAP taxonomy organizes incident-response capabilities into five dimensions: ENT, PER, BAS, VUL, and Q, comprising 52 capability items.The dimensions cover intrusion artifacts, persistence, baseline configuration risks, vulnerability confirmation, and investigation-and-response quality.
  • Evaluation protocol: 280 fine-grained checkpoints specify required evidence for range-specific findings, enabling comparable capability-level CAP-scores across ranges.Checkpoint scoring separates detection and planning axes before aggregating results for each capability item.

4 Experiments

Experiments show that agents detect exposed problems more reliably than they plan complete remediation, with performance varying by model, attack-chain complexity, and capability dimension. Designed skills substantially improve weak planning, while judge evaluations closely agree with human experts and show little self-preference bias.

  • Overall performance: Detection consistently outperforms planning across all ranges, with GPT-5.5 scoring 70.7% versus 36% and Claude Sonnet 4.5 showing an 8.5% gap.Agents often identify problems but fail to complete follow-up actions such as credential rotation, outbound-channel blocking, or service-health verification.
  • Overall performance: Claude Opus 4.7 leads average range-level CHK-scores at 79.0% for detection and 65.7% for planning, followed by Claude Opus 4.6, GLM-5.1, and Qwen3.7 Plus.The cited successors score 78.2%/58.0%, 76.3%/59.2%, and 75.6%/58.8%, respectively.
  • Model evolution: Performance generally improves across model versions but not universally: Claude Opus rises from 69.5%/56.4% in Opus 4.5 to 79.0%/65.7% in Opus 4.7.GLM-5.1 surpasses GLM-5 by 11.4%/13.3%, reaching 76.3%/59.2% versus 64.9%/45.9%.
  • Range difficulty: No model achieves complete detection and remediation on any range, and performance declines as attack chains become longer and broader.Models perform better on single-entry-point ranges such as Log4j-RCE, Docker-Escape, and Redis-RCE.
  • Capability dimensions: Models identify intrusion entities more reliably than persistence mechanisms, while standardized baseline and vulnerability remediation is stronger than intrusion-entity cleanup.Qwen3.7 Plus and Claude Opus 4.6 reach 88.4% and 86.0% on ENT; Claude Opus 4.7 reaches 74.8% on BAS and 72.6% on VUL.
  • Baselines and interventions: Agents substantially outperform the agentless detection baseline on average, especially for persistence detection, while designed skills improve CHK-scores most strongly on planning.GPT-5.4 rises from 26% to 83% on SSH-Miner and increases by 43% on Docker-Escape; GPT-5.4 Pro gains 45% on Jenkins-RCE and 52% on SSH-Miner.
  • Evaluation validity: LLM judges agree closely with human experts, with Pearson correlation 0.96 and quadratic-weighted Cohen’s kappa κ of 0.94; model rankings are not affected by self-preference bias.The human-expert comparison covers 60 checkpoints across all 10 ranges, with detection scored from 0 to 3 and planning from 0 to 2.

5 Conclusion

SECRESPOND is introduced as the first benchmark for evaluating LLM agents on post-compromise incident response. It requires agents to analyze compromised-host evidence and produce forensic reports plus a remediation plan.

  • Benchmark scope: SECRESPOND evaluates LLM agents on the post-compromise incident-response workflow.The paper presents it as the first benchmark targeting this setting.
  • Task inputs: Each task combines a forensic disk snapshot with alerts, vulnerability scans, and baseline checks from a host security product.Agents analyze these inputs as evidence from a compromised host.
  • Required outputs: Agents must report intrusions, baseline risks, and vulnerability risks, together with a remediation plan.These outputs define the benchmark’s required incident-response deliverables.

Ethics Statement · A Details on Range Construction · B Details on Capability Taxonomy

The benchmark uses authorized, isolated, synthetic cyber ranges and evaluates third-party models solely for academic comparison. Its ranges cover diverse post-compromise attack chains across Linux and Windows hosts, while the capability taxonomy maps checklist checkpoints to capability items.

  • Ethics Statement: Third-party models were evaluated solely for academic research in a controlled environment, and model names and scores do not imply provider endorsement.The reported results are intended for capability comparison and technical discussion.
  • Ethics Statement: All benchmark data came from authorized, isolated cyber ranges built by provisioning, compromising, and snapshotting hosts rather than collecting production data.Released forensic and security-product artifacts are synthetic and contain no customer, production, or real-world business-system data.
  • A Details on Range Construction: The SSH-Miner range models CentOS 7 compromise through brute-forced root SSH access, cryptomining, three persistence mechanisms, and authentication-log clearing.The agent receives a post-attack disk snapshot and alerts covering the brute-force success, miner execution, suspicious crontab, and autorun activity.
  • A Details on Range Construction: Redis-RCE and Docker-Escape model Ubuntu compromise through unauthenticated internet-facing services, followed by host access, cryptomining, and layered persistence.Redis-RCE abuses a root-running Redis server; Docker-Escape exploits an unauthenticated Docker Remote API with a privileged host-filesystem mount.
  • A Details on Range Construction: Jenkins-RCE, Shiro-Fastjson, and Log4j-RCE represent application-layer exploitation leading to code execution, privilege escalation or credential theft, cryptomining, and persistence.The ranges use Jenkins authorization misconfiguration, Shiro deserialization, and Log4j JNDI injection, respectively.
  • A Details on Range Construction: Next.js-RCE and NPM-Worm cover Ubuntu application compromises involving deserialization or webhook abuse, followed by root escalation, credential or network propagation, and mining.Next.js-RCE includes an LD_PRELOAD rootkit, while NPM-Worm propagates through malicious package hooks and uses a leaked cloud access key.
  • A Details on Range Construction: ASP.NET-ViewState and RDP-Service-Abuse extend the ranges to Windows Server 2019, using forged ViewState or exposed RDP and weak service permissions for SYSTEM execution.The scenarios include credential dumping, persistence, command-and-control activity, and Windows-specific artifacts or logging conditions.
  • B Details on Capability Taxonomy: Table 6 maps capability-taxonomy items to range checklists by listing the checkpoint indices, such as CHK-n, associated with each item.The table provides the correspondence between capability evaluation items and concrete range checkpoints.

C Implementation Details · C.1 Task Prompt for Linux Operating System

The Linux task prompt assigns agents forensic analysis of a compromised server using a read-only disk snapshot and security-platform data. It requires a comprehensive incident-response report, specified output files, evidence-based investigation, and prioritized, preservation-aware remediation.

  • C.1 Task Prompt for Linux Operating System: Agents analyze a compromised server’s read-only forensic disk snapshot alongside security platform alert data.The workspace is an analysis machine rather than the compromised server.
  • C.1 Task Prompt for Linux Operating System: The task requires a comprehensive incident response report based on the provided disk snapshot and security platform data.All findings must be grounded in disk evidence and SAS data.
  • C.1 Task Prompt for Linux Operating System: Inputs include the forensic disk snapshot, SAS alerts, SAS vulnerability scan, and SAS baseline check.The specified locations are DISK_PATH/, SAS_PATH/alerts.json, SAS_PATH/vulns.json, and SAS_PATH/baselines.json.
  • C.1 Task Prompt for Linux Operating System: Outputs must be written to OUTPUT_PATH/ and include progress, intrusion, vulnerability, baseline, and remediation reports.The remediation plan is natural language and prioritized P0-P3.
  • C.1 Task Prompt for Linux Operating System: The intrusion report must provide full intrusion analysis with an attack chain, while vulnerability and baseline reports document their respective assessments.progress.md records the investigation timeline and checklist.
  • C.1 Task Prompt for Linux Operating System: Agents must not SSH into any machine, fabricate CVEs or vulnerabilities, or treat uncertain findings as established facts.The prompt requires stating uncertainty with confidence levels.
  • C.1 Task Prompt for Linux Operating System: Remediation recommendations must preserve forensic evidence by favoring backup or quarantine over deletion.This constraint applies specifically to remediation-plan.md.

C.2 Task Prompt for Windows Operating System · C.3 Evaluation Prompt

The Windows task prompt directs agents to analyze a read-only forensic disk snapshot and SAS data using Linux tools, produce five reports, and write evidence-preserving, execution-safe remediation plans. The evaluation prompt scores each checklist item for detection quality and remediation correctness/completeness using only explicitly reported evidence.

  • C.2 Task Prompt for Windows Operating System: Agents analyze a read-only Windows filesystem dump alongside SAS alerts, vulnerability scans, and baseline checks on a Linux analysis machine.They must base findings on disk evidence and SAS data, without SSH access or fabricated vulnerabilities.
  • C.2 Task Prompt for Windows Operating System: The prompt permits direct use of find, grep, cat, stat, and strings against the forensic directory tree.It maps these tools to Windows artifacts including scheduled tasks, services, registry autostarts, event logs, hives, plugins, and WMI subscriptions.
  • C.2 Task Prompt for Windows Operating System: Agents must deduplicate evaluation-platform traces from attacker activity and avoid recommending blocking benchmark infrastructure or deleting marked control keys.The prompt identifies authorized_keys entries carrying BENCHMARK-CONTROL-DO-NOT-REMOVE as platform artifacts.
  • C.2 Task Prompt for Windows Operating System: Every remediation item requires a risk_class and automation level because downstream agents execute the plan as authoritative production-server instructions.Risk classes include auth, network, account, service, data, runtime, and none; automation levels are auto, needs_approval, and blocked.
  • C.2 Task Prompt for Windows Operating System: Commands that could sever the current control plane must be replaced with a risk description and recommended manual-intervention procedure.Examples include expiring passwords, stopping sshd, or deleting a legitimate administrator key; remediation should preserve evidence through backup or quarantine over deletion.
  • C.2 Task Prompt for Windows Operating System: Required outputs are progress.md, intrusion-report.md, vuln-report.md, baseline-report.md, and a prioritized P0-P3 remediation-plan.md.The reports cover investigation progress, intrusion analysis, vulnerability verification, baseline assessment, and structured remediation.
  • C.3 Evaluation Prompt: For each checklist item, evaluation assigns detection scores from Discovery, Evidence, and Attribution, and plan scores from Correctness and Completeness.Detection totals up to 3 points and plans up to 2 points; detection-only items receive N/A for plan scoring.
  • C.3 Evaluation Prompt: Evaluators read all five reports, score only explicitly documented findings, cite concrete evidence, and record per-checkpoint rationales plus overall scores.They must not credit unreported work and should note environment limitations while retaining a zero score when appropriate.

C.4 Evaluation Skill

The Security Evaluation SKILL uses LLM-as-a-Judge to score detection and remediation independently, item by item, against checklist criteria. It applies explicit deductions and diagnostic modes to distinguish analysis weaknesses from planning weaknesses.

  • Scoring dimensions: Each checkpoint is scored independently across detection and plan dimensions, which are never merged into a single total.Detection assesses issue identification, evidence, and attribution; plan assesses technical correctness, completeness, verification, and safety.
  • Scoring dimensions: Detection scoring awards up to 3 points for discovery, evidence, and attribution, while plan scoring awards up to 2 points for correctness and completeness.Plan-only items are excluded from detection aggregation, and detection-only items are excluded from plan aggregation.
  • Negative deductions: Fabrication sets the detection score to 0, wrong attribution sets attribution to 0, and dangerous advice sets the plan score to 0.Dangerous advice covers operations that could cause data loss or service disruption without flagging the risk.
  • Diagnostic modes: Diagnostic modes classify performance as strong or weak using a ≥70% threshold, including All-round, Plan bottleneck, Analysis blind spot, and Blind fix.The modes identify whether weaknesses primarily concern plan generation, foundational analysis, or untrustworthy fixes lacking analysis.

D Supplementary Experimental Results · D.1 Per-Range Performance Analysis

Across ten compromised-host ranges, models generally recover alert-visible intrusions, vulnerabilities, and persistence artifacts, but repeatedly miss silent host-level activity and fail to produce complete, verified remediation plans. The per-range results show that performance depends strongly on artifact type and attack path, with web-application and obvious post-compromise evidence easier to analyze than stealth, initial-access, or privilege-abuse mechanisms.

  • D.1 Per-Range Performance Analysis: SSH-Miner and Redis-RCE models identify the main intrusion paths, baseline risks, and persistence artifacts, but overlook silent residues and latent persistence beyond reported alerts.SSH-Miner weaknesses include brute-force tool residue and bashrc persistence; Redis-RCE models distinguish unauthorized Redis access from SSH brute force or a CVE exploit but still show investigation gaps.
  • D.1 Per-Range Performance Analysis: Docker-Escape models recover the visible attack chain but score close to zero on the proactive comprehensive persistence scan, indicating narrow artifact-focused investigation.They flag the miner, backdoor container, unauthenticated Docker TCP API, and host-level persistence while generally missing persistence artifacts outside the observed chain.
  • D.1 Per-Range Performance Analysis: Jenkins-RCE models reliably detect alert-visible attacks but miss inconspicuous persistence, while remediation plans rarely become complete and verified.Low planning scores on CHK-10, CHK-21, and CHK-23 demonstrate the remediation gap.
  • D.1 Per-Range Performance Analysis: Shiro-Fastjson detection is strong: most models recover webshells, mining activity, OS persistence, the Shiro entry point, Java process chain, privilege escalation, and attack timeline.The main weaknesses are not specified in the supplied continuation, so this point reports only the documented strengths.
  • D.1 Per-Range Performance Analysis: Log4j-RCE models broadly discover active intrusions and the Log4j vulnerability, yet latent login-triggered persistence remains a systematic blind spot.The profile.d backdoor and injected SSH key are explicitly identified as examples of this weakness.
  • D.1 Per-Range Performance Analysis: Next.js-RCE models thoroughly inspect the web application and trace its RSC deserialization exploit, but rarely pivot to surrounding-host artifacts.CHK-04, CHK-05, CHK-06, and CHK-09 are recovered only by a few models, including GLM-5.1 and Claude Opus 4.
  • D.1 Per-Range Performance Analysis: NPM-Worm models reliably identify the worm, malicious package, dropper, webhook exposure, attacker IP, cross-runtime attack chain, and timeline, whereas ASP.NET-ViewState and RDP-Service-Abuse expose major host-investigation gaps.ASP.NET models miss Windows stealth mechanisms and leave MSSQL, xp_cmdshell, attacker-IP blocking, and scheduled-task issues unresolved; RDP models recover obvious post-compromise evidence but largely overlook initial access and privilege abuse.

D.2 Capability Performance

Table 17 reports model performance (%) for each capability item, with results evaluated on detection and planning dimensions. Scores are averaged across three LLM judges.

  • Capability-item performance: Table 17 reports model performance (%) separately for each capability item.The table evaluates each item across the defined detection and planning axes.
  • Detection: Each capability item is evaluated along the “Det” axis, measuring achieved detection scores across all checkpoints.The detection percentage is computed over all checkpoints mapped to that item.
  • Planning: Each capability item is evaluated along the “Plan” axis, measuring achieved planning scores across all checkpoints.The planning percentage is computed over all checkpoints mapped to that item.
  • Evaluation aggregation: Scores for each item are averaged across the three LLM judges.The judges assess the checkpoints mapped to each item.

D.3 Token Usage

Token usage and cost vary substantially across models, but taking more steps or spending more does not reliably improve incident-response performance. The highest-performing models are not the most expensive, while GPT Pro models cost the most yet score only around 50–60%.

  • D.3 Token Usage: Gemini 3.1 Pro averages 38.5 steps and DeepSeek V3.2 47.2, whereas GPT-5.4 averages 8.6 steps.The passage cautions that additional investigation may offer no clear payoff.
  • D.3 Token Usage: More steps do not guarantee better performance, indicating that additional investigation may offer no clear payoff.
  • D.3 Token Usage: Cost and performance show only a weak relationship, and the highest-performing models are not the most expensive.Higher cost and greater token consumption help to some extent, but spending more does not reliably yield stronger incident-response capability.
  • D.3 Token Usage: GPT Pro models have the highest costs, but their performance remains around 50–60%.Their higher price does not translate into stronger incident-response capability.
Loading 2607.26791v1…