PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond End-to-End Success: Diagnosing Failures in Long-Horizon Security LLM Agents

Wei Shao, Chongzhou Fang, Zuxiong Tan, Zequan Liang, Setareh Rafatirad, Avesta Sasan, Houman Homayoun

arXiv:2608.20563v1cs.CRcs.AI

TL;DR

Long-horizon security-agent success is difficult to interpret when agents can fail before reaching the state needed to exercise the capability under evaluation. The paper uses checkpoints and controlled interventions to diagnose where failures occur, finding that the dominant bottleneck can shift across model generations.

  • Problem

    Final task success cannot distinguish lacking a capability from failing before that capability can be exercised in a dependent long-horizon workflow.

  • Method

    The paper instruments four long-horizon security task families with exposure checkpoints and uses controlled rescue–placebo interventions to test suspected upstream bottlenecks.

  • Results

    The dominant failure source shifts across model generations: many Gemini 2.5 Flash failures precede state observation, whereas Gemini 3.7 Flash shows an opposite intervention effect and downstream failures after observation.

  • Takeaways & Limitations

    Evaluations should report both whether agents reach capability-relevant states and where failures occur afterward, rather than relying only on aggregate task success.

  • Takeaways & Limitations

    Final success rates can have limited interpretability for secondary tasks when checkpoint evidence shows variant confounds, performance ceilings, or insufficient reach of the intended comparison branch.

Abstract

from arXiv · show

Long-horizon security LLM agents must carry information and decisions across many dependent interactions, where later actions often depend on services, state, or access discovered much earlier. This makes final task success difficult to interpret: an agent may fail before it ever reaches the point where the capability of interest can be exercised. We present a diagnostic methodology that instruments security tasks with checkpoints, separates failures before and after capability exposure, and uses controlled interventions to test suspected upstream bottlenecks. We evaluate the methodology across four task families involving delayed reuse of discovered information, reuse of observed state, recovery from failed strategies, and decision making after uncertain outcomes. On observed state reuse, checkpoint analysis shows that many Gemini 2.5 Flash failures occur before the model observes the state it is later expected to reuse. In a pre-specified 92-seed study, targeted protocol-disambiguation guidance increases state observation from 65.5\% under a matched non-guidance control message to 95.4\%. Repeating the same design with Gemini 3.7 Flash produces the opposite effect, while state observation no longer reliably predicts task completion. These results show that the dominant source of failure can shift across model generations, motivating evaluation that diagnoses where and why long-horizon security agents fail rather than relying only on aggregate task success.

I. INTRODUCTION

Long-horizon security agents can fail before reaching the state or information needed to exercise the capability being evaluated. The paper introduces checkpoint-based diagnosis and matched rescue–placebo interventions, finding that failure mechanisms can differ across model generations.

  • Intermediate progress does not establish that an agent exercised the capability a task was designed to test.An agent that never discovers the state it must later reuse cannot directly demonstrate state-reuse ability.
  • Checkpoint-based exposure analysis separates failures before a target capability becomes exercisable from failures occurring afterward.The methodology identifies the first capability-relevant checkpoint and reports behavior on both sides of it.
  • Many Gemini 2.5 Flash Controlled State Reuse failures occur before the model observes the state it is expected to reuse.After state observation, downstream completion is generally reliable, and targeted protocol-disambiguation guidance increases state observation relative to placebo.
  • Matched rescue–placebo interventions test whether targeted information affects a suspected upstream failure mechanism.Rescue supplies targeted information, whereas placebo provides a structurally matched message without task-relevant guidance.
  • The same discovery intervention reverses direction for Gemini 3.7 Flash, showing that the identified failure mechanism does not transfer across model generations.The study repeats the intervention under the same pre-specified design to test cross-generation transfer.

II. BACKGROUND AND RELATED WORK

Prior security-agent benchmarks and process-level evaluations measure milestones, stages, partial progress, or trajectory behavior beyond final success. This paper narrows the diagnostic question to whether the agent reaches the state where the intended capability can be exercised, then tests upstream explanations experimentally.

  • Security-agent systems increasingly perform multi-step offensive-security workflows, but performance depends on tools, memory, planning, and interaction design as well as the language model.The paper studies failure diagnosis rather than proposing a new offensive-security agent.
  • Existing benchmarks expose intermediate progress through subtasks, milestones, workflow stages, capability ladders, and partial-completion measures.These approaches reveal more trajectory information than a binary final success rate.
  • The paper focuses on whether an agent reaches the point where the task’s intended capability can actually be exercised.This distinguishes failure to reach exposure from failure after the capability becomes relevant.
  • The methodology records capability-relevant checkpoints alongside final task success to distinguish exposure from post-exposure behavior.The deterministic, seeded environment fixes the prompt, task specification, and checkpoint definition during execution.
  • Matched interventions experimentally test candidate upstream failure mechanisms rather than relying only on trajectory inspection.Matched runs vary targeted information, a structurally matched placebo, or no message while holding core execution settings fixed.

A. Diagnostic Task Design

The paper instantiates its diagnostic methodology in four task families covering delayed reuse, strategy recovery, and adaptation after ambiguous outcomes. Controlled State Reuse is the primary task because its variants isolate the number of intermediate actions between observation and reuse.

  • The task environments use deterministic seeded state and multi-service interfaces, with fixed prompts, tool protocols, and checkpoint specifications.The setup can expose HTTP services, files, APIs, and SSH interactions.
  • Four diagnostic tasks study delayed secret reuse, delayed state reuse, strategy recovery, and adaptation after ambiguous outcomes.Together they apply one diagnostic approach to several forms of long-horizon failure.
  • Controlled State Reuse varies only the number of mandatory intermediate actions between observing exact state and later reusing it.Its Short, Medium, and Long variants require 2, 8, and 20 mandatory actions, respectively.
  • Delayed Secret Reuse is treated as secondary evidence because its Long variant adds a discovery requirement beyond retention.Therefore, differences between its Short and Long variants are not interpreted as a clean measure of retention.
  • Strategy Recovery and Ambiguous-Outcome Adaptation extend diagnosis to fallback behavior and responses to uncertain outcomes.Their conditions vary whether failures are controlled, explicit, ambiguous, transient, or persistent.

B. Checkpoint-Conditioned Exposure and Capability Analysis

Checkpoint-conditioned analysis identifies the earliest exposure point for each capability, reports both overall and post-exposure outcomes, and uses matched interventions to test isolated upstream obstacles. The design treats trajectory patterns as hypotheses and preserves fixed comparisons across model generations.

  • Exposure is reaching the gating checkpoint where the agent first has the information or state needed to exercise the target capability.If many runs fail before exposure but most exposed runs succeed, failures are concentrated upstream of that capability.
  • In Controlled State Reuse, C3 records exact observation of the generated state required for later retention and reuse.The analysis reports overall success, C3 reach, and success among runs reaching C3.
  • Exposure conditioning is diagnostic rather than corrective: pre-exposure failures remain in end-to-end results and tasks are not modified to force exposure.Overall and exposure-conditioned outcomes are reported together.
  • Matched intervention triples compare baseline, rescue, and placebo runs on the same deterministic seed, model, task instance, scaffold, timing, and limits.Rescue versus placebo is the primary comparison because it isolates targeted information from the general effect of adding a message.
  • For the CSR discovery study, the primary intervention outcome is C3 reach, while end-to-end success is secondary because it depends on later workflow stages.Same-seed binary outcomes are compared with a two-sided exact McNemar test.
  • Trajectory patterns remain hypotheses because an intervention may support, fail to affect, or reverse the suspected mechanism.All three outcomes are interpreted under the same frozen study design.

D. Experimental Governance and Reproducibility

The experiments use frozen, pre-specified governance to preserve matched comparisons, distinguish infrastructure failures from task failures, and separate confirmatory analyses across model generations.

  • Frozen experimental state: Approved manifests, source revisions, execution order, and intervention content are frozen and cryptographically bound before formal execution.Validation rejects modified bound content, while the runner verifies approval artifacts before execution.
  • Run integrity: Append-only lifecycle records preserve interrupted runs and allow resumption without overwriting outcomes or changing predetermined execution order.Runs receive records for preparation, execution, completion or failure, and evaluation.
  • Failure accounting: Infrastructure failures are recorded separately from valid task failures and are neither silently rerun nor replaced.This distinction allows invalid measurements to be excluded from scientific task-failure analyses.
  • Confirmatory comparisons: Matched rescue–placebo studies use paired seeds, with rescue versus placebo as the primary comparison and baseline as descriptive context.The confirmatory Gemini 2.5 Flash study uses a two-sided exact McNemar test with α = 0.05 for C3 reach.
  • Confirmatory comparisons: The Gemini 3.7 Flash replication preserves the original design while keeping cross-generation comparisons secondary and descriptive.Its analysis plan is frozen before outcomes are examined.
  • Validity rules: The Gemini 2.5 pre-specification required clarification for truncated infrastructure-invalid executions, whereas Gemini 3.7 specified this rule before execution.Affected Gemini 2.5 analyses exclude such runs and report sensitivity bounds for missing placebo observations.

IV. EXPERIMENTAL SETUP

The evaluation uses deterministic, multi-service Docker tasks organized into four diagnostic families, with controlled state reuse isolating the number of intermediate actions.

  • Environments: Four long-horizon task families run in deterministic Docker environments with multiple services and seed-generated task values.Matched runs therefore share the same underlying task instance.
  • Delayed Secret Reuse: Delayed Secret Reuse requires discovering a secret early, reusing it at a protected API endpoint, and completing an SSH-gated retrieval.Its Long variant changes both reuse delay and route discovery, so it is treated as secondary evidence.
  • Controlled State Reuse: Controlled State Reuse requires discovering generated state, learning its downstream interface, completing intermediate actions, and reusing the state later.CSR-Short, CSR-Medium, and CSR-Long require 2, 8, and 20 intermediate actions, respectively; C3 marks state observation.
  • Recovery and adaptation: Strategy Recovery tests fallback after a preferred strategy becomes unavailable, while Ambiguous-Outcome Adaptation tests responses to uncertain outcomes.SR keeps the fallback path fixed; AOA varies whether ambiguity is transient, persistent, or explicit failure.
  • Checkpoint structure: CSR, SR, and AOA share C0–C10 checkpoint numbering, whereas DSR retains task-specific checkpoints and failure labels.The common scheme does not imply identical checkpoint semantics across tasks.

B. Models and Agent Execution

The study compares Gemini models under a shared agent scaffold and staged evaluation pipeline, then tests a finalized CSR discovery intervention in independent confirmatory cohorts.

  • Models: Gemini 2.5 Flash and Gemini 2.5 Pro use the same scaffold, tool protocol, task prompts, and execution limits in matched comparisons.CSR and AOA form the main cross-model cohorts; DSR is evaluated only with Gemini 2.5 Flash.
  • Study stages: CSR analysis progresses from descriptive characterization to exploratory intervention testing, independent Gemini 2.5 confirmation, and Gemini 3.7 replication.The intervention targets protocol discovery identified as a candidate upstream obstacle for Gemini 2.5 Flash.
  • Execution: Controlled CSR runs are bounded by 90 commands, 120 model turns, and termination after 12 consecutive protocol errors.These limits remain identical across baseline, rescue, placebo, and Gemini 3.7 replication runs.
  • Cohorts: The initial CSR characterization uses 15 runs per Gemini 2.5 model, while the exploratory intervention uses five matched seeds and 15 Gemini 2.5 Flash runs.The exploratory study contains one baseline, rescue, and placebo run per seed.
  • Cohorts: The confirmatory Gemini 2.5 Flash study uses 92 new matched seeds and 276 planned runs across baseline, rescue, and placebo.Rescue and placebo differ only in message content and are delivered after the second completed command.
  • Intervention: Rescue provides only HTTP-versus-SSH protocol mapping, revealing none of the artifact, state, route, credentials, or final result.The Gemini 3.7 replication preserves these messages, seeds, checkpoints, scaffold, and execution limits.

V. EVALUATION

Checkpoint analysis shows that Gemini 2.5 Flash often fails before exposure to the capability-relevant state, while targeted protocol guidance addresses this bottleneck in a controlled example and post-exposure completion is usually reliable.

  • Exposure-conditioned outcomes: Figure 3 decomposes outcomes into success, failure before exposure, and failure after exposure; Gemini 2.5 Flash loses much performance before reaching the relevant checkpoint.This decomposition distinguishes missed exposure from failure after capability exposure.
  • Controlled State Reuse: 5/15 Gemini 2.5 Flash CSR runs succeed, but only 6/15 reach C3 and 5/6 of those complete; nine failed runs never acquire the required state.Gemini 2.5 Pro reaches C3 and succeeds in 15/15 runs.
  • Ambiguous-Outcome Adaptation: 8/10 Gemini 2.5 Flash AOA runs reach C5 and all subsequently succeed, indicating that its lower success rate originates before the capability checkpoint.Gemini 2.5 Pro reaches C5 and succeeds in 10/10 runs.
  • Controlled State Reuse: Flash C3 reach is 1/5 for Short, 1/5 for Medium, and 4/5 for Long, but these small-sample differences are descriptive.Because C3 precedes the intermediate sequence, the pattern does not show that longer sequences improve discovery.
  • Failure diagnosis: Trajectory inspection suggests protocol discovery as an upstream obstacle, but repeated unauthenticated SSH attempts are treated as a hypothesis rather than a confirmed explanation.The relevant alternative path requires HTTP reconnaissance to discover the state.
  • Intervention evidence: In a matched CSR-Medium pair, protocol-disambiguation guidance lets rescue reach the required state and complete the task, while placebo fails before state observation.The single pair motivates, but does not establish, general causality; the confirmatory study tests generalization.

C. Causal Diagnosis of the CSR Discovery Bottleneck

Controlled intervention results identify protocol discovery as an upstream bottleneck for Gemini 2.5 Flash on CSR-Medium. Once C3 is reached, downstream completion is generally reliable, so end-to-end failures largely reflect failure to reach the relevant state.

  • The initial state-restoration intervention was uninformative because only 2 of 5 pairs reached the intervention point in both conditions, with no discordant outcomes.The study therefore moved the intervention earlier to target protocol-selection behavior.
  • 95.4% of rescue runs reached C3, versus 65.5% under placebo, a 29.9-percentage-point increase.The exact two-sided McNemar test gives p = 2.56×10^-6; 29 of 32 discordant pairs favor rescue.
  • Final success was 94.3% under rescue and 64.4% under placebo, but it remained secondary because the intervention directly targeted discovery.Task completion also depends on later workflow stages.
  • 97.6% of infrastructure-valid runs that reached both the intervention point and C3 subsequently succeeded.This supports an exposure bottleneck rather than a dominant downstream completion failure for Gemini 2.5 Flash.
  • The controlled evidence supports protocol discovery as a causal contributor to C3 exposure for Gemini 2.5 Flash on CSR-Medium.The conclusion is unchanged under either extreme assumption for the five missing placebo-side C3 outcomes.

D. Cross-Generation Replication and Intervention Reversal

Repeating the frozen CSR intervention with Gemini 3.7 Flash reverses the Gemini 2.5 Flash effect and changes the diagnostic interpretation of C3 exposure. For Gemini 3.7 Flash, substantial failures remain after C3 is reached, so exposure no longer explains most end-to-end failures.

  • 79.1% of Gemini 3.7 Flash rescue runs reached C3, versus 98.9% under placebo, a -19.8-percentage-point difference.Among discordant pairs, 19 favor placebo and one favors rescue; the exact McNemar test gives p = 4.01 × 10^-5.
  • The reversal remains significant under either treatment of the single excluded placebo observation.The exact McNemar p-value is 2.21 × 10^-4 when treated as failure and 4.01 × 10^-5 when treated as success.
  • Only 36.1% of Gemini 3.7 Flash C3-reaching runs ultimately succeeded, compared with 97.6% for Gemini 2.5 Flash.Gemini 3.7 Flash often observes the required state but fails later in the workflow.
  • Gemini 3.7 Flash final success was 30.8% under rescue and 44.0% under placebo, without statistical significance.Final task success remained a secondary outcome, with exact McNemar p = 0.0807.
  • Across the common eligible seeds, the rescue-favoring fraction was 0.906 for Gemini 2.5 Flash and 0.053 for Gemini 3.7 Flash.The cross-generation comparison was pre-specified as descriptive, with no post-hoc significance test.
  • The replication changes the diagnosis: protocol discovery is a major exposure bottleneck for Gemini 2.5 Flash but not for Gemini 3.7 Flash.For the newer model, state observation is common while substantial failure remains afterward.

VI. DISCUSSION

The discussion argues that end-to-end success and exposure-conditioned analysis answer different questions in long-horizon security tasks. Checkpoints localize failures, while controlled interventions test suspected mechanisms, but checkpoint evidence alone does not establish cause.

  • End-to-end success cannot distinguish lacking a capability from failing before that capability can be exercised.In CSR, this changes the interpretation of Gemini 2.5 Flash’s apparent state-reuse difficulty.
  • Checkpoint evidence reveals why secondary task results can be overinterpreted: DSR is confounded, SR reaches a performance ceiling, and AOA misses its intended branch.These limitations are not visible from final success rates alone.
  • End-to-end success measures overall reliability, whereas checkpoint analysis shows where failures occur relative to the capability under study.The paper treats both measures as complementary and recommends considering both for capability claims.
  • Checkpoint analysis localizes failure but does not by itself identify the underlying cause.Missed checkpoints can have multiple plausible explanations in long trajectories, so suspected mechanisms require controlled testing.
  • The Gemini 3.7 replication shows that a failure mechanism identified for one model should not be assumed to transfer to another.The same intervention reduced C3 reach for Gemini 3.7 Flash, and reaching C3 no longer reliably predicted downstream completion.
  • Benchmark variants should change only the mechanism they intend to study and verify that agents reach the states required for the comparison.The paper also frames interventions as diagnostic probes rather than general solutions.

E. Reproducibility and Analysis Discipline

The study preserves fixed experimental designs and records unexpected outcomes to distinguish model behavior from changes in the experiment. Its controlled diagnostic methodology also exposes model- and setting-specific boundaries on interpreting long-horizon failures.

  • Reproducibility: Frozen prompts, interventions, execution order, and analysis decisions make unexpected findings more attributable to model behavior than post hoc experimental changes.The Gemini 3.7 replication retained the unchanged intervention despite producing the opposite effect from Gemini 2.5.
  • Scope: Conclusions are limited to controlled deterministic Docker environments and the evaluated model–task combinations, not general autonomous penetration testing.The benchmark generates security-relevant state internally, and the Gemini 3.7 results do not identify the specific cause of later downstream failures.
  • Analysis discipline: Checkpoint analysis across four task families shows that end-to-end success can conceal whether failures arise before exposure or afterward.The methodology identifies where the target capability first becomes exercisable and examines subsequent behavior.
  • Cross-generation analysis: The dominant failure source can change across model generations even when the task and experimental design remain fixed.Gemini 2.5 Flash often failed before observing the reusable state, whereas Gemini 3.7 Flash showed the opposite intervention effect and unreliable downstream completion after state observation.
  • Causal testing: Controlled interventions test whether suspected upstream obstacles contribute to failures instead of treating trajectory patterns as definitive explanations.The approach distinguishes discovery and exposure failures from failures of the downstream capability itself.
  • Ethical boundary: The work has dual-use potential, so it focuses on evaluation, diagnosis, and benchmark design while reporting unexpected results.The paper explicitly includes the Gemini 3.7 intervention reversal rather than modifying experiments to obtain a desired outcome.
Loading 2608.20563v1…