PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems

Boxuan Zhang, Jianing Zhu, Zeru Shi, Dongfang Liu, Ruixiang Tang

arXiv:2605.08715v2cs.CLcs.AIcs.MA

TL;DR

AgentForesight addresses the gap between post-hoc failure attribution and detecting decisive errors during unfolding multi-agent trajectories. It introduces online auditing with a curated corpus and coarse-to-fine-trained auditor, outperforming proprietary judges by 19.9 Exact-F1 points and achieving 3× lower localization error.

  • Problem

    Existing failure analysis diagnoses responsible agents and steps only after trajectories finish, leaving online detection of decisive errors in unfolding prefixes underexplored.

  • Method

    AgentForesight combines AFTRAJ-2K, with curated safe trajectories and multi-judge decisive-error annotations, with coarse-to-fine reinforcement learning for prefix-restricted auditing.

  • Results

    19.9 Exact-F1 points above the strongest proprietary baseline and 3× lower step localization error are achieved across AFTRAJ-2K and Who&When.

  • Takeaways & Limitations

    Online auditing reframes agentic failure analysis as per-step continue-or-alarm oversight that can intervene before downstream propagation locks in failure.

  • Takeaways & Limitations

    The method assumes that safe trajectories contain no decisive errors, requiring verification because downstream recovery or permissive evaluation can mask locally degenerate steps.

Abstract

from arXiv · show

LLM-based multi-agent systems are increasingly deployed on long-horizon tasks, but a single decisive error is often accepted by downstream agents and cascades into trajectory-level failure. Existing work frames this as \emph{post-hoc failure attribution}, diagnosing the responsible agent and step after the trajectory has ended. However, this paradigm forfeits any opportunity to intervene while trajectory is still unfolding. In this work, we introduce AgentForesight, a framework that reframes this problem as online auditing: at each step of an unfolding trajectory, an auditor observes only the current prefix and must either continue the run or alarm at the earliest decisive error, without access to future steps. To this end, we curate AFTraj-2K, a corpus of agentic trajectories across Coding, Math, and Agentic domains, in which safe trajectories are retained under a strict curation pipeline and unsafe trajectories are annotated at the step of their decisive error via consensus among multiple LLM judges. Built on that, we develop AgentForesight-7B, a compact online auditor trained with a coarse-to-fine reinforcement learning recipe that first equips it with a risk-anticipation prior at the failure boundary on adjacent safe/unsafe prefix pairs, then sharpens this prior into precise step-level localization under a three-axis reward jointly targeting the what, where, and who of an audit verdict. Across AFTraj-2K and an external Who\&When benchmark, AgentForesight-7B outperforms leading proprietary models, including GPT-4.1 and DeepSeek-V4-Pro, achieving up to +19.9% performance gain and 3$\times$ lower step localization error, opening the loop from post-hoc failures detection to enabling deployment-time intervention. Project page: https://zbox1005.github.io/agent-foresight/

1 Introduction

AgentForesight reframes multi-agent failure analysis from post-hoc attribution to online auditing, requiring an auditor to judge each unfolding prefix without future information and intervene before failure propagates. It introduces a curated trajectory corpus and coarse-to-fine-trained AgentForesight-7B, which outperforms proprietary judges on online auditing benchmarks.

  • Motivation: Multi-agent systems extend LLM capabilities across long-horizon software development, scientific discovery, and open-ended web navigation, but these gains carry a structural cost.The introduction motivates failure analysis as a central reliability concern for increasingly capable agentic systems.
  • Problem formulation: Online auditing replaces post-hoc failure attribution with continue-or-alarm decisions on each trajectory prefix before downstream propagation locks in failure.The auditor observes only the current prefix and lacks access to future steps, tool responses, or the eventual outcome.
  • Dataset: AFTRAJ-2K spans Coding, Math, and Agentic domains, retaining safe trajectories through strict filtering and annotating decisive errors in failure trajectories via multi-judge verification.The corpus is designed for prefix-restricted online auditing and step-level decisive-error identification.
  • Method: AgentForesight-7B uses coarse-to-fine reinforcement learning to develop a risk-anticipation prior before precise step-level localization.The training recipe first focuses on the failure boundary, then optimizes structure, timing, and attribution.
  • Results: +19.9% higher Exact-F1 and 3× lower step localization error were achieved than the strongest proprietary baseline on AFTRAJ-2K and Who&When.AgentForesight-7B surpassed its Qwen2.5-7B-Instruct base model and proprietary judges including GPT-4.1 and DeepSeek-V4-Pro.

2 Problem Formulation

This section formalizes failure monitoring around decisive errors and contrasts retrospective attribution with online auditing, which makes prefix-only, per-step intervention possible during execution.

  • Shared Definitions: A decisive error is the earliest step whose correction can change a failed trajectory into a successful one under the system policy.The responsible agent is the role at that step; successful trajectories are labeled SAFE.
  • Post-hoc Failure Attribution: Post-hoc attribution observes a completed failed trajectory and emits one retrospective prediction of the decisive step and responsible agent.It uses full hindsight and offers no intervention window because prediction occurs after failure materializes.
  • Online Auditing: Online auditing evaluates each trajectory prefix and chooses CONTINUE or ALARM using only currently visible evidence.ALARM halts execution and reports the predicted decisive step and responsible agent within the observed prefix.
  • Online Auditing: Online auditing produces N verdicts per trajectory and can intervene before the next turn is committed, unlike post-hoc attribution.The auditor cannot access future turns or the terminal outcome, making direct application of post-hoc predictors to live prefixes ill-posed.

3 Methodology

AgentForesight operationalizes online auditing through the AFTRAJ-2K corpus, which provides verified prefix-level safe and decisive-error supervision, and a coarse-to-fine recipe that trains AgentForesight-7B for risk anticipation and step-level localization.

  • AFTRAJ-2K Construction: A three-stage filtering pipeline retains trajectories that are safe at every prefix by enforcing outcome equivalence, valid tool use, and sub-goal coherence.Safe means successful completion without any step whose correction would have changed the outcome.
  • AFTRAJ-2K Construction: Failure supervision combines controlled decisive-error injection with a propose-and-verify ensemble that localizes responsible steps and agents in naturally failed trajectories.Injected candidates are rejected when downstream recovery occurs or the targeted turn is unchanged; diagnostic candidates are re-checked against four binary criteria.
  • AFTRAJ-2K Construction: AFTRAJ-2K comprises ∼2.3K annotated safe and unsafe trajectories with SAFE labels on every safe prefix and (k∗, a∗) labels at each unsafe trajectory’s decisive step.The corpus pools verified-safe and verified-unsafe streams into D_AFTRAJ = D_safe ∪ D_unsafe.
  • Coarse-to-Fine Training: The two training stages produce AgentForesight-7B, a compact auditor that combines risk anticipation with precise step-level localization and issues per-step CONTINUE/ALARM verdicts.The recipe addresses the need to learn both the safe-versus-unsafe boundary and localization of the decisive step and responsible agent.
  • Coarse-to-Fine Training: Stage 1 trains on adjacent pre- and post-boundary prefixes whose optimal verdicts are respectively CONTINUE and ALARM, establishing a risk-anticipation prior at the decisive step.The paired prompts differ by exactly one turn at the decisive step, isolating the failure boundary as the learning signal.

4 Experiments

Experiments evaluate AgentForesight-7B under strict online auditing on AFTRAJ-2K and an external Who&When benchmark, using exact-step localization metrics. The model outperforms strong baselines, benefits from its two-stage training recipe, and transfers to unseen agentic trajectories.

  • Datasets: AFTRAJ-2K covers Math, Coding, and Agentic tasks from AutoGen, MetaGPT, and Smolagents, with a trajectory-grouped held-out split.The task corpora include MATH-500, HumanEval+, MBPP+, GAIA, and HotpotQA.
  • Metrics: Exact-F1 measures decisive-step precision and recall, while ASS averages absolute distance from the true decisive step on detected unsafe trajectories.The metrics capture both exact localization and informative near-misses when alarms miss the exact step.
  • AFTRAJ-2K Results: 66.44 Exact-F1 exceeds DeepSeek-V4-Pro by 19.88 points, while overall ASS decreases from 1.77 to 0.59 (3×) on AFTRAJ-2K.AgentForesight-7B improves both metrics in every domain, with Exact-F1 gains of 77.36 vs. 50.34 on Math and 78.87 vs. 49.32 on Coding.
  • External Generalization: 19.59 points on Step-Acc and 6.41 on Agent-Acc separate AgentForesight-7B from GPT-4.1 on Who&When, while ASS falls from 2.35 to 1.62.The benchmark uses trajectories from multi-agent frameworks disjoint from AFTRAJ-2K and unseen during training.
  • Ablations: 66.4 Exact-F1 results from combining Stage 1 and Stage 2, versus 21.1 for the base, 35.6 with Stage 1, and 50.4 with Stage 2.Stage 2 alone reaches 63.6 on Math and 72.7 on Coding, while the reported recovery to 48.70 supports predict-then-localize coupling.
  • Deployment Trade-off: The deployment analysis defines a deployable region as False Alarm Rate ≤20% and Step-Acc ≥50%.Figure 4 evaluates the trade-off between false alarms on safe trajectories and step accuracy on unsafe trajectories.

5 Conclusion

AgentForesight reframes agentic failure analysis as an online, per-step continue-or-alarm decision over unfolding trajectory prefixes. It introduces AFTRAJ-2K and AgentForesight-7B, combining curated decisive-error annotations with coarse-to-fine reinforcement learning for precise audit localization.

  • AgentForesight reframes failure analysis from post-hoc diagnosis of completed trajectories into per-step continue-or-alarm decisions on unfolding prefixes.
  • AFTRAJ-2K pairs strictly filtered safe runs with multi-judge verified decisive-error annotations across Coding, Math, and Agentic domains.
  • AgentForesight-7B is a compact online auditor trained with coarse-to-fine reinforcement learning.Training first establishes a risk-anticipation prior at the failure boundary using adjacent safe/unsafe prefix pairs, then sharpens it into precise step-level localization.
  • The training reward jointly targets the what, where, and who of an audit verdict.

Appendices … B Additional Experiment Setups

The appendices document reproducibility resources, datasets, and the online-auditing assumption, then formalize AFTraj-2K construction and AgentForesight-7B’s two-stage training pipeline. The method combines verified-safe curation with constructive and diagnostic failure streams, followed by boundary alignment and three-axis verdict sharpening.

  • Reproducibility Statement: AFTraj-2K spans Coding, Math, and Agentic domains, while Who&When provides an external transfer benchmark.Coding uses HumanEval+ and MBPP+, Math uses MATH-500, and Agentic uses GAIA and HotpotQA, collected with AutoGen, MetaGPT, and Smolagents using GPT-5.4-mini.
  • Reproducibility Statement: The experiments consistently use online auditing, querying the auditor at every visible trajectory prefix for a continue-or-alarm decision.The auditor must commit using only the visible window, without access to future steps.
  • Reproducibility Statement: The release includes the AFTraj-2K construction pipeline and the coarse-to-fine training recipe for AgentForesight-7B.Source code is provided in the submitted supplementary materials.
  • B Additional Experiment Setups: Both training stages run on 2×NVIDIA H200 GPUs with Python 3.10 and PyTorch 2.9.Learning rate, batch size, and group size are reported in Table 4 of Appendix B.3.
  • A Algorithmic Pipeline: Algorithm 1 collects multi-agent rollouts and filters successful trajectories through three predicates to form the verified-safe pool Dsafe.Dsafe supplies positive auditor supervision and scaffolds constructive decisive-error injection, while diagnostic processing recovers unknown decisive steps in naturally failed trajectories.
  • A Algorithmic Pipeline: The construction pipeline runs constructive and diagnostic failure streams in parallel after safe curation.Constructive failures receive by-construction labels through injected decisive errors; diagnostic failures use propose-and-verify processing to identify naturally occurring decisive steps.
  • A Algorithmic Pipeline: Stage 1 builds boundary pairs and preference data, then minimizes the dual-subset BPPO loss to produce πθ1.Boundary pairs come from Dunsafe, while preferences are obtained by classifying base-policy rollouts.
  • A Algorithmic Pipeline: Stage 2 sharpens πθ1 with the three-axis reward through GRPO, freezing πθ1 as πref for KL regularization.The reference policy keeps updates aligned with the failure-boundary prior learned in Stage 1 rather than the generic base policy πθ0.

B.1 Details of Datasets … B.3.1 Implementation details of AFTRAJ-2K Construction

The paper introduces AFTRAJ-2K, a 2,272-family trajectory corpus for online auditing, and evaluates auditors step by step using structured verdicts and localization metrics. It also specifies external evaluation and construction procedures spanning multi-agent templates, strict safe curation, and annotated failure generation.

  • B.1 Details of Datasets: AFTRAJ-2K contains 2,272 family-level multi-agent trajectories across Coding, Math, and Agentic domains, with safe trajectories filtered and unsafe trajectories collected through two complementary streams.Table 3 reports per-domain composition, while Figure 6 characterizes decisive-error-step distributions.
  • B.1 Details of Datasets: The dataset applies additional domain-specific filtering, reducing Coding safe trajectories from 411 to 361 and retaining 395 of 396 Math safe trajectories.Agentic safe trajectories total 402 after combining GAIA and HotpotQA with two additional task pools.
  • B.1 Details of Datasets: Who&When is an external benchmark of 127 multi-agent systems with annotated decisive agent-step pairs, spanning algorithm-generated and hand-crafted systems.Its trajectories and underlying tasks are disjoint from AFTRAJ-2K.
  • B.2 Details on Evaluation Metrics: During evaluation, the auditor walks each trajectory prefix incrementally and emits a categorical label, predicted decisive step, responsible agent, and earliest alarm point.The earliest alarm is denoted d(τ), with d(τ)=∞ when no alarm is raised.
  • B.2 Details on Evaluation Metrics: Exact-F1 is the harmonic mean of Step Recall and Step Precision, jointly penalizing missed errors and incorrectly localized alarms under exact-match localization.Step Recall evaluates exact localization among unsafe trajectories, while Step Precision conditions on alarm-triggered trajectories.
  • B.2 Details on Evaluation Metrics: ASS↓ averages the absolute predicted-to-ground-truth decisive-step distance over detected unsafe trajectories, providing graded localization information when exact-step matching fails.ASS is undefined for unsafe trajectories without an alarm because no predicted step exists.
  • B.3.1 Implementation details of AFTRAJ-2K Construction: AFTRAJ-2K uses AutoGen Swarm for Coding, a two-role CodeWriter-CodeTester system, and Smolagents for Agentic tasks, with Math covered by the stated multi-domain template setup.Coding uses HumanEval+ and MBPP+; Agentic uses GAIA and HotpotQA with delegated web search and Wikipedia retrieval.
  • B.3.1 Implementation details of AFTRAJ-2K Construction: The construction pipeline uses GPT-5.4-mini with greedy decoding and a 40-step task cap, admits successful rollouts only after three independent verification predicates, and creates failures through injection and diagnostic streams.Injection failures receive by-construction decisive labels, while diagnostic annotation uses proposer and verifier calls to recover decisive candidates.

B.3.2 Implementation details of training … D.1 Full Two-Stage Ablation Results

The paper specifies two-stage training and strict online baseline protocols, situates AgentForesight within related work on agent failures, monitoring, reinforcement learning, and LLM judges, and reports detailed ablations showing complementary gains from the two stages.

  • B.3.2 Implementation details of training: Stage 1 trains Qwen2.5-7B-Instruct with dual-subset BPPO on 8,192-token boundary-pair prompts using FSDP across 2×NVIDIA H200 GPUs.Training uses a frozen same-base reference policy, 8-bit AdamW, bfloat16 mixed precision, gradient checkpointing, cosine decay, and 50-step warmup.
  • B.3.2 Implementation details of training: Stage 2 initializes trainable and frozen policies from Stage 1, applies KL directly in the loss, and uses verl’s DAPO reward manager with a soft overlong-response buffer.The buffer smoothly penalizes responses approaching the 4,096-token budget, preventing reward saturation beyond the budget.
  • B.3.3 Implementation details of baselines: Ten LLM auditors are evaluated through strict step-by-step incremental walks, querying each prefix under the same system and incremental-view user prompts used by AgentForesight-7B.The evaluated models include five open-source and five proprietary LLMs.
  • B.3.3 Implementation details of baselines: Perplexity-7B and ToT-7B produce prefix-only per-step scores, while Reflexion-7B estimates correctness from binary self-reflection probabilities on candidate turns.Perplexity uses length-normalized log-likelihood, ToT uses SURE/LIKELY/IMPOSSIBLE scores mapped to 2/1/0, and Reflexion aggregates capitalization variants.
  • B.3.3 Implementation details of baselines: These incremental baselines use first-crossing alarms, with Reflexion’s threshold tuned on a clean 300-trajectory held-out pool by maximizing detection F1 over 19 score quantiles.The threshold is frozen for held-out testing, and Reflexion performs one bfloat16 forward pass per step on one H200 GPU.
  • B.3.3 Implementation details of baselines: AgentDebug-7B is the only baseline that consumes the completed trajectory in one shot, adapting its critical-step JSON schema to support SAFE outcomes for both safe and unsafe trajectories.This mirrors the original AgentDebug post-hoc protocol while enabling evaluation on both trajectory types.
  • C Detailed Related Work: Related work spans multi-agent coordination frameworks, failure characterization and post-hoc attribution, runtime monitoring, reinforcement learning for agentic models, and LLM-based judges and step-level critics.Examples include AutoGen, MetaGPT, Camel, MAST, monitoring scaffolds, Search-R1, AgentGym-RL, AgentFlow, GiGPO, MT-Bench, G-Eval, and process reward models.
  • D.1 Full Two-Stage Ablation Results: Stage 1 raises overall Exact-F1 from 21.05 to 35.63, while Stage 2 alone reaches 50.42 and particularly sharpens Math to 63.64 and Coding to 72.73.Table 5 provides the full per-domain Exact-F1 and ASS results underlying Figure 3 and separates each stage’s contribution.

D.2 Full Deployment Trade-Off Results · D.3 Computational and Cost Analysis

AgentForesight-7B is the only auditor in the deployable region, combining low false alarms with strong step accuracy. It also supports economical local deployment, with substantially lower latency than the strongest proprietary baseline.

  • D.2 Full Deployment Trade-Off Results: 2.37% FAR and 59.51% Step-Acc place AgentForesight-7B inside the deployable region defined by FAR ≤20% and Step-Acc ≥50%.FAR is computed on Dsafe, while Step Accuracy is computed on Dunsafe.
  • D.2 Full Deployment Trade-Off Results: 43.20% FAR and 53.99% Step-Acc leave DeepSeek-V4-Pro just outside the deployable region despite its strong step accuracy.DeepSeek-V4-Pro is identified as the strongest proprietary baseline.
  • D.2 Full Deployment Trade-Off Results: Table 6 evaluates every auditor on complementary AFTRAJ-2K halves, using FAR on Dsafe and Step Accuracy on Dunsafe.The reported values underpin the scatter plot in Figure 4.
  • D.3 Computational and Cost Analysis: Per-call deployment efficiency is assessed using wall-clock latency and API cost, with costs based on input and output token counts.The pricing basis is the official 2026-05 pricing of each provider.
  • D.3 Computational and Cost Analysis: Open-source auditors incur compute time but no per-call charge, whereas API-served baselines are evaluated on both latency and monetary cost.This distinction motivates cost analysis for auditors queried at every trajectory prefix.
  • D.3 Computational and Cost Analysis: 1.03 s/call is AgentForesight-7B’s local latency on a single H200.This deployment incurs compute time without a per-call API charge.
  • D.3 Computational and Cost Analysis: 4.73 s/call is AgentForesight-7B’s latency on the smaller RTX 4500 Ada.The result demonstrates local serving across two hardware configurations.
  • D.3 Computational and Cost Analysis: 25.77 s/call is DeepSeek-V4-Pro’s measured latency, making AgentForesight-7B faster than the strongest proprietary baseline.Latency is hard-measured from evaluation logs as total elapsed seconds divided by total audit calls.

D.4 Failure Mode Analysis · D.5 Additional Case Study · E Prompt Templates

AgentForesight-7B’s remaining errors are rare and mostly involve false alarms during self-correction or off-by-one localization, while a case study shows it uniquely identifies a late decisive error. The accompanying prompts formalize online auditing and diagnostic supervision through strict evidence-based verdict and verification procedures.

  • D.4 Failure Mode Analysis: False alarms affect only 4/169 safe runs (FAR = 2.37%), while 21/28 unsafe mis-localizations (75%) are off-by-one shifts.Both failure modes concentrate in < 10% of evaluation cases and do not reverse the +19.88 Exact-F1 lead or 3× tighter ASS.
  • D.4 Failure Mode Analysis: A transient VERIFIED_WRONG string triggers a false alarm even though the Verifier self-corrects by the next step and safely concludes with 11.The verdict incorrectly flags step 10 for the Verifier, based on the transient contradictory string.
  • D.4 Failure Mode Analysis: In an unsafe HotpotQA trajectory, the auditor flags the Manager’s first unsupported “Yes” at step 1 rather than the annotated decisive python-wrap step 2.The alarm still correctly classifies the trajectory as unsafe and identifies the responsible agent; the discrepancy is upstream-root versus downstream-commit localization.
  • D.5 Additional Case Study: AgentForesight-7B alone identifies the decisive Math error at Step 6, whereas Gemini-3-Flash and GPT-4.1 flag earlier still-recoverable steps 4 and 3.The case study concerns a MATH-500 trajectory with a late-committing decisive error and identifies MathSolver as responsible.
  • E Prompt Templates: The online-auditor system prompt requires chronological analysis, evidence-supported earliest-error detection, and SAFE when no decisive error appears in the visible prefix.The incremental-view prompt supplies only the trajectory through the current step and requires the first supported decisive-error index or SAFE.
  • E Prompt Templates: The auditor’s strict two-block output encodes answer, exact agent role, and a concise reason, with null agent and reason for SAFE verdicts.UNSAFE verdicts require a step integer, non-empty agent string, and non-empty reason grounded in the visible evidence.
  • E Prompt Templates: The diagnostic proposer generates up to three distinct causal steps, prioritizing substantive, decisive, earliest errors and exact step-agent attribution.Candidates also include failure type, propagation reason, suggested fix, and confidence on a 1–5 scale.
  • E Prompt Templates: The diagnostic verifier conservatively checks step existence, substantive error, decisive causality, and earliest-error status before accepting a diagnosis as a training label.Its output records four binary criteria, any earlier better step, confidence, and explanatory notes.

F Qualitative Examples from AFTRAJ-2K · G Discussions · G.1 External Auditing vs. Agent Self-Reflection

The paper illustrates AFTraj-2K with safe and unsafe trajectories spanning Math, Coding, and Agentic domains, then motivates external auditing over agent self-reflection. External auditing isolates verdict-making from execution and substantially improves auditing accuracy and localization under the reported comparison.

  • F Qualitative Examples from AFTRAJ-2K: F Qualitative Examples from AFTRAJ-2K: Three trajectories exemplify AFTraj-2K supervision sources: verified-safe Math, constructive-stream injected unsafe Coding, and diagnostic-stream natural-failure Agentic records.Frame colors encode safe/unsafe labels, while box titles identify source benchmarks.
  • F Qualitative Examples from AFTRAJ-2K: F Qualitative Examples from AFTRAJ-2K: The verified-safe Math trajectory reaches 3*sqrt(13), terminates with ANSWER_VERIFIED, and includes independent re-derivation by a second agent.The Verifier performs its own compute call before verify_answer rather than merely echoing the MathSolver.
  • F Qualitative Examples from AFTRAJ-2K: F Qualitative Examples from AFTRAJ-2K: The injected Coding trajectory flips ALL_TESTS_PASSED into TESTS_FAILED when CodeTester fabricates an off-by-one concern, defining a verdict_misread at Step 9.The decisive agent is CodeTester, and the contradictory final verdict appears at Step 10.
  • F Qualitative Examples from AFTRAJ-2K: F Qualitative Examples from AFTRAJ-2K: The naturally failed Agentic trajectory makes search_agent’s Step 4 decisive by confusing the requested geographic location with the hotel’s name.The Manager later propagates the incorrect answer verbatim.
  • G.1 External Auditing vs. Agent Self-Reflection: G.1 External Auditing vs. Agent Self-Reflection: External auditing is preferred because judging whether a prefix remains on track is separated from planning, retrieval, computation, and coordination.The design also avoids reusing the generator’s posterior and preserves the execution policy’s primary objective.
  • G.1 External Auditing vs. Agent Self-Reflection: G.1 External Auditing vs. Agent Self-Reflection: An external auditor leaves agent execution untouched while enabling retraining or hot-swapping, post-hoc threshold tuning, and calibrated continue-or-alarm deployment.These operational properties are presented as benefits unavailable to in-agent self-reflection.
  • G.1 External Auditing vs. Agent Self-Reflection: G.1 External Auditing vs. Agent Self-Reflection: 23.38 overall Exact-F1 with 3.17 ASS is achieved by Reflexion-7B, versus 66.44 Exact-F1 and 0.59 ASS for AgentForesight-7B on the same backbone.The comparison holds capacity fixed while varying the audit paradigm.

G.2 Limitations · G.3 Broader Impact

AgentForesight-7B introduces a per-prefix monitoring dependency and may generate false alarms that prematurely halt useful trajectories. The paper therefore recommends calibrated confidence, tiered intervention, and human oversight in safety-critical deployments.

  • G.2 Limitations: G.2 Limitations: Auditing every trajectory prefix creates a lightweight operational dependency compared with one-shot post-hoc attribution.The auditor must be queried throughout the unfolding trajectory.
  • G.2 Limitations: G.2 Limitations: The auditor is kept at 7B to remain practical for colocation with a host agent.The paper notes that per-step monitoring is also shared by concurrent runtime safeguards in agent systems.
  • G.3 Broader Impact: G.3 Broader Impact: Cascaded failures can spill into tool calls, file writes, and external API requests with potentially irreversible side effects.The paper links this risk to the rapid deployment of multi-agent LLM systems in real-world workflows.
  • G.3 Broader Impact: G.3 Broader Impact: Continue-or-alarm decisions are committed at every step of an unfolding trajectory.This decision protocol is presented in the context of limiting cascaded failures and their downstream effects.
  • G.3 Broader Impact: G.3 Broader Impact: False alarms can prematurely halt useful trajectories and adversely affect end users and operators.The paper identifies false alarms as a deployment risk rather than treating every alarm as harmless.
  • G.3 Broader Impact: G.3 Broader Impact: Deployers should expose calibrated alarm confidence instead of treating AgentForesight verdicts as hard kill switches.The paper recommends pairing the auditor with a tiered intervention policy or human-in-the-loop oversight in safety-critical settings.
Loading 2605.08715v2…