PaperlayerView exact revision on arXiv

Source-linked AI summary

PrefixGuard: From LLM-Agent Traces to Online Failure-Warning Monitors

Xinmiao Huang, Jinwei Hu, Rajarshi Roy, Changshun Wu, Yi Dong, Xiaowei Huang

arXiv:2605.06455v1cs.AI

TL;DR

Long-horizon LLM agents need lightweight online warnings over heterogeneous traces because final outcome checks may arrive too late, while hand-authored schemas and deployment-time LLM judging are limited. PrefixGuard induces offline typed-step adapters and trains supervised prefix monitors, improving over raw-text controls across four benchmarks while exposing when ranking supports actionable alerts.

  • Problem

    Long-horizon LLM agents require online failure warnings over heterogeneous traces, but hand-authored event schemas are brittle and deployment-time LLM judging is costly.

  • Method

    PrefixGuard uses offline StepView induction to create deterministic typed-step adapters, then trains failure-aligned event abstractions and prefix-risk monitors.

  • Results

    +0.03–+0.22 AUPRC over matched Raw-text control GRU across four benchmarks, while LLM judges remain weak under the matched prefix-warning protocol.

  • Takeaways & Limitations

    AUPRC ranking alone does not establish deployment utility, so observability and first-alert diagnostics identify whether warnings are early and low-false-alarm enough for intervention.

  • Takeaways & Limitations

    PrefixGuard requires visible prefix evidence, low-false-alarm separability, and a deployment action after an alert; its fixed horizons and FAR caps are evaluation controls.

Abstract

from arXiv · show

Large language model (LLM) agents now execute long, tool-using tasks where final outcome checks can arrive too late for intervention. Online warning requires lightweight prefix monitors over heterogeneous traces, but hand-authored event schemas are brittle and deployment-time LLM judging is costly. We introduce PrefixGuard, a trace-to-monitor framework with an offline StepView induction step followed by supervised monitor training. StepView induces deterministic typed-step adapters from raw trace samples, and the monitor learns an event abstraction and prefix-risk scorer from terminal outcomes. Across WebArena, $τ^2$-Bench, SkillsBench, and TerminalBench, the strongest PrefixGuard monitors reach 0.900/0.710/0.533/0.557 AUPRC. Using the strongest backend within each representation, they improve over raw-text controls by an average of +0.137 AUPRC. LLM judges remain substantially weaker under the same prefix-warning protocol. We also derive an observability ceiling on score-based area under the precision-recall curve (AUPRC) that separates monitor error from failures lacking evidence in the observed prefix. For finite-state audit, post-hoc deterministic finite automaton (DFA) extraction remains compact on WebArena and $τ^2$-Bench (29 and 20 states) but expands to 151 and 187 states on SkillsBench and TerminalBench. Finally, first-alert diagnostics show that strong ranking does not imply deployment utility: WebArena ranks well yet fails to support low-false-alarm alerts, whereas $τ^2$-Bench and TerminalBench retain more actionable early alerts. Together, these results position PrefixGuard as a practical monitor-synthesis recipe with explicit diagnostics for when prefix warnings translate into actionable interventions.

1. Introduction

PrefixGuard synthesizes online prefix-warning monitors from heterogeneous LLM-agent traces using offline deterministic StepView adapters and learned event abstractions. Its evaluation treats warning quality, representation, finite-state auditability, and low-false-alarm deployment utility as distinct diagnostic questions.

  • Problem and framework: PrefixGuard targets irreversible errors in long-horizon agent tasks by providing online warning signals before final verification.The framework addresses raw-trace monitor synthesis without hand-authored event alphabets or deployment-time LLM inference.
  • Evaluation questions: The evaluation diagnoses prefix-warning signal, trace representation, finite-state compression, and whether ranked risk scores support low-FAR alarms.It explicitly separates ranking from deployment utility rather than treating the benchmarks as a single leaderboard.
  • Problem and framework: StepView performs one-time offline LLM-assisted induction of deterministic typed-step adapters, enabling a common warning objective across heterogeneous browser, dialogue, coding, and CLI traces.The pipeline exposes post-action evidence in fixed fields before learned event abstraction and online prefix-risk scoring.
  • Evaluation questions: Across WebArena, τ^2-Bench, SkillsBench, and TerminalBench, zero-shot LLM judges are weak under the matched prefix-warning protocol.Outcome-labeled prefixes nevertheless retain learnable signal, motivating monitor synthesis rather than repeated LLM judging.
  • Deployment diagnostics: PrefixGuard pairs an AUPRC ceiling with observability and first-alert diagnostics to distinguish ranking scale, visible prefix evidence, and low-FAR alert utility.These diagnostics address deployment implications beyond aggregate ranking.

2. Related Work

Prior agent evaluation typically verifies completed trajectories or uses retrospective LLM judges, whereas PrefixGuard learns from outcome-labeled prefixes to produce online, domain-calibrated risk scores without deployment-time LLM inference. Its positioning also connects to classical runtime verification and specification-based monitoring.

  • LLM-agent evaluation and judges: Agent benchmarks usually assign success after trajectories end through task verifiers, while LLM-as-judge methods provide retrospective semantic assessment.These approaches evaluate completed behavior rather than producing stepwise online warnings.
  • LLM-agent evaluation and judges: PrefixGuard learns domain-calibrated temporal statistics from outcome-labeled prefixes and produces online risk scores at each step without deployment-time LLM inference.This design targets intervention before final outcome checks arrive.
  • Runtime verification and specification mining: PrefixGuard is positioned alongside classical runtime verification and specification-based monitoring.The related-work framing links the framework to established monitoring traditions.

3. Problem Formulation

PrefixGuard formulates online warning as risk scoring over structured trajectory prefixes, labeling imminent failure prefixes within an inclusive horizon as positives and evaluating ranking with AUPRC. It also frames observability as a representation-level limit: trace-only monitors cannot recover failures lacking distinguishable evidence in the observed prefix, so the resulting ceiling is an evaluation diagnostic rather than a latent-parameter estimate.

  • 3. Problem Formulation: Each execution step is a structured record containing the agent role, invoked tool, arguments, and environment response.A trajectory is an ordered sequence of such steps, while a raw trace is the original benchmark log before StepView conversion.
  • 3. Problem Formulation: A prefix is a positive warning target exactly when it belongs to a failed trajectory with at most H remaining steps.The inclusive horizon yields up to H+1 positive positions per failed trajectory, including the terminal prefix; other prefixes are negative.
  • 3. Problem Formulation: Per-trajectory 1/T normalization gives trajectories of different lengths equal weight.This normalization is part of the prefix-level learning formulation.
  • 3. Problem Formulation: At test time, the monitor alerts at step t when s_t > γ, with γ calibrated on a validation set.Performance is evaluated by AUPRC across prefixes, measuring risk-score ranking against imminent failure targets.
  • 3.1. A Diagnostic Observability Ceiling: An observable failed prefix contains distinguishable evidence already present in the current trace representation, whereas a hidden failed prefix does not.Examples of observable evidence include repeated tool errors, invalid retries, abnormal state, or clear drift from the task goal.
  • 3.1. A Diagnostic Observability Ceiling: Even with unlimited training data, a trace-only scorer cannot rank the hidden positive component above negatives from the observed trace alone.The positive-prefix representation is modeled as a mixture of observable positives and negatives, with hidden failures indistinguishable from negatives.
  • 3.1. A Diagnostic Observability Ceiling: The observability ceiling is tight, equals r when π=0 and 1 when π=1, and increases strictly with π.Here r is the positive-prefix rate and π is the observable-positive mixture weight.
  • 3.1. A Diagnostic Observability Ceiling: Forward π grids calibrate the AUPRC scale at each benchmark prevalence, but grid crossings do not estimate the true latent π.The ceiling is used only as an evaluation diagnostic.

4. Method

PrefixGuard turns heterogeneous agent traces into online risk monitors through a fixed offline StepView adapter, TF-IDF encoding, learned event abstraction, and differentiable sequence backends. It also supports post-hoc DFA extraction from hard learned symbols for compact symbolic auditing.

  • Trace normalization: PrefixGuard uses an offline LLM-assisted StepView adapter to map heterogeneous raw steps into fixed-schema canonical records before monitor training.The fixed adapter processes validation, test, and deployment traces without deployment-time LLM inference or step-level annotation.
  • Trace representation: Each canonical StepView record is serialized in fixed field-tagged order and encoded with a frozen TF-IDF vectorizer retaining d=4096 unigram and bigram features.The vectorizer is fit only on training-step strings and then frozen for validation, test, and deployment.
  • Event abstraction: A two-layer GELU projection with Gumbel-softmax maps each step embedding to a K-symbol event alphabet, trained jointly with the monitor against the warning objective.The soft symbol assignment remains differentiable, allowing gradients to flow into the abstraction layer while preventing symbol collapse through entropy balancing.
  • Prefix monitoring: The prefix-warning monitor emits a scalar risk score at every prefix and is instantiated as PrefixGuard-GRU, PrefixGuard-Transformer, or PrefixGuard-FSM.The GRU is the default online backend, the Transformer uses causal global attention at higher per-step compute, and the soft-FSM maintains a distribution over abstract states.
  • DFA extraction: Hard symbols z_t = arg max_k α_t,k can be used after training to fit an RPNI-style DFA whose states receive calibrated risk scores from held-out trajectories.This provides an exact symbolic representation for finite-state auditing distinct from the differentiable soft-FSM deployment backend.

5. Experiments

PrefixGuard evaluates lightweight prefix monitors across four agent benchmarks, showing that StepView-based learned monitors extract warning signal more effectively than raw-text controls and zero-shot LLM judges. The experiments also show that ranking quality, finite-state auditability, prevalence, and low-false-alarm intervention utility can diverge.

  • RQ1: Across WebArena, τ 2-Bench, SkillsBench, and TerminalBench, the strongest PrefixGuard monitors reach 0.900/0.710/0.533/0.557 AUPRC, respectively.The evaluation uses fixed train, calibration, validation, and test splits with H=3 prefix labels.
  • RQ1: The best zero-shot LLM judge reaches 0.450 AUPRC on WebArena, stays below 0.40 on τ 2-Bench, and falls near 0.10 on SkillsBench and TerminalBench.LLM judges and learned monitors are compared under the same H=3 warning labels and held-out splits.
  • RQ2: PrefixGuard outperforms raw-text controls by +0.029/+0.113/+0.218/+0.187 AUPRC on WebArena, τ 2-Bench, SkillsBench, and TerminalBench, averaging +0.137.The comparison holds the monitor model, split, horizon, and metric fixed while changing only the input view.
  • RQ3: Neural monitors provide the strongest ranking, while soft-FSM and exact DFA compression trade away ranking signal for increasingly constrained, auditable monitor forms.The comparison uses direct neural sequence monitors, a differentiable soft-FSM, and post-hoc DFAs extracted from learned hard symbols.
  • RQ4: WebArena’s positive-prefix rate is r = 0.363, versus r ≈0.07–0.09 elsewhere, so cross-benchmark AUPRC gaps require prevalence and visible-evidence calibration.The ceiling and MPE diagnostics separate prevalence scale effects from failures lacking evidence in the observed prefix.
  • RQ4: At a 10% calibration FAR cap, WebArena remains terminal-window triage despite 0.900 AUPRC, whereas τ 2-Bench is clearest for intervention and TerminalBench retains useful lead time.SkillsBench catches failures with high precision but late timing, demonstrating that low-FAR actionability differs from ranking.

6. Limitations and Conclusion … B.8. FSM Head and DFA Extraction

PrefixGuard combines offline, deterministic trace canonicalization with lightweight supervised prefix monitors and post-hoc finite-state audits. Its deployment value remains conditional on observable prefix evidence, low-false-alarm separability, and an actionable response after alerts.

  • 6. Limitations and Conclusion: PrefixGuard’s warnings require visible prefix evidence, low-FAR separability, and a deployment action after an alert; fixed horizons and FAR caps are evaluation controls.MPE coordinates are probe- and protocol-specific diagnostics rather than certified population π estimates, and DFA extraction is audit-friendly only in compact regimes.
  • A. Extended Related Work: Unlike completed-trajectory benchmark judges, PrefixGuard adapts prefix prediction to LLM-agent execution by learning representations and risk states from typed, partially benchmark-specific traces.Its observability ceiling relates to precision-recall limits under class skew and failures lacking evidence in the observed prefix.
  • B.4. StepView Canonicalization and B.5. StepView Adapter-Induction Design and Prompt: Offline StepView induction maps sampled raw traces into deterministic typed-step adapters, after which train/test conversion uses fixed code without LLM inference.The canonical fields are metadata, observation, action, tool, args, result, and status.
  • B.4. StepView Canonicalization: 100% coverage for tool_name and status, with 0% fallback across all 1.8M steps, while result fill reaches 87% on WebArena.Missing structured returns from navigation actions are treated as expected behavior rather than parsing failures.
  • B.6. TF-IDF Encoding: A single frozen TF-IDF vectorizer encodes canonical step strings under a shared d = 4096 feature budget across monitor heads and extracted-DFA audits.The same representation cap is used for GRU, Transformer, FSM, and extracted-DFA runs.
  • B.7. Trainable Monitor Hyperparameters: The event symbolizer is a two-layer MLP with hidden dimension 128, producing K soft symbols for downstream GRU or soft-FSM risk scoring.The direct GRU uses a single-layer GRU with hidden size Qmax and a linear sigmoid scoring head; training uses AdamW with learning rate 10^-3 and weight decay 10^-4.
  • B.8. FSM Head and DFA Extraction: DFA audits assess structure, calibrated state risk, coverage, filtering, and abstention, but their audit scores are single-artifact results rather than extracted-twin seed aggregates or human interpretability studies.Abstention is the fraction of prefixes rejected by the minimum-count DFA-state filter.
  • B.8. FSM Head and DFA Extraction: K and Qmax are protocol-level capacity controls matched across the soft-FSM, with default K=16 and TerminalBench using K=32, Qmax=32 for longer command-line trajectories.The default alphabet is the smallest final-protocol budget preserving stable validation behavior while keeping automata inspectable.

B.9. Automated Cross-Benchmark DFA Posthoc Audit

An automated posthoc DFA audit evaluates finite-state monitor artifacts using calibrated, coverage, warning, and routing-concentration diagnostics. WebArena is the most compact and risk-separating benchmark, while routed-DFA diagnostics address cases where single extraction weakens without claiming locked-test performance or human interpretability.

  • Automated audit: The audit automatically reports calibrated DFA metrics, trusted-state coverage, warning-state counts, and concentration of routed prefixes in the five most frequent states.It uses existing locked-test DFA artifacts across WebArena, τ 2-Bench, SkillsBench, and TerminalBench.
  • Cross-benchmark findings: WebArena has the most compact and risk-separating automaton among the four benchmarks, while τ 2-Bench is compact but not uniformly easy to inspect.The audit separates DFA inspection evidence from human interpretability evidence.
  • Limitations: These diagnostics assess finite-state auditability limits rather than locked-test performance, and do not measure human agreement, actionability, or annotation reliability.Table 10 defines ΔG as gain over the matched global DFA and ΔP as gain over the route-only prior.
  • Routed-DFA diagnostics: Validation-only routed-DFA diagnostics target SkillsBench and TerminalBench, where a single extracted DFA is largest, while keeping the trained monitor and hard-symbol protocol fixed.Only post-hoc DFA extraction changes to deployment-visible routes, with comparison against route-only calibration baselines.

C. Dataset Details … D. Extended Ablation Results

The paper fixes benchmark splits and an inclusive H=3 prefix-label protocol, evaluates monitors with locked-test ranking and calibration procedures, and supplements ranking with first-alert diagnostics. Extended ablations isolate recoverable signal, StepView evidence, label-geometry confounds, and Transformer variability.

  • C. Dataset Details: Fixed split artifacts and calibration semantics are reused across WebArena, τ 2-Bench, SkillsBench, and TerminalBench without changing membership or the H=3 label contract.WebArena uses train-internal calibration; the other benchmarks use prepared train/calibration/validation/test or fit/calibration/validation/test fields.
  • C.1. Additional Label Statistics: The positive-prefix rate defines each benchmark’s random-baseline AUPRC, with WebArena’s larger rate reflecting shorter trajectories under the inclusive H=3 window.These prevalence differences motivate reporting both ranking quality and FAR-constrained first-alert diagnostics.
  • C.2. Prefix Label Construction: Under H=3, failed-trajectory prefixes are positive when T − t ≤ 3, yielding up to four positive positions; all other prefixes receive label 0.The convention targets steps immediately preceding the failure point.
  • C.3. Evaluation Protocol and Metrics: Evaluation fixes splits, uses a 10% held-out calibration subset for threshold and DFA score calibration, unlocks test once per variant, and reports three-seed mean ± standard deviation.The protocol also treats undefined zero-denominator ratios as omitted rather than wins.
  • C.3. Evaluation Protocol and Metrics: AUPRC is the primary threshold-free ranking metric, while AUROC, calibration metrics, and thresholded accuracy, precision, recall, F1, and FPR provide auxiliary diagnostics.Trained-monitor thresholds are selected on calibration data and evaluated once on the locked test split; LLM baselines use γ = 0.5.
  • C.4. Alert Lead Time: First-alert diagnostics measure lead time as (T_i − a_i)/T_i, count missed failures as 0, and evaluate calibration-selected successful-trajectory FAR constraints.Early recall requires alerts before the terminal H-step label window; these operating points do not establish intervention utility or deployment-specific alert costs.
  • D. Extended Ablation Results: Extended ablations vary one mechanism at a time through supervised non-sequential probes, StepView field drops, label-geometry confound controls, and per-seed Transformer breakdowns.These controls target recoverable signal, StepView evidence, confounds, and backend variability.

D.1. Main-Table Auxiliary Metrics … D.5. Neural Encoder Diagnostic Controls

The appendix reports auxiliary calibration and operating diagnostics, signal-availability probes, sequence controls, continuous StepView controls, and frozen neural-encoder diagnostics. These comparisons clarify what information prefixes contain and why PrefixGuard retains its fixed TF-IDF StepView encoder.

  • D.1. Main-Table Auxiliary Metrics: Table 14 reports prefix counts, positive-prefix prevalence, Brier scores, AUPRC, and thresholded operating metrics for the main-table cells.Operating metrics use the scored LLM subset at pfail ≥0.5 and stored or replayed non-LLM thresholds; replay-mismatched runs are excluded.
  • D.2. Non-Sequential Supervised Prefix-Signal Probes: Table 15 uses non-sequential supervised probes to test whether observed prefixes contain recoverable warning signal under the same H=3 labels and held-out splits.StepView MLP pools observed StepView TF-IDF vectors without recurrent, Transformer, FSM, DFA, or causal monitor state.
  • D.3. Predictive Process Monitoring Activity-LSTM Control: Table 16 evaluates an outcome-oriented activity-LSTM control using one-hot categorical StepView activities, the same H=3 labels, and benchmark split protocols.The control uses a train-only vocabulary and a single-layer LSTM, without TF-IDF text features, learned PrefixGuard symbols, FSM state, or DFA state.
  • D.4. Continuous StepView Sequence Controls: WebArena continuous sequence controls retain StepView TF-IDF embeddings and causal prefix supervision while removing PrefixGuard’s Gumbel symbolizer and discrete alphabet.The controls score prefixes directly from continuous StepView embeddings using a causal GRU or causal Transformer.
  • D.5. Neural Encoder Diagnostic Controls: Frozen dense encoder controls replace the TF-IDF step encoder while keeping the remainder of the monitor-learning recipe unchanged.They are presented as negative diagnostics addressing whether stronger off-the-shelf semantic embeddings can replace the fixed lexical encoder.
  • D.5. Neural Encoder Diagnostic Controls: Frozen dense encoders do not improve the monitor pipeline, so the main experiments retain TF-IDF as the fixed StepView encoder.StepView preserves sparse operational cues including tool names, statuses, error fragments, file paths, and task-specific identifiers under a small fixed feature budget.

D.6. Position and Task-Prior Confound Controls

Position-only controls remain far below PrefixGuard-GRU, while adding trajectory length creates an intentionally strong but deployment-infeasible oracle signal. Corrected within-prefix scrambling preserves online visibility and labels, sharpening interpretation of content versus sequential-order effects.

  • Position and task-prior controls: Position-only t features stay far below PrefixGuard-GRU on every benchmark, indicating that step position alone cannot explain the main results.The t-only control uses deployment-realistic features without content or future information.
  • Position and task-prior controls: Adding trajectory length T produces an intentionally strong t+T oracle because the label definition creates a near-end-of-trajectory signal when T is known.Trajectory length is future information unavailable at deployment, so this advantage is not exploitable online.
  • Content-scrambled control: Within-prefix scrambling permutes only already-visible steps and preserves each original prefix label, preventing scrambled examples from observing steps that were future under the online order.This corrected no-leakage control separates step content from sequential ordering.
  • Content-scrambled control: WebArena and τ 2-Bench barely change under corrected within-prefix shuffling, sharpening the interpretation of their monitor behavior.The reported WebArena value after shuffling is 0.908.

D.7. PrefixGuard-GRU Calibration Metrics … D.11. Operating-Point Analysis

The appendices assess PrefixGuard’s calibration, field dependence, seed variation, DFA-state alignment, and operating-point tradeoffs. Together, they show benchmark-specific evidence requirements and interpretability limits, while distinguishing ranking quality from deployment-ready alerting.

  • D.7. PrefixGuard-GRU Calibration Metrics: PrefixGuard-GRU calibration uses 15-bin ECE and Brier score on locked test data, with SkillsBench showing notably higher ECE variance across seeds.The SkillsBench variance is attributed to one seed reaching ECE = 0.113 versus 0.054–0.058 for the other two, while Brier is more stable.
  • D.8. StepView Field Ablation: StepView field ablations show benchmark-specific dependencies: post-action result evidence is crucial on WebArena, while observation-only inputs sharply degrade τ 2-Bench and TerminalBench.The audit concludes that no universal single-field dependency holds across all benchmarks.
  • D.10. DFA State Behavioral Alignment: 29 WebArena DFA states were qualitatively aligned using tools, typed text, action arguments, and normalized trajectory position, with all 27 trusted states coded.Two states were excluded as untrusted because they had fewer than 10 calibration prefixes.
  • D.10. DFA State Behavioral Alignment: All 6 WebArena warning states were assigned to distinct, semantically coherent failure-precursor categories in a single-coder, one-seed diagnostic.This is preliminary evidence rather than a validated interpretability study; multi-coder reliability and trajectory-level annotations remain future work.
  • D.10. DFA State Behavioral Alignment: The 20-state τ 2-Bench DFA had 13 trusted states coded, while 7 untrusted states were excluded; its three trusted warning states formed recognizable but weaker behavioral groups.The alignment used the adaptive StepView+GRU run R313, seed 13, and excluded states with fewer than 10 calibration prefixes.
  • D.10. DFA State Behavioral Alignment: SkillsBench and TerminalBench use representative rather than exhaustive state alignments because their DFAs contain 151 and 187 states, respectively.The reported SkillsBench warning states mainly correspond to fragile coding-workflow phases, while the supplied passage does not provide the complete TerminalBench categories.
  • D.11. Operating-Point Analysis: Operating-point analysis selects trajectory-level thresholds using calibration-set successful-trajectory FAR constraints and compares them with prefix-level PR and ROC curves.These diagnostics separate score ranking from alarm burden; practical usefulness depends on deployment-specific costs and reversibility.

E. Horizon Sensitivity · F. LLM Baseline Prompts · G. Observability Ceiling: Proofs

The validation scan favors H=1 for score-based monitor quality, although H=3 remains the fixed comparison contract. The LLM baseline uses full observed prefixes, while the observability proof bounds attainable AUPRC when some failures leave no distinguishing prefix evidence.

  • E. Horizon Sensitivity: H=1 achieves the best validation score-based AUPRC, score-based AUROC, and ECE across all four benchmarks.Larger H increases positive-prefix prevalence and alert lead time.
  • E. Horizon Sensitivity: H=3 remains the fixed operating contract for comparability, despite validation results not supporting it as globally optimal.Future dataset-specific locked-test comparisons should freeze the horizon using validation metrics first.
  • F. LLM Baseline Prompts: The LLM-as-judge baseline receives a fixed system message and a per-prefix user message rendered from the observed StepView prefix.The prompt exposes only the observed prefix and asks for the probability of an imminent-failure warning prefix.
  • F. LLM Baseline Prompts: Roughly 75M input tokens would be required for one full-prefix training example per prompt across four N=200 evaluations, so calibrated examples were not tested.WebArena additionally received a stronger zero-shot DeepSeek-V4-Pro baseline with a 1M context window and thinking disabled.
  • G. Observability Ceiling: Proofs: For fixed observable-positive fraction π, Proposition 1 gives an upper bound on population ranking performance measured by AUPRC.The bound is tight over mixture-model instances and is strictly increasing in π for fixed r.
  • G. Observability Ceiling: Proofs: Monotonicity of the AUPRC bound permits inversion to lower-bound the π required to reach a target population AUPRC.The benchmark figure uses the bound in the forward direction rather than this inverted interpretation.
  • G. Observability Ceiling: Proofs: The observability model represents positive prefixes as a mixture of observable failed prefixes and hidden failures distributionally identical to negative prefixes.The setup defines π as the observable-positive fraction and r as the positive-prefix rate.
  • G. Observability Ceiling: Proofs: The empirical average_precision_score is a consistent estimator of population AUPRC under i.i.d. test sampling with continuous score laws.Consistency is not used in the proof, and the evaluation follows the standard PR-curve convention.

G.1. Proof of Proposition 1 · G.2. MPE Audit Protocol

G.1 proves a tight observability ceiling for population AP under the mixture model, showing how hidden-failure prevalence shapes the best attainable precision-recall curve. G.2 specifies an independent MPE audit protocol and cautions that its finite-sample estimates depend on prefix construction and trajectory length.

  • G.1. Proof of Proposition 1: Precision is maximized at the minimum feasible false-positive rate, yielding the upper envelope used to define the AP ceiling.The recall constraint implies q ≥ (s −π)/(1 −π) for s > π and q ≥0 for s ≤π; precision decreases strictly in q.
  • G.1. Proof of Proposition 1: A(1/2, 1/2) ≈0.821, exceeding the naive linear bound and matching the tight counterexample described in the proof.The boundary cases are A(0, r) = r and A(1, r) = 1.
  • G.1. Proof of Proposition 1: The ceiling is constructively tight: disjoint-support score distributions attain Precmax(s) at every recall level, while π = 0 and π = 1 are tight endpoints.For 0 < π < 1, thresholds traverse regions with FPR = 0 and then Robs = 1, realizing the envelope throughout recall.
  • G.1. Proof of Proposition 1: 0.776, 0.621, 0.430, and 0.478 are the PG-GRU required-π values for WebArena, τ 2-Bench, SkillsBench, and TerminalBench, respectively.These values are descriptive finite-sample diagnostics rather than confidence-certified estimates of the true full-prefix population π.
  • G.2. MPE Audit Protocol: The MPE audit independently estimates whether failed prefixes remain distinguishable from negative-prefix references, without using PrefixGuard scores or plotted AUPRC values.Figure 2 combines ceiling curves, independent probe-based MPE diagnostics, PrefixGuard backend AUPRCs, and required-π markers.
  • G.2. MPE Audit Protocol: WebArena uses all prefixes, whereas τ 2-Bench, SkillsBench, and TerminalBench use matched non-terminal near-end prefixes after removing each terminal prefix.Failed kept prefixes are positive and successful kept prefixes are negative in the matched non-terminal construction.
  • G.2. MPE Audit Protocol: Full-prefix MPE can overestimate π on long benchmarks because many easy early negative prefixes separate from near-end failed prefixes; matched non-terminal auditing reduces this artifact.All estimates remain finite-sample diagnostics rather than confidence-certified population values.
Loading 2605.06455v1…