PaperlayerView exact revision on arXiv

Source-linked AI summary

PrefixGuard: From LLM-Agent Traces to Online Failure-Warning Monitors

Xinmiao Huang, Jinwei Hu, Rajarshi Roy, Changshun Wu, Yi Dong, Xiaowei Huang

arXiv:2605.06455v1cs.AI

TL;DR

Online failure warnings for long-running LLM agents are difficult to synthesize across heterogeneous traces without brittle schemas or deployment-time LLM judging. PrefixGuard induces typed trace adapters offline and trains prefix monitors; across four benchmarks, StepView improves over matched Raw-text controls by +0.03–+0.22 AUPRC.

  • Problem

    Existing runtime verification assumes stable, hand-authored mappings from raw traces to events, while LLM-as-judge approaches are costly for online warning.

    Source: §1 · p. 1

  • Method

    PrefixGuard uses an offline LLM-assisted adapter to map heterogeneous traces into typed StepView records, then trains event abstractions and prefix-risk monitors.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    Across four benchmarks, StepView improves over matched Raw-text control GRU by +0.03–+0.22 AUPRC.

    Source: §1 · p. 2

  • Takeaways & Limitations

    At a 10% calibration FAR cap, WebArena supports terminal-window triage, whereas τ 2-Bench is the clearest intervention setting.

    Source: §5.5 · p. 9

    Warnings require visible prefix evidence, low-FAR separability, and a deployment action after an alert; evaluation uses fixed horizon and FAR caps.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large language model (LLM) agents now execute long, tool-using tasks where final outcome checks can arrive too late for intervention. Online warning requires lightweight prefix monitors over heterogeneous traces, but hand-authored event schemas are brittle and deployment-time LLM judging is costly. We introduce PrefixGuard, a trace-to-monitor framework with an offline StepView induction step followed by supervised monitor training. StepView induces deterministic typed-step adapters from raw trace samples, and the monitor learns an event abstraction and prefix-risk scorer from terminal outcomes. Across WebArena, $τ^2$-Bench, SkillsBench, and TerminalBench, the strongest PrefixGuard monitors reach 0.900/0.710/0.533/0.557 AUPRC. Using the strongest backend within each representation, they improve over raw-text controls by an average of +0.137 AUPRC. LLM judges remain substantially weaker under the same prefix-warning protocol. We also derive an observability ceiling on score-based area under the precision-recall curve (AUPRC) that separates monitor error from failures lacking evidence in the observed prefix. For finite-state audit, post-hoc deterministic finite automaton (DFA) extraction remains compact on WebArena and $τ^2$-Bench (29 and 20 states) but expands to 151 and 187 states on SkillsBench and TerminalBench. Finally, first-alert diagnostics show that strong ranking does not imply deployment utility: WebArena ranks well yet fails to support low-false-alarm alerts, whereas $τ^2$-Bench and TerminalBench retain more actionable early alerts. Together, these results position PrefixGuard as a practical monitor-synthesis recipe with explicit diagnostics for when prefix warnings translate into actionable interventions.

1. Introduction

PrefixGuard synthesizes online prefix-warning monitors from raw LLM-agent traces by inducing typed trace adapters offline and learning failure-aligned event abstractions and risk scores. Its four-benchmark evaluation examines warning signal, representation, finite-state auditability, and whether risk rankings yield actionable low-false-alarm alerts.

Introduction

  • Online warning is needed because erroneous actions in long-horizon, high-stakes tasks can cause irreversible damage before final verification, while runtime schemas are brittle and per-prefix LLM judging is costly.Predictive prefix classifiers may recover signal but do not by themselves provide calibrated monitor state or inspectable symbolic artifacts [35].Source: §1 · p. 1
  • PrefixGuard uses one-time offline StepView induction to create deterministic typed adapters, then learns a discrete failure-aligned alphabet and trains replaceable neural or structured monitor backends.Hard symbols can also be compiled into DFAs for post-hoc audit; the instantiated backends include GRU, Transformer, and soft-FSM monitors.Source: §1 · p. 2, Figure 1 · p. 2
  • +0.03–+0.22 AUPRC: StepView improves over the matched Raw-text control GRU across four benchmarks, with field dependence varying by benchmark.The benchmarks cover browser, dialogue, coding, and CLI traces.Source: §1 · p. 2
  • Post-hoc DFA extraction remains compact on WebArena and τ 2-Bench but expands on SkillsBench and TerminalBench, while neural monitors provide the strongest warning performance.The evaluation characterizes WebArena as the clearest compact regime, τ 2-Bench as compact but concentrated, and the other two benchmarks as larger monolithic DFAs.Source: §1 · p. 2
  • WebArena ranks failures well but is not alarm-separable for early low-FAR alerts, whereas τ 2-Bench and TerminalBench retain stronger failed-trajectory and early-intervention recall.Prevalence calibration, observability probes, and first-alert diagnostics show why AUPRC alone does not establish deployment utility.Source: §1 · pp. 2–3

2. Related Work

Related work covers retrospective agent evaluation, formal trace monitoring, process prediction, and inspectable automata. PrefixGuard targets online prefix-risk warning over heterogeneous LLM-agent traces.

LLM-agent evaluation and judges

  • Unlike terminal verifiers [46] [3] [20] [23] and retrospective LLM judges [45], PrefixGuard learns domain-calibrated prefix statistics for per-step risk scoring without deployment-time LLM inference.Source: §2 · p. 3

Runtime verification and specification mining

  • Runtime verification and specification mining assume stable vocabularies; StepView induces typed adapters from raw examples without deployment-time LLM dependence.LLM-agent traces mix browser actions, tool calls, and dialogue turns across evolving formats.Source: §2 · p. 3

Trace abstraction and predictive process monitoring

  • Dialogue-flow [9] [34], predictive-process [35], and alarm systems target recurrent behaviors, typed events, or interventions; PrefixGuard jointly learns representations for fixed-horizon warnings.Its finite pre-failure warning window follows early time-series classification.Source: §2 · p. 3

Auditable neural-symbolic monitors

  • Prior automata methods [40] [32] [37] rely on fixed symbolic observations or queryable systems; PrefixGuard uses post-hoc DFA extraction, reliable for audit only when automata are compact and risk-separating.Source: §2 · p. 3

3. Problem Formulation

Prefix warning scores causal trace prefixes for imminent failure, defined by a fixed horizon and evaluated by AUPRC. The section also establishes an observability-based AUPRC ceiling as a diagnostic for failures that lack visible evidence in the current prefix.

3. Problem Formulation

  • A prefix is positive only for a failed trajectory with at most H remaining steps; the inclusive horizon allows up to H+1 positive positions.All prefixes of successful trajectories and earlier prefixes of failed trajectories are negative.Source: §3 · p. 3
  • The monitor maps each observed prefix τ:t to a risk score using only its cumulative history, preserving earlier errors and recovery attempts.The target outcome is binary, with success or failure determined by a task-specific verifier.Source: §3 · p. 3
  • Training minimizes expected binary cross-entropy over prefixes, with per-trajectory 1/T normalization to weight trajectories equally despite different lengths.Source: §3 · pp. 3–4
  • AUPRC is the primary evaluation measure for how well prefix risk scores rank imminent-failure targets.At test time, the monitor alerts when its score exceeds a validation-calibrated threshold.Source: §3 · p. 4

3.1. A Diagnostic Observability Ceiling

  • The AUPRC ceiling arises because hidden positive prefixes have the same observed-representation distribution as negatives, while observable positives form fraction π.The analysis builds on prevalence-sensitive PR analysis [7] and contaminated-distribution label-noise models.Source: §3.1 · p. 4
  • The observability bound is tight and strictly increases with π, with A(0, r)=r and A(1, r)=1.The ceiling applies to trace-only monitors with continuous score distributions.Source: §3.1 · p. 4
  • Forward π grids calibrate the AUPRC scale at each benchmark prevalence, but grid crossings do not estimate the true latent observability fraction.The ceiling is used only as an evaluation diagnostic.Source: §3.1 · p. 4

4. Method

PrefixGuard converts raw agent traces into online failure-warning monitors by combining StepView adapters, learned event abstractions, and prefix-risk scoring; hard symbols can also support DFA audit diagnostics.

4.1. StepView: LLM-Assisted Adapter Induction

  • StepView replaces format-specific manual parsers with an LLM-assisted offline adapter that maps sampled training traces into a fixed canonical schema.The adapter is fixed before monitor training; validation, test, and deployment use it without LLM inference or step-level annotation, and review is limited to structural validity.Source: §4.1 · p. 4

4.2. TF-IDF Step Encoder

  • The TF-IDF step encoder serializes tagged StepView fields, fits and freezes its vectorizer on training steps, and retains 4,096 top-frequency unigram and bigram features.The resulting vector is L2-normalized and emphasizes corpus-distinctive n-grams while downweighting common boilerplate.Source: §4.2 · p. 5

4.3. Differentiable Event Abstraction Layer

  • A two-layer projection with GELU and Gumbel-softmax maps each TF-IDF step embedding to soft assignments over K event symbols, jointly optimized with the monitor against warning loss.The assignments are passed directly to the prefix monitor, allowing gradients to shape the event alphabet through the warning objective rather than relying on fixed symbols.Source: §4.3 · p. 5

4.4. Prefix-Warning Monitor

  • The monitor scores each prefix from learned soft symbols using GRU, causal Transformer, or soft-FSM backends, with GRU as the cross-domain default.Training combines binary cross-entropy over prefix positions with a symbol-balance regularizer. During deployment, each raw step is converted by StepView, TF-IDF encoded, assigned a soft symbol representation, and scored by the selected backend.Source: §4.4 · pp. 5–6

4.5. Extracted DFA Monitor

  • PrefixGuard-DFA is extracted after training by fitting an RPNI-style automaton to hard-symbolized training traces and calibrating state risks on held-out trajectories.At deployment it follows DFA transitions and alerts when current-state risk exceeds a threshold; the extraction is an audit diagnostic, not an assumption that one monolithic DFA suits every benchmark.Source: §4.5 · p. 6

5. Experiments

Across four benchmarks, PrefixGuard monitors recover prefix-warning signal and StepView improves ranking over raw serialization, but prevalence, visible evidence, and low-false-alarm timing shape whether warnings are actionable.

5.1. Experimental Setup

  • Experiments used fixed train/calibration/validation/test splits and H=3 prefix labels, with score-based AUPRC as the primary metric and positive-prefix rate r as its random baseline.Calibration selected thresholds; auxiliary AUROC, calibration, and operating-point diagnostics were reported separately.Source: §5.1 · p. 6

5.3. RQ2: Typed Evidence Beyond Raw Serialization

  • PrefixGuard’s strongest monitors reached 0.900/0.710/0.533/0.557 AUPRC across WebArena, τ 2-Bench, SkillsBench, and TerminalBench, while StepView gains over Raw-text were +0.029/+0.113/+0.218/+0.187 AUPRC (average +0.137).Source: Table 2 · p. 7, §5.2 · p. 7, §5.3 · p. 7

5.2. RQ1: Recoverable Prefix-Warning Signal

  • The best zero-shot judge reached 0.450 AUPRC on WebArena, stayed below 0.40 on τ 2-Bench, and fell near 0.10 on SkillsBench and TerminalBench, below learned PrefixGuard monitors.Source: Table 2 · p. 7, §5.2 · p. 7

5.4. RQ3: Signal Preservation Under Finite-State Compression

  • Neural monitors ranked strongest and soft-FSM remained competitive on WebArena and τ 2-Bench, while exact DFA compression yielded 29/20 states there versus 151/187 on SkillsBench/TerminalBench.Exact DFA extraction preserved less ranking signal, especially on τ 2-Bench and TerminalBench.Source: Table 2 · p. 7, Table 4 · p. 8, §5.4 · p. 8

5.3. RQ2: Typed Evidence Beyond Raw Serialization

  • Field ablations showed WebArena was result-sensitive (−0.204), TerminalBench status-sensitive (−0.106), and observation-only inputs removed signal on τ 2-Bench and TerminalBench (−0.266/−0.270).The representation comparison held monitor model, split, horizon, and metric fixed while replacing only the input view.Source: §5.3 · p. 7, Table 3 · p. 8

5.5. RQ4: From Prefix Ranking to Deployment Utility

  • At H=3, WebArena’s positive-prefix rate r=0.363 versus ≈0.07–0.09 elsewhere makes raw AUPRC prevalence-conditioned; PG-GRU’s required observable fractions were 0.776/0.621/0.430/0.478.The independent MPE probe found high matched-prefix separability on τ 2-Bench and TerminalBench, so their lower raw AUPRCs do not imply that failures are hidden.Source: Figure 2 · p. 8, §5.5 · pp. 8–9
  • At a 10% calibration FAR cap, WebArena’s 0.900-AUPRC ranking still yielded terminal-window triage; τ 2-Bench was clearest for intervention, TerminalBench retained lead time, and SkillsBench alerts came late.The results distinguish risk ranking from the requirement for low-false-alarm alerts early enough to support intervention.Source: Figure 3 · p. 9, §5.5 · p. 9

6. Limitations and Conclusion … B.8. FSM Head and DFA Extraction

PrefixGuard converts heterogeneous agent traces into typed prefixes and online risk scores without deployment-time LLM judging, but actionable warnings still depend on visible prefix evidence, low-false-alarm separability, and an available response. The accompanying appendices specify how the adapters, monitors, and finite-state audits are built and where their limits lie.

6. Limitations and Conclusion

  • PrefixGuard learns failure-aligned event symbols and online risk scores from typed trace prefixes, but warnings require visible evidence, low-FAR separability, and a deployment action.Its fixed horizon and FAR caps are evaluation controls; MPE coordinates are probe- and protocol-specific diagnostics, not certified population π estimates. DFA extraction remains audit-friendly only in compact regimes, with deployment-visible routed extraction left for future work.Source: §6 · p. 9

A. Extended Related Work

  • PrefixGuard learns temporal risk from heterogeneous trace prefixes rather than relying on terminal grading or a predefined event alphabet.The related-work discussion frames this as prefix prediction over typed, partially benchmark-specific records and notes that observability limits trace-only AUPRC when some failed prefixes are indistinguishable from negatives. DFA extraction is treated as an audit artifact, not a guarantee of compact automata.Source: A. Extended Related Work · pp. 12–13

B. Implementation Details; B.2. Compute Resources; B.3. LLM API Checkpoints

  • The monitor pipeline fits its encoder on training StepView text, selects checkpoints by validation AUPRC, and uses LLM calls only for offline induction or judge baselines.Optional DFA extraction applies RPNI to hard-assigned training symbols and calibrates state risk on a held-out split. Final runs used one A100 per job, with training times ranging from about 30–45 minutes to 2–3.5 hours.Source: B. Implementation Details · p. 13, §B.3 · p. 14, §B.2 · p. 14

B.4. StepView Canonicalization

  • All four benchmarks achieve 100% tool_name and status coverage with 0% fallback across 1.8M steps; WebArena result fill is 87%.The variable result fill reflects navigation actions without structured returns, not parsing failures; no human re-editing of extraction logic was required.Source: §B.4 · p. 15, Table 7 · p. 15

B.5. StepView Adapter-Induction Design and Prompt

  • StepView induction uses a deterministic 12-step pack from the first 64 training trajectories, with fixed bucket quotas of 4/4/2/2 and training-only data.The protocol uses gpt-5.4-nano at temperature 0.0, validates the adapter schema and executor support, and freezes the accepted adapter before downstream conversion.Source: §B.5 · p. 15

B.6. TF-IDF Encoding

  • A single frozen TF-IDF vectorizer caps the StepView vocabulary at d = 4096 features across neural monitors and extracted-DFA audits.The common cap holds the lexical feature budget constant across methods while retaining benchmark-specific unigrams and bigrams.Source: §B.6 · p. 16

B.7. Trainable Monitor Hyperparameters

  • Trainable monitors use H = 3, 24 epochs, batch size 64 trajectories, and a fixed balance-loss weight λbalance = 0.1.The balance term discourages symbol collapse while prefix-warning binary cross-entropy remains the dominant objective; sequences are capped at 64 steps and longer traces are truncated to their most recent steps.Source: §B.7 · p. 16

B.8. FSM Head and DFA Extraction

  • SkillsBench DFA audits show 1.47% abstention and 117 ± 12 seed-level states, while WebArena and TerminalBench abstention is about 0.1% or lower.The soft-FSM state budget matches K to Qmax; TerminalBench uses K=32, Qmax=32, while SkillsBench selected K=16. RPNI induction filters ambiguous training traces, and held-out calibration assigns per-state risk.Source: §B.8 · p. 17, Table 8 · p. 17

B.9. Automated Cross-Benchmark DFA Posthoc Audit

The automated posthoc audit finds that DFA compactness and prefix concentration vary across benchmarks, while validation-only routed-DFA diagnostics motivate hierarchical extraction without establishing deployment performance.

B.9. Automated Cross-Benchmark DFA Posthoc Audit

  • 151 SkillsBench states and 187 TerminalBench states contrast with WebArena’s most compact, risk-separating DFA; τ 2-Bench’s top five states cover 92.0% of prefixes.These results weaken claims that extracted DFAs are uniformly easy to inspect across benchmarks.Source: §B.9 · pp. 17–18
  • The fully automatic audit separates DFA inspection from human interpretability and does not assess human agreement, actionability, or annotation reliability.It reports calibrated DFA metrics, trusted-state coverage, warning-state counts, and concentration of routed prefixes in the five most frequent states.Source: §B.9 · pp. 17–18
  • Validation-only routed-DFA diagnostics for SkillsBench and TerminalBench hold the trained monitor and hard-symbol protocol fixed, changing extraction to deployment-visible routes; they motivate future work, not locked-test claims.The diagnostics compare against route-only calibration baselines, which use no DFA transitions.Source: Table 10 · p. 18, §B.9 · p. 18

C. Dataset Details … D. Extended Ablation Results

The appendices define fixed dataset splits, inclusive H=3 prefix labels, evaluation and alert diagnostics, and the scope of extended ablations. They also clarify that ranking metrics and first-alert operating points address different aspects of monitoring.

C. Dataset Details

  • Fixed benchmark split artifacts and calibration semantics are reused throughout; no main-paper result changes split membership or the H=3 label contract.WebArena uses its repository protocol split with train-internal calibration, while the other benchmarks use their prepared split fields listed in Table 11.Source: C. Dataset Details · p. 19, Table 11 · p. 19

C.1. Additional Label Statistics

  • Positive-prefix prevalence sets the random AUPRC baseline, while shorter WebArena trajectories create a larger positive-prefix rate than the other benchmarks under the same H=3 window.The appendix also summarizes verifier-side failure rates, which identify failures structurally opaque to prefix monitors; it gives no cell values in the supplied passages.Source: Table 12 · p. 19, §C.1 · p. 19

C.2. Prefix Label Construction

  • Under the inclusive H=3 convention, failed-trajectory prefixes are positive when T − t ≤ 3, covering up to four positions before failure but not earlier root-cause steps.Successful-trajectory prefixes and failed-trajectory prefixes with more than three remaining steps are negative.Source: §C.2 · p. 19

C.3. Evaluation Protocol and Metrics

  • Evaluation fixes splits, selects trained-monitor thresholds on calibration data, unlocks each test split once per variant, and uses average precision (AP) as the primary threshold-free AUPRC ranking metric.Multi-seed results use three independent training seeds and are reported as mean ± standard deviation; LLM baselines use γ = 0.5, while trained-monitor thresholds are calibration-selected.Source: §C.3 · pp. 19–20

C.4. Alert Lead Time

  • First-alert diagnostics measure successful-trajectory FAR, early recall before the terminal H-step label window, and lead time, but do not establish intervention utility or deployment-specific alert costs.Lead time is the fraction of a failed trajectory remaining after its first alert; missed failures count as zero, and thresholds are selected under calibration-set successful-trajectory FAR constraints.Source: §C.4 · pp. 20–21, Table 13 · p. 21

D. Extended Ablation Results

  • The extended ablations are organized around recoverable signal, StepView evidence, label-geometry confounds, and per-seed Transformer breakdowns.The supplied passage states the ablation axes but reports no ablation outcomes.Source: D. Extended Ablation Results · p. 21

D.1. Main-Table Auxiliary Metrics … D.5. Neural Encoder Diagnostic Controls

The appendix tests PrefixGuard against auxiliary signal probes and alternative monitor and encoder controls. It also reports calibration and operating-point diagnostics alongside the main ranking metric.

D.1. Main-Table Auxiliary Metrics

  • Table 14 reports locked test-split prefix means, including prevalence, Brier score, and thresholded operating metrics for the main-table cells.Operating metrics use the scored LLM subset at pfail ≥0.5 and stored or replayed non-LLM thresholds; replay-mismatched runs are excluded.Source: Table 14 · p. 22, §D.1 · p. 22
  • Auxiliary calibration and thresholded metrics complement AUPRC by exposing cases where strong ranking quality may still carry a costly false-positive operating point.The Pareto diagnostic plots AUPRC against false-positive rate, with recall encoded at the same alert threshold.Source: Figure 4 · p. 23, §D.1 · p. 24

D.2. Non-Sequential Supervised Prefix-Signal Probes

  • Non-sequential supervised-prefix probes test whether observed prefixes contain outcome-related warning signal without recurrent monitoring, differentiable symbolization, or DFA state.The StepView MLP pools observed StepView TF-IDF vectors; these probes are signal-availability diagnostics, not deployable online monitors.Source: §D.2 · p. 24, §D.1 · p. 24, Table 15 · p. 24

D.3. Predictive Process Monitoring Activity-LSTM Control

  • Across all four benchmarks, PG-GRU remains substantially higher than the outcome-oriented activity-LSTM control.The control uses one-hot StepView activities with a single-layer LSTM, H=3 labels, and the same benchmark split protocols, but no TF-IDF features, learned PrefixGuard symbols, FSM state, or DFA extraction.Source: §D.3 · p. 24, Table 16 · p. 24, §D.1 · p. 24

D.4. Continuous StepView Sequence Controls

  • WebArena continuous GRU and Transformer controls retain StepView TF-IDF embeddings and causal prefix scoring while removing PrefixGuard’s learned discrete event abstraction.They use the same split, H=3 labels, seed, training and calibration protocol, frontend, and StepView text mode as the single-seed supervised WebArena controls.Source: §D.4 · p. 24, Table 17 · p. 25

D.5. Neural Encoder Diagnostic Controls

  • Frozen dense encoders do not improve the monitor pipeline, so the main experiments retain TF-IDF as the fixed step encoder.The passage attributes TF-IDF’s fit to sparse operational cues such as tool names, statuses, errors, paths, and identifiers; it suggests fine-tuning, structured field pooling, or monitor-specific contrastive objectives for future neural-encoder work.Source: §D.5 · p. 25, Table 18 · p. 25

D.6. Position and Task-Prior Confound Controls

Matched controls indicate that step position and task priors alone do not explain PrefixGuard-GRU’s results. Within-prefix shuffling shows that chronological order matters more on SkillsBench and TerminalBench than on WebArena and τ 2-Bench.

D.6. Position and Task-Prior Confound Controls

  • Across all four benchmarks, t-only and task-prior controls remain below PrefixGuard-GRU, though task priors carry non-trivial signal on WebArena and TerminalBench.The t-only control uses deployment-realistic position features without content or future information; task priors use task id, whitelisted metadata, and pre-observation context.Source: §D.6 · p. 25, Table 19 · p. 26
  • The t+T control confirms a near-end-of-trajectory signal from the label rule t≥T−H, but its use of future trajectory length makes it unavailable at deployment.The result motivates keeping full trajectory length out of online scoring.Source: §D.6 · p. 25
  • The corrected scrambled-control reruns use soft AUPRC only, with final DFA/RPNI induction canceled for SkillsBench and skipped by design for TerminalBench.WebArena and τ 2-Bench results are from corrected canonical summaries; SkillsBench uses the best soft-validation epoch, and TerminalBench uses the completed corrected scrambled-only soft rerun.Source: §D.6 · p. 26
  • Within-prefix shuffling changes AUPRC from 0.908 →0.901 on WebArena, 0.687 →0.681 on τ 2-Bench, 0.526 →0.269 on SkillsBench, and 0.548 →0.375 on TerminalBench.The smaller drops indicate content and task-local evidence dominate on WebArena and τ 2-Bench, while order or temporal structure carries material signal on SkillsBench and TerminalBench; these are soft-AUPRC controls.Source: §D.6 · p. 26, Table 20 · p. 26

D.7. PrefixGuard-GRU Calibration Metrics … D.11. Operating-Point Analysis

These appendices report calibration and field-ablation diagnostics, describe behaviorally named states in extracted DFAs, and distinguish score ranking from deployment operating points. The qualitative DFA alignment is preliminary and single-coder, while practical alert usefulness depends on deployment-specific costs and reversibility.

D.7. PrefixGuard-GRU Calibration Metrics

  • PrefixGuard-GRU calibration was evaluated on the locked test split with 15-bin ECE and Brier score, and SkillsBench ECE varied substantially across seeds.WebArena, SkillsBench, and TerminalBench used three-seed averages, while τ 2-Bench used two; one SkillsBench seed had ECE = 0.113 versus 0.054–0.058 for the other two, while Brier score was more stable.Source: §D.7 · p. 26

D.8. StepView Field Ablation

  • The StepView field-drop audit found post-action result evidence crucial on WebArena, but no universal single-field dependency; observation-only inputs sharply degraded τ 2-Bench and TerminalBench.TerminalBench also lost signal when status was removed; the audit reports locked-test soft AUPRC under matched single-seed protocols.Source: §D.8 · p. 26, Table 22 · p. 27

D.9. Transformer Per-Seed Breakdown

  • Table 23 reports locked-test score-based AUPRC for each PrefixGuard-Transformer seed across WebArena, τ 2-Bench, SkillsBench, and TerminalBench.The supplied passage identifies the per-seed breakdown but does not provide the individual values.Source: Table 23 · p. 27

D.10. DFA State Behavioral Alignment

  • WebArena’s six warning states mapped to five coherent failure-precursor patterns, including navigation resets, error signals, click loops, misaligned searches, external-search redirects, and unproductive browsing.All 27 trusted states in the 29-state DFA were coded; two states were excluded as untrusted. The alignment is preliminary, based on one coder and one seed, and does not establish validated interpretability.Source: §D.10 · pp. 28–29, Table 25 · p. 28
  • τ 2-Bench’s trusted warning states had recognizable but weaker behavioral groups, while prefix concentration and a flatter risk range limited the diagnostic strength of its DFA alignment.Thirteen of 20 states were trusted and coded; seven were excluded as untrusted. The top five states covered 92.0% of prefixes, and a high-risk, low-support state was excluded by the trusted-state filter.Source: §D.10 · p. 29, Table 26 · p. 29
  • SkillsBench and TerminalBench DFA states were often behaviorally nameable, but their larger automata weakened auditability and produced warning states specific to task and tool families.SkillsBench had 151 states and 27 trusted warning states; TerminalBench had 187 states and 6 trusted warning states. These alignments are diagnostic artifacts, not standalone human-interpretability evidence.Source: §D.10 · p. 29, Table 27 · p. 30

D.11. Operating-Point Analysis

  • Trajectory-level operating points were selected by calibration-set successful-trajectory false-alarm constraints, while prefix-level PR and ROC curves showed ranking before a deployment threshold was chosen.The diagnostics separate score ranking from alarm burden; practical intervention usefulness depends on deployment-specific costs and reversibility.Source: §D.11 · p. 30, Figure 6 · p. 31

E. Horizon Sensitivity · F. LLM Baseline Prompts · G. Observability Ceiling: Proofs

The sections report a validation horizon scan, define the zero-shot LLM-judge protocol and its cost limits, and formalize an observability-based ceiling on population AUPRC. They also clarify that the horizon choice and ceiling diagnostics require careful interpretation.

E. Horizon Sensitivity

  • H=1 yielded the best validation score-based AUPRC, score-based AUROC, and ECE across all four benchmarks, while larger horizons raised positive-prefix prevalence and alert lead time.Table 28 reports a validation-only scan; its values do not use locked-test results.Source: Table 28 · p. 31, E. Horizon Sensitivity · p. 31
  • H=3 remains the fixed operating contract for locked-test comparability, not a validation-supported global optimum; future dataset-specific horizons should be frozen on validation before one locked-test evaluation.The paper recommends choosing the horizon from validation metrics, then evaluating it once on the locked test split.Source: E. Horizon Sensitivity · p. 31

F. LLM Baseline Prompts

  • The LLM judge receives the observed StepView prefix in a per-prefix user message alongside a fixed system message, then returns an integer 0–100 p_fail in one JSON object under imminent-warning labels.A failing trajectory is still negative when failure is not imminent within WARNING_HORIZON_STEPS.Source: F. LLM Baseline Prompts · p. 32
  • Few-shot prompting was skipped because one full-prefix example was estimated to require roughly 75M input tokens across four N=200 evaluations; a zero-shot DeepSeek-V4-Pro WebArena run reached 0.450 AUPRC.The baseline used a 1M context window with thinking disabled and parsed all prefixes; few-shot monitors would require a separate compression or retrieval design.Source: F. LLM Baseline Prompts · p. 32

G. Observability Ceiling: Proofs

  • Under observability model (A2), positive prefixes mix observable failed prefixes with hidden failures distributionally identical to negatives; for positive-prefix rate r, Proposition 1 upper-bounds population AUPRC at fixed observable fraction π.The bound applies to monitors with continuous score distributions.Source: G. Observability Ceiling: Proofs · p. 33
  • The ceiling is tight over mixture-model instances and strictly increases with π, so its monotonicity permits a lower bound on the observable fraction needed to reach a target population AUPRC.The theorem gives A(0, r) = r and A(1, r) = 1.Source: G. Observability Ceiling: Proofs · pp. 32–33
  • Figure 2’s ceiling curves are conditional on assumed π, while its MPE markers are finite-sample diagnostics rather than certified population values; average_precision_score is a consistent estimator under i.i.d. test sampling with continuous score laws, following PR-curve conventions [7].The bound calibrates AUPRC scale without estimating the true population π.Source: G. Observability Ceiling: Proofs · pp. 32–33

G.1. Proof of Proposition 1 · G.2. MPE Audit Protocol

The proof establishes a tight observability ceiling on population AP, while the MPE audit supplies independent, finite-sample diagnostics of prefix evidence. The audit’s prefix construction differs for short WebArena trajectories and longer benchmarks.

G.1. Proof of Proposition 1

  • The observability ceiling A(π,r) is attainable at every recall level, with disjoint-support score distributions giving tightness also at π = 0 and π = 1.At π = r = 1/2, the ceiling is approximately 0.821, matching the tight counterexample to the naive linear bound.Source: §G.1 · p. 34
  • For 0 < π < 1, A(π,r) strictly exceeds the naive linear bound r + π(1 − r), because precision remains above r over a positive-measure recall interval.The ceiling also strictly increases with π.Source: §G.1 · p. 34
  • Required-π values for PG-GRU are 0.776, 0.621, 0.430, and 0.478 on WebArena, τ2-Bench, SkillsBench, and TerminalBench, respectively.These are descriptive finite-sample diagnostics, not confidence-certified estimates of true full-prefix population π; SkillsBench’s explicit-evidence anchor is πE = 0.489.Source: §G.1 · p. 34

G.2. MPE Audit Protocol

  • The audit uses all-prefix H=3 labels for short WebArena trajectories, but matched non-terminal near-end prefixes for τ2-Bench, SkillsBench, and TerminalBench.Prefixes expose only chronological status, action_text, and result_text, with no future suffix, failure bucket, monitor score, or AUPRC-derived quantity.Source: §G.2 · p. 35
  • An independent TF-IDF logistic-regression probe scores held-out audit prefixes, and a trimmed lower-tail CDF-ratio estimator uses those scores to estimate the hidden-negative mixture weight.The tail trim is 0.2; bootstrap intervals resample positive and negative held-out scores separately for 200 replicates.Source: §G.2 · p. 35
  • All-prefix MPE can overestimate π on long benchmarks because easy early negatives separate from near-end failures, so the audit matches non-terminal windows there.For WebArena, matched non-terminal sampling leaves only 46 held-out successful near-end negatives and yields the coarse interval 0.746 [0.529, 0.921], motivating its all-prefix marker.Source: §G.2 · p. 35

Related papers

Loading 2605.06455v1…