OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport

1 Introduction

• Online safeguards add latency and cost through an extra agent call, while post-hoc evaluation finds issues only after the run has been paid for.Existing approaches also do not examine run topology, where loops and out-of-order calls appear.Source: §1 · p. 1
• OnTrack models agent steps and their dependencies as a growing DAG, comparing it with successful references using optimal transport to flag issues in about a millisecond per step.Its contribution is making the alignment operate on a live stream, rather than simply rerunning the post-hoc method from Otap [4].Source: §1 · pp. 1–2
• OnTrack supports reference-free operation: schemas alone retain loop and stall detection and prerequisite checks for irreversible actions, while event logs alone retain loop and stall detection.Capabilities switch off as their required inputs disappear.Source: §1 · p. 2
• Prefix comparisons must avoid penalizing an agent for reference steps it has not reached yet, since the same reference coverage can mean progress or a stall depending on trajectory length.A comparison should focus on the reachable part of a reference rather than the whole completed run.Source: §1 · p. 2
• New dependencies may become visible only when later steps use earlier outputs, so judging a new step’s provisional structure immediately risks false positives.Online dependency extraction is harder because the monitor cannot look ahead [4].Source: §1 · p. 2
• Streaming decisions use per-step signals to continue, warn, or halt, rather than a fixed threshold on an aggregate score whose meaning changes with trajectory length.The method uses a short grace window for new steps; one warm-started solver iteration per event takes about a millisecond, with convergence only before intervention.Source: §1 · p. 2

2 Related Work

• Outcome-only benchmarks [5] [6] [7] omit the steps behind a result, while text metrics [8] assume one correct wording and can rank a broken reference-like plan above a valid differently worded one.Prior work [4] found this ranking can fall below chance.Source: §2 · p. 2
• LLM judges [2] and process reward models [3] score individual steps, but judges add latency, cost, and run-to-run noise, while reward models require step-level annotations.These methods also wait until a run has finished.Source: §2 · p. 2
• OnTrack uses standard optimal-transport components: entropic regularization and Sinkhorn iterations, unbalanced transport, Gromov–Wasserstein, and fused Gromov–Wasserstein [12].Warm-started incremental Sinkhorn schemes already exist; OnTrack’s changing support and reference marginal are described as a new combination.Source: §2 · p. 3
• Otap [4] supplied the graph formalism and node costs, and its ablations found structure important, a 16.9-AUROC-point loss from using one reference, and heuristic dependency extraction as the largest error source.Those findings constrained OnTrack’s streaming design.Source: §2 · p. 3
• Post-hoc Otap [4] scores completed trajectories by aligning attributed execution DAGs; OnTrack instead addresses whether a run should be allowed to continue.The streaming design adds frontier masking, provisional-structure handling, a verdict engine, and calibration considerations absent from batch scoring.Source: §2 · p. 3

3 Problem Formulation

• Each event records an action, tool, arguments, generated artifact, and dependency metadata; the prefix graph has one node per event and may gain dependency edges retroactively.The metadata can be incomplete depending on the harness.Source: §3 · p. 3
• Dependencies can be inferred from metadata or tool-schema artifact identifiers, but deterministic string matching misses links without matchable identifiers, leaving the graph potentially incomplete.The monitor maintains an artifact ledger to recover the step that first surfaced a referenced artifact.Source: §3 · p. 3
• When known-good references are available, each is represented as a graph with a precomputed matrix of normalized dependency-hop distances; the growing prefix graph has an analogous matrix updated row by row.Figure 1 depicts the stream, references, distance matrices, and stepwise alignment.Source: §3 · pp. 3–4, Figure 1 · p. 4
• With references and schemas, the monitor can align structure, measure per-step leakage, and gate irreversible actions; schemas without references retain pathology detection and prerequisite checks, while no prior knowledge leaves vital-sign monitoring.The capabilities diminish as their inputs disappear, and verdict authority is scaled by reference provenance.Source: §3 · p. 4
• At every event, the evaluator must output diagnostics and a verdict under a compute budget far below tool-call latency, then match the batch evaluator’s output when the stream terminates.The verdict set includes OK, EXPLORING, WARN, LOOP, STALLED, CAUSAL INVERSION, OFF TRACK, and BLOCK.Source: §3 · p. 4

4 The OnTrack Evaluator

• Three decoupled layers separate always-on intrinsic vital signs, reference-based transport alignment, and a pre-execution policy gate for irreversible actions.The gate is the only synchronous component and requires tool schemas.Source: §4 · p. 5

4.1 Mass semantics on prefixes

• Constant per-node agent mass prevents matched prefix steps from leaking solely because masked reference mass is below one; termination may renormalize for batch comparability.The agent’s total mass grows as t/¯m.Source: §4.1 · p. 5

4.2 The prefix problem: frontier-masked marginals

• Frontier masking activates reference nodes only after their parents are satisfied, preserving order freedom for executable independent steps and distinguishing hallucinations, valid jumps, and causal inversions.A leaking step is checked against future reference nodes to distinguish these cases.Source: §4.2 · p. 6

4.3 Keeping the quadratic term: streaming conditional gradient

• One warm-started conditional-gradient iteration updates alignment per event; a converged solve confirms would-be interventions before any intervention is taken.This prevents acting on a stale linearization.Source: §4.3 · p. 6

4.4 Provisional structure

• New nodes’ structural rows are discounted while artifacts are young, then become exact when consumed; finalization disables age weights so emission age does not affect the final score.The bound concerns structural cost entries, not the full transport solution.Source: §4.4 · p. 7

4.5 Signals, lifecycle, and verdicts

• Leaking nodes receive an exploration grace period, while schema-checked irreversible proposals can be blocked before execution when prerequisites or reference alignment fail.The default grace TTL is three events; reversible actions are never blocked.Source: §4.5 · p. 7

4.6 Multi-reference streaming and reference regimes

• Independent reference couplings are aggregated by soft-min; reference provenance limits intervention authority, while no-reference operation retains reference-free diagnostics and the schema gate.Planner-generated references can inform but never trigger an automatic halt.Source: §4.6 · p. 8

Anytime reduction to the implemented batch procedure

• At termination, finalization returns the same score as batch evaluation using the same deterministic implemented procedure, provided online and batch dependency graphs agree.The finalization path removes the mask, renormalizes marginals, disables age weights, and uses the batch convention.Source: §5 · p. 8

Qualifications

• This reduction is not a guarantee of exact UFGW minimization: it retains clipped gradients and surrogate diagnostics, and online dependency extraction may differ from batch extraction.Even an exact solve would guarantee the solver’s output, not a global optimum, because the objective is nonconvex.Source: §5 · p. 8

Relabeling invariance of the masked objective

• For a fixed covered sub-DAG and order-independent structural weights, the masked objective and its exact minimizer set are invariant to graph-preserving relabelings.Executable antichain members carry full mass simultaneously, so their order is not preferred.Source: §5 · p. 8

Serialization invariance

• Mid-stream serialization invariance is empirical, not guaranteed: age-weighted costs and warm-start couplings can differ, although finalization removes both sources of path dependence.Scenario B tracks and finalizes to the identical score.Source: §5 · pp. 8–9, Table 2 · p. 9

Streaming tolerance band

• A redundant leaf increases the optimal objective by at most λ/¯m, with aggregate perturbation O(1/t) relative to transported mass; its leakage is 1 exactly when its mass is destroyed.The increase is an upper bound, not an identity, because the optimizer may rearrange existing mass.Source: §5 · p. 9

Bounded influence of young structure

• A young node’s direct structural cost ramps from zero toward full magnitude over h_age events, unless an artifact is consumed earlier; this bound does not cover the full transport solution.Consumption lifts the discount along evidenced edges.Source: §5 · p. 9

6.2 SWE-bench setup

• The evaluation replays 2,294 SWE-agent trajectories, with 2,288 usable event streams, and scores methods on the same held-out traces at each checkpoint.The primary protocol uses a fixed held-out evaluation and bootstrap 95% confidence intervals; λ is calibrated on a separate set and frozen thereafter.Source: §6 · p. 9, §6.2 · p. 11

6.1 Controlled fault injection

• 100% loop recall (latency 0), 100% hallucination recall (latency 2), and 94.1% causal-inversion recall (latency 0), with 0 false interventions over 200 benign runs.Across the campaign, end-of-stream discrepancy never exceeded 0.013 and per-event runtime was 0.2–1.6 ms.Source: §6.1 · pp. 10–11

6.3 Early warning and cost savings

• At k ≤10, OnTrack’s AUROC advantage over cosine has a confidence interval excluding zero, but is gone at k=15 (cosine .633 vs. OnTrack .635).The clearest regime is k=8–10, after the trace-length signal fades and before content accumulates.Source: Table 5 · p. 11, §6.3 · p. 11
• At threshold 0.60, severe-flag-density aborts save 17.9% of unresolved-trace cost, with a 20.7% false-stop rate.An independent profile draw yielded 17.0% savings; the false-stop rate varied from 13% to 21%, and the threshold was illustrative rather than deployment-ready.Source: §6.3 · p. 12

6.4 Process anomaly detection

• On cost-limit traces, OnTrack reaches TPR 37% / 81% / 92% at FPR 1% / 5% / 10%, matching n steps; in-distribution hallucinations and inversions remain near the detection floor (0.572 and 0.496 AUROC).The severe-flag trigger fires on 90% of cost-limit traces at a median 58.6% elapsed, leaving about 41% of steps ahead; logged costs do not support converting this to dollars.Source: §6.4 · pp. 12–13

6.5 The λ finding: relaxation strength is the detector

• The calibrated λ is 0.1 on controlled tasks and 0.3 on SWE-bench, below the batch default of 1; it must be recalibrated when the embedding model changes.The paper attributes the detection difference to relaxation strength: at the batch default, high-cost hallucinated steps can retain substantial mass.Source: §6.5 · p. 13

6.6 Outcome correlation: a boundary study

• On a balanced sample, neither OnTrack nor cosine reliably predicts resolved versus unresolved outcomes (AUROC 0.581 and 0.643); length-matched signals approach chance.The paper attributes the modest full-trace correlation largely to unresolved traces being longer, while patch correctness depends on which line was edited.Source: §6.6 · p. 14
• Adding observation alignment to transport cost lowered mean ablation AUROC from 0.825 to 0.715 on a held-out tool-calling corpus, so the shipped default keeps that cost term off.Mass-weighted content divergence remains a monitoring signal separate from the transport cost.Source: §6.6 · p. 14

• Where a formal validity checker exists, OnTrack's proximity metrics are not a substitute; their intended scope is open-ended domains without verifiers.Source: §7 · p. 14
• OnTrack detects process anomalies correlated with failure, not success; length-controlled AUROC is 0.526 versus cosine at 0.540, while other trajectory signals are similarly near chance.Source: §7 · p. 14
• On homogeneous real traces, short in-distribution splices sit near the detection floor; detecting them requires task-level semantics such as matched references or a verifier.Source: §7 · p. 14
• Online dependency extraction remains the binding constraint despite age weights and retroactive repair, while formal streaming-CG tracking-error bounds remain open.Source: §7 · p. 14

• OnTrack brings structure-aware optimal transport into the agent loop: frontier-masked marginals handle prefixes and turn agent-side mass leakage into a real-time signal, with interventions based on per-node transport marginals.Source: §8 · p. 14
• At k=8, OnTrack beats cosine by +0.057 AUROC under the primary held-out protocol (95% CI [+0.033, +0.080]), with the advantage excluding zero through k=10.Source: §8 · p. 14
• The candidate flag-density abort policy saves 17–18% compute on failed runs at ≈83% deployment-adjusted precision; rival policies can match or exceed savings at matched false-stop rates.Source: §8 · p. 15
• At shipped thresholds, the quadratic structure term flags 13/40 benign traces versus 29–32/40 for reference-free variants, selecting a lower-false-alarm operating point rather than improving ranking.Source: §8 · p. 15
• On homogeneous SWE-agent traces, in-distribution splices sit near the detection floor; the paper claims process monitoring, not general hallucination detection or patch-correctness prediction.Source: §8 · p. 15

A Balanced multi-seed evaluation

• The secondary protocol uses five seeds, each redrawing a balanced ≈200-trace evaluation sample while holding the instance partition, reference set, and calibrated λ fixed.Source: A Balanced multi-seed evaluation · p. 15
• Balanced sampling raises absolute AUROC and deltas relative to the primary protocol, but the direction and early-k concentration of the advantage agree with Table 5.Source: Table 9 · p. 16