Source-linked AI summary
TrajAD: Trajectory Anomaly Detection for Trustworthy LLM Agents
Yibing Liu, Chong Zhang, Zhongyi Han, Hansong Liu, Yong Wang, Yang Yu, Xiaoyan Wang, Yilong Yin
TL;DR
LLM agents need runtime auditing because intermediate trajectory anomalies can create safety risks even when tasks do not immediately fail. The paper formalizes trajectory anomaly detection, builds TrajBench, and trains TrajAD with fine-grained process supervision; experiments show general-purpose models struggle with detection and localization, while TrajAD outperforms larger baselines. The authors conclude that specialized supervision is necessary for reliable process auditing.
Problem
Existing approaches do not provide datasets and runtime mechanisms for distinguishing anomalous trajectories and precisely localizing their error steps.
Method
The paper constructs TrajBench from normal and perturb-and-complete anomalous trajectories, then fine-tunes Qwen3-4B into TrajAD for trajectory verification.
Results
General-purpose LLMs struggle with anomaly detection and exact localization, while TrajAD achieves superior detection and localization performance and outperforms larger baselines.
Takeaways & Limitations
The findings support shifting agent evaluation from outcome-based correctness toward process auditing with specialized supervision.
Takeaways & Limitations
Existing datasets rarely include annotated negative trajectories, limiting supervision for learning anomaly decision boundaries.
Abstract
from arXiv · showhide
We address the problem of runtime trajectory anomaly detection, a critical capability for enabling trustworthy LLM agents. Current safety measures predominantly focus on static input/output filtering. However, we argue that ensuring LLM agents reliability requires auditing the intermediate execution process. In this work, we formulate the task of Trajectory Anomaly Detection. The goal is not merely detection, but precise error localization. This capability is essential for enabling efficient rollback-and-retry. To achieve this, we construct TrajBench, a dataset synthesized via a perturb-and-complete strategy to cover diverse procedural anomalies. Using this benchmark, we investigate the capability of models in process supervision. We observe that general-purpose LLMs, even with zero-shot prompting, struggle to identify and localize these anomalies. This reveals that generalized capabilities do not automatically translate to process reliability. To address this, we propose TrajAD, a specialized verifier trained with fine-grained process supervision. Our approach outperforms baselines, demonstrating that specialized supervision is essential for building trustworthy agents.
1 Introduction
The paper frames runtime trajectory anomaly detection as necessary for reliable LLM agents because intermediate errors can create safety risks without immediate task failure. It introduces TrajAD and TrajBench to detect and localize anomalies, enabling rollback-based recovery.
- Motivation: LLM agents’ complex interleaving of reasoning, tool use, and feedback creates intermediate anomalies such as invalid parameters, loops, and redundant actions.These errors may remain locally plausible and avoid immediate task failure while wasting resources or creating safety risks.
- Motivation: Existing capability-training and static-safety methods do not serve as runtime monitors for specific execution instances.The paper distinguishes parameter optimization and static checks from auditing an individual trajectory during execution.
- Problem formulation: The paper identifies trajectory anomaly detection as a process-centric task requiring both anomaly discrimination and exact error-step localization.Localization supports recovery by allowing rollback to the error step rather than restarting the entire task.
- Dataset and framework: TrajBench combines normal trajectories with perturb-and-complete negative samples to cover procedural anomalies across agent tasks.The benchmark includes Task Failure, Process Inefficiency, and Unwarranted Continuation across reasoning, mathematics, coding, web navigation, and embodied AI.
- Dataset and framework: TrajAD is a specialized auditing framework that models global execution context to detect anomalies and localize errors at the step level.The framework is designed to support a rollback-and-retry mechanism instead of restarting the whole task.
2 Related Work
Prior agent research emphasizes architectural enhancements, parameter updates, and static trustworthiness checks, but these approaches do not adequately monitor dynamic execution processes. TrajAD instead uses synthesized anomaly trajectories to train a specialized verifier that identifies logical dependencies and localizes errors.
- Existing paradigms: Agent research has pursued architectural reasoning frameworks and parameter-update methods such as instruction tuning, distillation, process reward models, and trajectory fine-tuning.These paradigms respectively modify agent organization or embed capabilities through model training.
- Reliability gap: Capability-oriented advances can produce blind goal-directedness, where agents optimize final outcomes while neglecting process rationality.The paper associates this pattern with uncontrollable execution, computationally wasteful loops, and risky unverified actions.
- Trustworthiness gap: Trustworthiness methods including hallucination detection, safety guardrails, and LLM-as-a-Judge primarily perform static checks, passive defense, or zero-shot evaluation.These approaches are described as insufficient for monitoring the dynamic execution process.
- Specialized verification: TrajAD synthesizes anomaly trajectories for fine-tuning a specialized verifier that learns logical dependencies between steps.This training enables anomaly identification and exact-step localization within execution processes.
3 Problem Formulation
The paper models agent execution as a sequential process and defines trajectory anomaly detection as jointly judging validity and locating the first error step. It distinguishes task failure, process inefficiency, and unwarranted continuation as process-rationality anomalies.
- 3.1 Preliminaries: Agent execution is represented as n cycles of reasoning, action, and environmental observation under a task instruction.
- 3.1 Preliminaries: The trajectory representation explicitly captures the interleaving of reasoning, execution, and feedback.
- 3.2 Taxonomy of Anomalies: The framework evaluates process rationality rather than only final outcomes, using three primary anomaly categories.
- 3.2 Taxonomy of Anomalies: Task Failure occurs when flawed reasoning or an incorrect action prevents task completion, including runtime exceptions.
- 3.2 Taxonomy of Anomalies: Process Inefficiency occurs when redundant steps preserve the outcome despite a shorter trajectory being available, including loops and unnecessary actions.
- 3.2 Taxonomy of Anomalies: Unwarranted Continuation occurs when an agent continues after a task is impossible, unnecessary, or already finished.
- 3.3 Task Definition: Trajectory Anomaly Detection learns a mapping from a trajectory to a binary validity verdict and the location of its first error.
- 3.3 Task Definition: Accurate first-error localization enables rollback to the pre-error state instead of restarting the entire task.
4 TrajBench: A Dataset for Trajectory Anomaly Detection
TrajBench is a semi-automatically constructed benchmark pairing verified golden trajectories with controlled anomalies and precise labels. It provides balanced, diverse supervision for anomaly classification and error-step localization, with human review supporting label reliability.
- TrajBench pairs golden trajectories with strictly defined anomalies, supplying supervision for anomaly verdicts and error localization.
- AgentBank seeds are validator-filtered to retain trajectories with logically sound reasoning chains before anomaly synthesis.
- Perturb-and-Complete modifies a target step and conditionally completes the altered trajectory to generate controlled negative samples.
- Controlled perturbations receive automatic labels by assigning the perturbed step as the error location and the anomaly verdict, while the seed remains normal.
- 60,000+ trajectories are balanced 1:1 between normal and anomalous samples across 13 tasks and five domains, with roughly one-third in each anomaly type.
- Seed verification retained 34,436 of 37,625 raw trajectories, a 91.6% pass rate, and synthesis produced 31,742 valid anomalous trajectories.
- Human review found 96.2% agreement for anomaly classification and 94.5% agreement for error localization on 500 samples.
5 TrajAD: A Generative Verifier for Agent Trajectories
TrajAD is a generative verifier that converts an instruction and execution trajectory into a structured anomaly verdict and error location. During runtime, its Check-and-Act protocol continues normal executions and interrupts anomalous ones for targeted rollback.
- TrajAD formulates auditing as conditional text generation from a system instruction and trajectory to a structured diagnostic report.
- The diagnostic report contains a Normal-or-Anomaly verdict and an index identifying the error step.
- The verifier uses a decoder-only Transformer with LoRA, freezing pretrained weights while training low-rank matrices A and B.
- Figure 2 depicts TrajBench construction through filtered seeds, Perturb-and-Complete generation, three anomaly types, balanced labels, and coverage of 13 tasks across five domains.
- Its training objective learns the joint distribution of anomaly verdicts and error locations from TrajBench supervision.
- At fixed intervals, the runtime monitor continues when the verdict is Normal and interrupts execution to roll back to the predicted pre-error state when it detects an anomaly.
6 Experiments
Experiments evaluate TrajAD against zero-shot baselines for anomaly detection and step-level localization across in-distribution domains, cross-domain transfer, and scaling conditions. TrajAD improves detection and localization, transfers to an unseen domain, and achieves its best efficiency with a 4B model trained on 50k stratified samples.
- Experimental Setup: The evaluation compares fine-tuned TrajAD with zero-shot baselines using Precision, Recall, Macro-F1, and Joint Exact Match for detection and localization.The benchmark uses a balanced 60k-sample TrajBench dataset with stratified testing.
- Main Results: Zero-shot models show severe localization weakness, with JEM scores consistently below 10%, while Qwen3-4B has 79.07% Precision but 68.97% Recall.Phi-3 reaches only 28.46% Recall, indicating conservative anomaly detection.
- Main Results: TrajAD improves Macro-F1 by 11.38% to 81.81% and JEM by 48.21% to 53.75% over the strongest baseline.The generative objective couples logical reasoning with structural verification for error diagnosis.
- Domain-Specific Analysis: TrajAD consistently outperforms baselines across Math, Reasoning, Coding, Web Navigation, and Embodied AI domains.Zero-shot baselines show near-zero localization in Embodied AI, whereas TrajAD maintains high precision there.
- Out-of-Distribution Generalization: On an unseen domain, TrajAD-TM raises Macro-F1 from 70.89% to 83.09% and JEM from 11.48% to 38.25% over the zero-shot baseline.Detection nearly matches full supervision at 83.09% versus 83.84% F1, while localization remains lower at 38.25% versus 52.54% JEM.
- Scaling and Efficiency Analysis: Training on 50k samples peaks at 85.31% F1 and 61.02% JEM, while increasing model size alone provides limited gains.The fine-tuned 8B model reaches 78.97% F1, below the 4B model’s 81.81% on the same partition and 85.31% optimum.
7 Conclusion
The paper defines trajectory anomaly detection, introduces TrajBench, and shows that general-purpose LLMs struggle to localize execution errors regardless of scale. TrajAD demonstrates that specialized supervision enables a smaller model to audit trajectories effectively.
- Conclusion: The paper formalizes Trajectory Anomaly Detection and introduces TrajBench as a large-scale benchmark for process auditing.The conclusion shifts evaluation from outcome-based metrics toward rigorous process auditing.
- Conclusion: General-purpose LLMs fail to link detected anomalies to specific execution steps, and scaling model size does not solve this issue.The conclusion identifies specialized supervision as necessary for reliable localization.
- Conclusion: TrajAD outperforms larger baselines, showing that a small model can be effective when trained to generate explicit error details.The stated conclusion connects effectiveness to explicit error-detail generation.