PaperlayerView exact revision on arXiv

Source-linked AI summary

Lightweight, Rubric-Guided Trajectory Evaluation for Production AI Agents

Linh-An Phan, MingXue Wang, Guangyu Wu, Feng Pan, Zhaoyu Pang, Yanbin Zhang

arXiv:2610.03315v1cs.AI

TL;DR

Trajectory evaluation matters because final answers do not reveal all agent failures, while production traces are long and repeated diagnosis must manage cost and latency. LiteTrajEval combines offline rule profiling with online preprocessing, fixed-budget serialization, and one rubric-guided judge call. It improves failure-localization alignment over AgentRx while reducing evaluation cost and runtime.

  • Problem

    Production agents generate long traces, and repeated trajectory diagnosis must balance cost and latency with preserving enough execution context to localize failures.

    Source: §I · p. 1

  • Method

    LiteTrajEval derives domain-specific rules offline, preprocesses and serializes each trajectory under a fixed budget, then invokes one rubric-guided LLM judge.

    Source: §II · p. 2, C. Evidence-Aware Budget Serialization · p. 3

  • Results

    Against AgentRx, failure-localization alignment improves roughly 20–35 percentage points on Magentic-One and up to 23 points on τ-retail, while cost falls about 6× and evaluation time more than 8×.

    Source: §I · p. 2

  • Takeaways & Limitations

    The fixed-budget evaluation layer supports repeated agent debugging, regression testing, and offline failure analysis within a predictable evaluation budget.

    Source: §V · p. 6

    The current judge prompt makes user-side interaction-state failures harder to identify and represent, including incorrect user information or missing confirmation.

    Source: B. RQ1: Can Lightweight Judges Detect… · p. 5

Abstract

from arXiv · show

Trajectory evaluation is essential for improving the reliability of LLM-based agents, but production use makes it expensive to run repeatedly. Modern agents generate long traces containing tool calls, observations, retries, and external outputs, while not all raw tokens are equally useful for diagnosis. We present \textit{LiteTrajEval}, a lightweight architecture for budget-bounded trajectory evaluation. LiteTrajEval derives compact domain-specific rule profiles offline, then preprocesses each trajectory online, marks heuristic failure signals, serializes it under a fixed global budget, and invokes a single rubric-guided LLM judge to produce structured diagnostic reports. Evaluated on public Magentic-One-style and $\tau$-bench-style trajectory datasets, LiteTrajEval improves failure-localization alignment with human annotations by roughly 20--35 percentage points on Magentic-One and up to 23 percentage points on $\tau$-retail compared with AgentRx, while reducing cost by about 6$\times$ and evaluation time by more than 8$\times$. This solution has also been deployed in our enterprise agentic platform.

I. INTRODUCTION

Production agents need trajectory-level diagnosis because final answers can conceal unsafe or poorly grounded steps, while traces are long and repeated evaluation is cost- and latency-constrained. LiteTrajEval addresses this with a fixed-budget, single-call evaluator and reports stronger failure localization than AgentRx at lower evaluation cost and time.

I. INTRODUCTION

  • Final answers alone can miss unsafe tool use or policy violations, while incorrect outcomes may stem from earlier misreads, tool failures, missing evidence, or poor recovery.Trajectories are therefore central artifacts for understanding agent behavior and improving reliability.Source: §I · p. 1
  • Repeated production evaluation must control cost and latency without losing the causal chain needed to diagnose long traces.Raw traces can contain verbose outputs and redundant loops, while aggressive truncation can remove diagnostic context.Source: §I · p. 1
  • Multi-call methods assess steps or spans before aggregating a trajectory judgment, but their cost and latency complicate production regression workflows.Prior trajectory-diagnosis methods capture process-level failures that final-answer grading can miss, but do not address repeated-production demands.Source: §I · p. 1
  • LiteTrajEval separates reusable offline domain profiling from online preprocessing and fixed-budget serialization, then uses one rubric-guided LLM call for structured diagnostics.The report includes candidate failure steps, root causes, categories, key observations, and rubric scores.Source: §I · pp. 1–2
  • Against AgentRx, alignment rises roughly 20–35 percentage points on Magentic-One and up to 23 points on τ-retail; cost falls about 6× and evaluation time more than 8×.The paper also reports about 22× lower cost using DeepSeek-v4-flash than AgentRx with GPT-5-mini.Source: §I · p. 2

II. SYSTEM DESIGN

LiteTrajEval separates offline rule profiling from online trajectory evaluation to provide a lightweight evaluation layer for production agent workflows.

II. SYSTEM DESIGN

  • LiteTrajEval is a lightweight trajectory-evaluation layer intended for production agent workflows.Its offline path generates domain-specific profiles from accumulated trajectories; its online path applies them, serializes each trace under a fixed budget, and invokes one rubric-guided judge.Source: §II · p. 2
  • Domain-specific rule profiles are generated offline from accumulated trajectory outputs and reused during online evaluation.The offline process uses LLMs for semantic rule discovery without adding LLM calls to online preprocessing.Source: §II · p. 2
  • Online evaluation applies the rules, serializes each trajectory under a fixed budget, and uses one rubric-guided LLM judge.Source: §II · p. 2

A. Offline Rule Generation

LiteTrajEval builds compact domain-specific rule profiles offline from representative step outputs, then applies the resulting rules deterministically during online preprocessing.

A. Offline Rule Generation

  • Rule generation operates on step outputs rather than full trajectories, producing a compact domain-specific profile from accumulated trajectory data.The profile is applied deterministically online.Source: A. Offline Rule Generation · p. 2
  • Text canonicalization normalizes volatile tokens so recurring operational patterns become easier to detect.Examples include URLs, UUIDs, timestamps, paths, hexadecimal strings, request identifiers, and long numeric values.Source: A. Offline Rule Generation · p. 2
  • Near-duplicate outputs are grouped using Jaccard similarity over normalized token sets, with a threshold example of 0.7.A grouped output updates the representative’s support count rather than adding another representative.Source: A. Offline Rule Generation · p. 2
  • An offline LLM proposes or refines structured rules from token-budgeted representative groups; unmatched outputs default to ok, while matched rules assign warning or error.Rules include identifiers, priorities, scopes, patterns, optional exclusions, statuses, and candidate taxonomy labels, and can be reviewed or updated by developers.Source: A. Offline Rule Generation · p. 2

B. Trajectory Preprocessing

Preprocessing normalizes execution logs and adds weak failure-oriented guidance; evidence-aware serialization then compacts and allocates a fixed budget while retaining task, order, and diagnostic evidence.

B. Trajectory Preprocessing

  • Preprocessing converts heterogeneous logs into an ordered normalized trajectory containing the task, final output, and steps with IDs, types, statuses, inputs, and outputs.Statuses come from the centralized rule profile; unmatched steps default to ok.Source: B. Trajectory Preprocessing · p. 2
  • Warning and error steps anchor small candidate failure regions, expanded using nearby order, shared action or tool type, repeated inputs, or similar fingerprints.Matched rules also supply candidate taxonomy labels; the hints guide attention but are not final judgments.Source: B. Trajectory Preprocessing · p. 2
  • Preprocessing yields the normalized trajectory plus a compact guidance artifact for budget-aware serialization and single-call judging.The artifact contains candidate failure regions, matched rule identifiers, and candidate taxonomy labels.Source: B. Trajectory Preprocessing · p. 2

C. Evidence-Aware Budget Serialization

  • The pipeline overview separates offline rule-profile generation from online processing, fixed-budget serialization, and rubric-guided judging.Source: Figure 1 · p. 3
  • Serialization treats long trajectories as a fixed-global-budget optimization that preserves the task definition, step order, and failure-relevant evidence.The online serializer is deterministic and addresses verbose outputs and redundant orchestration messages.Source: C. Evidence-Aware Budget Serialization · pp. 2–3
  • Local tool-result compaction shortens structured or semi-structured outputs, while cross-step deduplication replaces repeated segments with references and retains the step skeleton.The skeleton preserves input, type, and status.Source: C. Evidence-Aware Budget Serialization · p. 3
  • Status tiers determine minimum preservation floors, and a waterfall procedure shrinks lower-priority tiers first before using hard truncation if needed.The procedure repeatedly reduces the longest outputs in each tier toward the next length level or their floor.Source: C. Evidence-Aware Budget Serialization · p. 3
  • Within each step’s assigned budget, MMR selects evidence units for relevance while penalizing redundancy against already selected units.The selection anchor is built from the task and step inputs.Source: C. Evidence-Aware Budget Serialization · p. 3

D. Rubric-Guided LLM-as-a-Judge

LiteTrajEval uses a single rubric-guided LLM judge to evaluate final-output and trajectory quality and produce a structured diagnostic report. Its serialization algorithm reduces trajectory content by tier to meet a global budget, with hard truncation if needed.

D. Rubric-Guided LLM-as-a-Judge

  • LiteTrajEval consolidates scoring, failure localization, taxonomy classification, and root-cause analysis into a single rubric-guided LLM call.The report's outputs condition on the same serialized trajectory.Source: D. Rubric-Guided LLM-as-a-Judge · pp. 3–4
  • The budget-allocation algorithm reduces content across tiers toward per-step minimums, then hard-truncates if the rendered trajectory still exceeds the global budget.It starts from normalized steps and a global budget, rendering the trajectory after reductions.Source: D. Rubric-Guided LLM-as-a-Judge · p. 4
  • The judge returns rubric scores, candidate failure steps, failure categories, root-cause analysis, and key observations in a structured JSON report.The rubric covers goal progress and completion, evidence grounding and verification, execution and tool handling, and recovery and adaptive control.Source: D. Rubric-Guided LLM-as-a-Judge · p. 4

III. EVALUATION

The evaluation asks whether LiteTrajEval detects failed trajectories, localizes failures in alignment with human annotations relative to a heavier diagnostic baseline, and keeps resource use predictable under a fixed trajectory budget.

III. EVALUATION

  • The evaluation tests whether LiteTrajEval detects that a failed trajectory contains a problem.This is the first of three practical evaluation questions.Source: §III · p. 4
  • The evaluation tests whether predicted failure regions align with human annotations and how that alignment compares with a heavier diagnostic baseline.This is the second practical evaluation question.Source: §III · p. 4
  • The evaluation tests whether token usage, runtime, and evaluation cost remain predictable under a fixed trajectory budget.This is the third practical evaluation question.Source: §III · p. 4

A. Datasets, Baseline, and Metrics

The evaluation uses failed-trajectory samples from Magentic-One-style and τ-retail settings, compares LiteTrajEval with AgentRx and a status-only variant, and reports detection, alignment, and resource-use measures.

A. Datasets, Baseline, and Metrics

  • The datasets contain 44 Magentic-One-style samples with 78 normalized failure groups and 29 τ-retail samples with 31 normalized failure groups.The τ-retail trajectories involve policy-constrained interactions among an agent, a user, and external tools.Source: A. Datasets, Baseline, and Metrics · p. 4
  • LiteTrajEval is compared with AgentRx in a cost-minimized configuration and with a No LLM, Status Only variant using rule-based step status signals.AgentRx requires multiple LLM calls per sample, while LiteTrajEval uses one judge call after preprocessing and budget-aware serialization.Source: A. Datasets, Baseline, and Metrics · p. 4
  • Detection Rate measures the fraction of failed samples with at least one candidate failure step; because all samples are failed trajectories, it measures failure recall.A candidate failure step is indicated by a non-empty failure_steps field for LiteTrajEval.Source: A. Datasets, Baseline, and Metrics · p. 4
  • Align.|Det.@k measures group-level agreement with human annotations among detected samples, after normalizing annotations into failure groups; a predicted step matches a group when its distance is at most k, reported at k = 1 and k = 3.Source: A. Datasets, Baseline, and Metrics · p. 4
  • The evaluation also reports token usage, estimated cost, and runtime; LiteTrajEval's measures cover its single judge call under a fixed 50,000-character trajectory budget.AgentRx measures aggregate all pipeline LLM calls.Source: A. Datasets, Baseline, and Metrics · p. 4

B. RQ1: Can Lightweight Judges Detect and Localize Failed Trajectories?

LiteTrajEval’s LLM judges improve failure-region alignment with human annotations over AgentRx once failures are detected, while rule-based status signals alone miss many failed trajectories. Detection is mixed across judge models, and some user-side interaction failures remain difficult to identify.

RQ1: Detection and Localization

  • The No LLM, Status Only variant detects 56.8% of Magentic-One and 31.0% of τ-retail failed trajectories, limiting diagnosis coverage.Its alignment on detected samples is not negligible, especially at @1, but low detection coverage is its primary limitation.Source: B. RQ1: Can Lightweight Judges Detect… · p. 4
  • LLM-based LiteTrajEval improves detection coverage across both datasets and group-level alignment over the rule-based variant.Detection varies by judge model; DeepSeek-v3.2 without reasoning mode reaches 72.4% on τ-retail versus AgentRx’s 86.2%.Source: B. RQ1: Can Lightweight Judges Detect… · p. 4
  • Alignment|Detected@3 exceeds AgentRx by roughly 20–35 percentage points across reported judge models on both datasets.Table II reports failure detection rate and group-level alignment on detected samples.Source: B. RQ1: Can Lightweight Judges Detect… · pp. 4–5, Table II · p. 5
  • On a τ-retail case, LiteTrajEval identifies the order-level cancellation error and downstream effects, while AgentRx flags an unrelated policy violation.The AgentRx localization mismatches the human annotation.Source: Figure 2 · p. 5
  • Some undetected τ-retail failures involve user-side behavior, such as incorrect information or missing confirmation, which the current judge prompt handles less readily.The prompt focuses mainly on tool execution, model reasoning, and agent reactions; the authors identify prompt refinement as a future direction.Source: B. RQ1: Can Lightweight Judges Detect… · p. 5

C. RQ2: Does LiteTrajEval Keep Evaluation Lightweight?

LiteTrajEval lowers evaluation cost and runtime relative to AgentRx. Its fixed-budget, single-call design also confines per-sample runtime to a narrower range.

RQ2: Evaluation Cost and Runtime

  • Table III reports raw trace lengths, per-sample evaluator token usage, total evaluation cost, and runtime across 44 Magentic-One and 29 τ-retail samples.AgentRx token usage aggregates all LLM calls per sample; LiteTrajEval’s per-sample usage corresponds to its single online judge call.Source: C. RQ2: Does LiteTrajEval Keep… · p. 6
  • Under GPT-5-mini, LiteTrajEval reduces total evaluation cost from $2.4628 to $0.4155, about 6×.Using DeepSeek-v4-flash lowers total cost to $0.1087, about 22.7× below AgentRx with GPT-5-mini.Source: C. RQ2: Does LiteTrajEval Keep… · p. 6
  • Total evaluation time falls from 15,990s to 1,840s (8.7×), with LiteTrajEval’s per-sample runtime at 12–63 seconds versus AgentRx’s 48–914 seconds.The authors attribute this stability to the fixed-budget, single-call design.Source: C. RQ2: Does LiteTrajEval Keep… · p. 6

IV. LESSONS FROM PRODUCTION DEPLOYMENT

LiteTrajEval is deployed in an enterprise platform, where it supports daily evaluation of an internal SRE agent within routine cost and latency budgets. Production observations highlight the usefulness of compacting large tool outputs, while human labeling and judge calibration remain ongoing work.

IV. LESSONS FROM PRODUCTION DEPLOYMENT

  • The evaluation harness processes hundreds of daily SRE-agent trajectories and can reevaluate them after prompt or configuration changes within routine cost and latency budgets.Compacting the largest-output steps accounts for most of the reduction in serialized trace length; this compaction is structural rather than semantic.Source: §IV · p. 6
  • Small self-hosted open-weight judges were sufficient for production cost and latency budgets, with plausible failure steps largely stable across models in preliminary manual inspection.This stability is a qualitative observation because production traffic lacks step-level ground-truth labels.Source: §IV · p. 6
  • A lightweight human-in-the-loop labeling process and calibration assessment across judge models remain ongoing work.Source: §IV · p. 6
  • The judge is loosely coupled with preprocessing and serialization; ongoing skill-autooptimization and agent-memory experiments reuse the pipeline with different rubrics and output schemas.Selective LLM summarization of outlier steps is proposed as a natural extension that would not add a per-step LLM call.Source: §IV · p. 6

V. CONCLUSION

LiteTrajEval combines offline rule-profile generation, deterministic preprocessing, fixed-budget serialization, and one rubric-guided judge call for production-agent trajectory evaluation. On public trajectory datasets, it improves localization alignment with human annotations while reducing evaluation cost and runtime relative to a multi-call diagnosis baseline.

V. CONCLUSION

  • LiteTrajEval combines offline rule-profile generation, deterministic trajectory preprocessing, fixed-budget serialization, and a single rubric-guided LLM judge call.Source: §V · p. 6
  • On public Magentic-One-style and τ-retail trajectories, LiteTrajEval improves failure-localization alignment with human annotations while reducing evaluation cost and runtime versus a strong multi-call diagnosis baseline.Source: §V · p. 6
  • By preserving the global execution skeleton and removing redundant low-value content, LiteTrajEval supports repeated debugging, regression testing, and offline failure analysis within a predictable evaluation budget.Source: §V · p. 6

Related papers

Loading 2610.03315v1…