Source-linked AI summary
Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluation
Dan Liu, Jian Li
TL;DR
LLM-agent research lacks a principled way to determine whether trajectory-level resource consumption is justified by task contribution. This paper develops a five-stage cost–utility alignment framework with dual cost and contribution ledgers, then uses it to organize attribution, diagnosis, adaptation, and evaluation. Its central conclusion is that resource-aware deployment requires connecting observable cost records with causal contribution evidence, while several attribution, adaptation, evaluation, and accounting challenges remain.
Problem
Existing work addresses inference optimization, agent capabilities, evaluation, safety, or memory in isolation, leaving resource consumption and task contribution unconnected despite trajectory-level costs.
Method
The paper develops a trajectory-centric framework with five stages: cost profiling, utility attribution, misalignment diagnosis, targeted adaptation, and evaluation.
Results
The framework organizes recent methods and protocols across efficiency, reliability, economic value, five misalignment forms, and targeted adaptations into a closed analytical loop.
Takeaways & Limitations
Resource-aware agent design requires linking observable cost records to causal contribution evidence rather than evaluating efficiency without task-level utility attribution.
Takeaways & Limitations
Open environments prevent reliable counterfactual replay, while adaptation must account for attribution cost, infrastructure drift, human review overhead, and cost transfers beyond token budgets.
Abstract
from arXiv · showhide
LLM agents execute tasks through multi-step trajectories that accumulate cost in tokens, latency, monetary fees, and environmental risk while producing utility only at the aggregate task level. Prior surveys address inference optimization, agent capabilities, or evaluation in isolation, leaving practitioners without principled tools to determine whether a trajectory's resource expenditure is justified by its task contribution. We address this gap by developing a trajectory-centric cost-utility alignment framework that treats resource consumption and task contribution as dual ledgers over the same execution, organized around five analytical stages: cost profiling, utility attribution, misalignment diagnosis, targeted adaptation, and evaluation. Utility attribution is central to this structure: rather than relying on aggregate outcomes, it organizes contribution methods by evidential strength, from process proxies and information dependency to counterfactual replay, supplying the causal evidence that grounds diagnosis and guides adaptation. Using this framework, we analyze recent agent systems, attribution methods, and evaluation protocols covering efficiency, reliability, and economic value, as well as five forms of misalignment spanning cognitive and context use, external interaction, recovery-loop control, resource-capability allocation, and multi-agent coordination, together with their targeted adaptations. The result is a closed analytical loop connecting the cost side of agent execution to its utility side, providing a structured basis for resource-aware agent design and deployment.
1 Introduction
LLM agents incur trajectory-level costs while utility emerges from aggregate task outcomes, creating a need to connect resource consumption with task contribution. The paper introduces a five-stage alignment framework centered on utility attribution as the counterpart to cost profiling.
- LLM agent trajectories are the appropriate efficiency-analysis unit because costs accumulate across calls, tools, actions, observations, and recovery while utility emerges from the aggregate task outcome.
- Additional reasoning, retrieval, verification, and collaboration can improve task performance while increasing tokens, latency, fees, supervision, and external-side-effect risk.Repeated context, serial calls, and irreversible actions extend costs beyond token counts alone.
- Existing research treats inference optimization, agent capabilities, evaluation, safety, and memory largely in isolation, leaving resource consumption and task contribution unconnected.
- The proposed framework uses a typed agent trajectory and five stages: cost profiling, utility attribution, misalignment diagnosis, targeted adaptation, and evaluation.
- Utility attribution complements cost profiling by organizing contribution methods from process proxies and information dependency to counterfactual replay, then linking evidence to diagnosis, adaptation, and evaluation.
2 The Cost–Utility Alignment Framework
The framework models agent execution as a typed trajectory with parallel cost and utility ledgers. It uses event-level contribution evidence to diagnose misalignment and guide adaptation and evaluation.
- The framework represents an agent trajectory as typed hierarchical events linked by execution order and data flow, with each event carrying resource, monetary, and risk costs.
- Utility summarizes goal completion, partial progress, constraint satisfaction, and output quality at the run level, while local contribution compares the outcome with an event-removal counterfactual.
- Misalignment occurs when a feasible alternative achieves comparable utility at lower cost or higher utility at no greater cost within the budget constraint.
- Cost profiling localizes consumption across events and actors, whereas utility attribution identifies states, actions, and members causally responsible for task outcomes.
- The framework closes its two-ledger view through misalignment diagnosis, targeted adaptation, and evaluation over the same trajectory.
3 Cost Profiling
Cost profiling builds an auditable, multidimensional ledger that localizes resource use across execution structure, actions, phases, and workloads. The analysis shows why token counts alone miss important cost and risk patterns.
- Cost Profiling: Cost profiling identifies consumed resources, their execution locations, and distribution across tasks, runs, and actions, producing an auditable cost ledger.
- Multidimensional Cost: The cost vector independently records resource consumption, monetary cost, and risk across models, tools, infrastructure, and long-horizon repair cycles.
- Multidimensional Cost: Historical context, tool schemas and results, KV-cache occupancy, and serial calls make raw token counts understate token and latency costs.
- Hierarchical Cost Localization: Hierarchical localization assigns costs to tasks, runs, events, and executing subjects, preserving phase structure and invocation relationships for multi-agent attribution.
- Hierarchical Cost Localization: Phase-level and span-level profiling connects token, latency, command, model, tool, and sub-agent costs to workflow stages and parent-child execution relationships.
- Semantic Action Annotation: CostBench shows that the same objective can follow different execution sequences, so token costs must be matched to the realized path.
- Semantic Action Annotation: Read and write operations can have identical token counts but different risk costs because writes may require verification, rollback, correction, or incur irreversible harm.
- Workload Characterization: Approximately 59.5% of TraceLab’s estimated cost comes from cached prefixes, while repeated runs produce substantial token variation through sampling, exploration, and repair.
4 Utility Attribution
Utility attribution builds a contribution ledger for trajectory events, progressing from broad proxy signals through information dependencies to counterfactual interventions. This evidence chain supports increasingly causal assessment of how events affect task utility.
- The framework organizes utility attribution as an evidence chain from proxy signals through information dependency to counterfactual intervention.Figure 3 presents these three layers as progressively stronger forms of evidence.
- Proxy-Based Attribution: Proxy methods estimate step-level contribution from state changes, subgoal completion, checks, rollout returns, or other intermediate observations without requiring step-level human annotation.IPR, AgentPRM, InversePRM, iStar, and PABU operationalize proxy utility signals in different ways.
- Information Dependency and Evidence Contribution: Information-contribution analysis links output quality or constraint satisfaction to retrieved content, tool observations, program outputs, and memory through provenance, information flow, and decision dependency.TRACER, NeuroTaint, and ContextCite identify evidence-bearing or influential information units and tool turns.
- Information Dependency and Evidence Contribution: Information-dependency methods establish provenance and influence propagation but do not determine whether those dependencies change task outcomes.They stop short of outcome-level causal evidence.
- Counterfactual Intervention: Counterfactual attribution deletes, replaces, or neutralizes events, re-executes the trajectory, and measures utility change between factual and counterfactual conditions.CAR, ErrorProbe, CausalFlow, and C3 apply executable interventions across single-agent, repair, and cooperative multi-agent settings.
- Counterfactual Intervention: State recoverability limits counterfactual precision when hidden state, irreversible operations, or changing external services cannot be reconstructed from text history alone.Environment snapshots and random seeds may be required for matched replay.
5 Cost–Utility Misalignment Diagnosis and Targeted Adaptation
The framework diagnoses misalignment when costly trajectory behavior lacks sufficient utility support relative to a feasible alternative. It organizes targeted adaptations across context use, external interaction, recovery control, resource allocation, and coordination.
- Misalignment occurs when a trajectory incurs cost without sufficient utility support from cognitive use, interaction, recovery, capability allocation, or coordination.The framework compares costly behavior against feasible alternatives preserving or improving utility.
- Cognitive and Context Misalignment: Cognitive and context misalignment includes verbosity, unnecessary retention, and redundant retrieval that increase cost without changing task-relevant beliefs, actions, or state.Compaction, omission, retrieval optimization, and fixed-budget context maps target these mismatches, while naive omission can remove future dependencies.
- External Interaction and Exploration Misalignment: External-interaction adaptations gate calls, select tools by risk-adjusted cost, and prune unproductive search branches when no-call, better-tool, or earlier-stop alternatives preserve utility.Tool-use policies account for expected information gain, reliability, retries, branch survival, and budget visibility.
- Recovery-Loop and Verification-Control Misalignment: Recovery-control misalignment arises when retries, verification, repair, or rollback continue without restoring task state or increasing success.Fault-specific recovery, equal-budget resampling, structured feedback, rollback, and marginal-gain stopping address these mismatches.
- Resource–Capability Misalignment: Resource–capability adaptations route steps among models, reasoning intensities, samples, devices, and cloud resources according to quality gaps, uncertainty, and step importance.The surveyed methods reserve stronger or more expensive capability for pivotal, uncertain, or recovery states.
6 Cost–Utility Evaluation
Cost–utility evaluation measures complete agent runs under matched conditions while retaining multidimensional costs, utility, reliability, and economic value. The surveyed benchmarks show that simpler configurations can match elaborate systems, action and failure costs can dominate, and higher budgets do not always improve quality.
- Cost-Efficiency Metrics and Quality–Cost Trade-offs: Evaluation compares complete runs under matched constraints, preserving raw utility and multidimensional cost measures before aggregation.Execution cost and action cost remain separate, with efficiency ratios and budget curves exposing quality–cost trade-offs.
- Cost-Efficiency Metrics and Quality–Cost Trade-offs: Simpler model–scaffold pairings and controlled workflows can match more elaborate agents at lower cost, while planning and tool benefits remain task-conditional.TPS-Bench additionally reports higher completion with fewer tokens, rounds, and time.
- Cost-Efficiency Metrics and Quality–Cost Trade-offs: Action-path expense can substantially exceed the model bill, and quality may decline at high budgets when additional tools and evidence enlarge the search space.Terminal-Bench jointly logs outcome, dollars, tokens, calls, and runtime.
- Reliability and Risk-Adjusted Cost Evaluation: Reliability evaluation uses repeated runs, perturbations, and injected faults, with risk-adjusted cost incorporating failure severity, recovery expenditure, and catastrophic violations.Pass@k and Pass_k distinguish occasional from sustained success.
- Reliability and Risk-Adjusted Cost Evaluation: Neither larger models nor more complex architectures guarantee greater reliability under stress, and failed or timed-out trajectories can cost as much as successful ones.These findings motivate evaluation beyond standard success rates and nominal execution cost.
- Economic Value Evaluation: Economic evaluation combines skilled-human time, professional quality, review, rework, reliability, failure risk, and market payment to estimate deployment value.A cheaper per-run configuration can increase total workflow expenditure when rework rises.
- Evaluation Infrastructure: Credible evaluation requires observable environments, reproducible initialization, cost-complete logs, executable grading, and audited task and scoring consistency.These infrastructures support repeated runs, counterfactual replay, fault injection, and trustworthy comparisons.
7 Open Challenges
The framework identifies unresolved challenges in causal attribution, closed-loop adaptation, reproducible reliability evaluation, and human-centered economic assessment. These gaps concern replayability, attribution cost, infrastructure drift, rare failures, and unmeasured reviewer effort or transferred costs.
- Closed-Loop Adaptation under Attribution Cost: No current system closes the adaptation loop with attribution evidence while accounting for evidence cost and task-level reliability or risk constraints.Independent studies of compression, routing, pruning, and verification do not fully capture their interactions.
- Across the framework, current protocols leave four unresolved gaps involving causal replay, adaptation, reproducibility, and human-centered cost accountability.These gaps constrain the completeness of cost–utility alignment analysis.
- Scalable Causal Attribution in Open Environments: Counterfactual attribution remains difficult in open environments because irreversible writes and hidden state prevent faithful replay.Multi-agent interventions also alter shared information and may miss joint failures requiring combinations of steps.
- Reliability under Nonstationary Infrastructure: Reliability results may fail to reproduce as models, prices, cache rules, tools, and websites change, while rare-failure confidence intervals can require 10,000 or more episodes.Few leaderboards preserve the metadata, environment images, and replayable logs needed to separate infrastructure drift from agent behavior.
- Human Value, Cost Transfer, and Accountability: Existing evaluations omit human review effort, correction overhead, and failure loss, and do not jointly observe agent cost, reliability, and workflow effort under matched conditions.They also do not establish whether cost optimizations transfer expenses to reviewers.
8 Conclusion
The paper frames LLM agent efficiency as cost–utility alignment and organizes analysis through five connected stages. It identifies open challenges requiring links between observable cost records and causal contribution evidence.
- The framework treats agent efficiency as whether resource consumption is justified by contribution to task outcomes.
- It spans cost profiling, utility attribution, misalignment diagnosis, targeted adaptation, and evaluation, while analyzing representative methods across these stages.
- Four open challenges concern scalable attribution, closed-loop adaptation, reproducible evaluation under infrastructure drift, and accounting for human labor and cost transfers.
- The paper concludes that current efficiency research does not yet reliably connect observable cost records to causal contribution evidence.
A TraceCard Schema and Worked Example
The appendix defines a concrete schema for trajectory-level cost–utility profiling and demonstrates it with an end-to-end example covering all five framework stages.
- The appendix provides a concrete trajectory-level schema and applies it in an end-to-end example covering all five framework stages.
A.1 TraceCard Schema
The TraceCard schema records diagnostic information for each trajectory event, while TrajectoryRecord aggregates event data into a task-level representation.
- Each trajectory event has a TraceCard containing identity, cost profile, attribution evidence, misalignment diagnosis, and adaptation outcome.
- A TrajectoryRecord aggregates events, total cost, task-level utility, a budget constraint, and whether a lower-cost alternative was feasible.
A.2 Illustrative TraceCard Instances
The worked example contrasts a high-verbosity reasoning event with a downstream corrective write event. Attribution evidence determines whether diagnosis and adaptation are applied.
- Table 4 contrasts a high-verbosity reasoning step with a downstream corrective write step using schematic TraceCard values.
- The reasoning step is misaligned because counterfactual replay finds dep_flag = false and ΔU≈0 after removal.
- Agent-Omit removes the redundant reasoning step, while the dependency-linked corrective write is preserved.
- The corrective write is causally necessary, with dep_flag = true and ΔU≈1, so no misalignment or adaptation is applied.