PaperlayerView exact revision on arXiv

Source-linked AI summary

Reducing Cost of LLM Agents with Trajectory Reduction

Yuan-An Xiao, Pengfei Gao, Chao Peng, Yingfei Xiong

arXiv:2509.23586v2cs.SEcs.AI

TL;DR

Coding agents accumulate input-token costs in their growing trajectories, and this efficiency concern has been overlooked. The paper proposes AgentDiet, an inference-time approach for reducing trajectory waste. Across two LLMs and two benchmarks, it reports input-token reductions of 39.9%–59.7%.

  • Problem

    High input-token costs from growing agent trajectories are an efficiency concern that existing studies have largely overlooked.

    Source: §1 · p. 2

  • Method

    AgentDiet uses inference-time trajectory reduction to remove waste during agent execution.

    Source: §1 · p. 2

  • Results

    39.9%–59.7% fewer input tokens were reported across two LLMs and two benchmarks.

    Source: §8 · p. 20

  • Takeaways & Limitations

    The paper concludes that inference-time trajectory reduction is a promising direction for coding agents and can reduce costs without harming performance.

    Source: §1 · p. 2

    Benchmark success is based on passing test cases, so a plausible patch may not be semantically equivalent to the developer’s patch; held-out tests mitigate this threat.

    Source: §6.2.3 · p. 18

Abstract

from arXiv · show

Multi-turn agent systems based on Large Language Models (LLMs) have become increasingly popular for software engineering tasks. While LLM agents demonstrate promising effectiveness, the high computational cost of input tokens due to ever-growing trajectories remains a significant efficiency concern. Efficiency has been largely overlooked in existing studies and agent products, and this paper addresses this gap by introducing an inference-time trajectory reduction approach that reduces computational costs. By analyzing existing agent trajectories, we demonstrate that useless, redundant, and expired information is widespread across trajectories. Such waste can be identified and reduced without compromising the agent's performance. We propose a simple yet effective trajectory reduction approach, AgentDiet, which automatically removes such waste during agent execution. We implement AgentDiet on a top-performing coding agent, and our evaluation on two LLMs and two benchmarks shows that AgentDiet can reduce input tokens by 39.9%-59.7% and the total computational cost by 21.1%-35.9%, while maintaining the same agent performance. These results indicate that inference-time trajectory reduction is a promising direction for agent systems.

1 Introduction

The introduction frames trajectory-token cost as an overlooked efficiency concern for coding agents. It proposes AgentDiet and reports that trajectory reduction can lower costs without harming agent performance.

1 Introduction

  • 53% of surveyed participants considered AI-agent cost a barrier, while efficiency had received limited attention in existing studies.The paper identifies the gap as a reason to investigate trajectory reduction for coding agents.Source: §1 · p. 2
  • Coding-agent trajectory reduction differs from single-turn token reduction in its timing, structured-code inputs, and need to avoid changes to proprietary model inference.The introduction says the potential efficiency gains from trajectory reduction remained unknown.Source: §1 · p. 2
  • AgentDiet is proposed as an inference-time approach to remove trajectory waste; evaluation reports cost reduction without harming agent performance.The paper identifies useless, redundant, and expired information as sources of waste and describes frequent reduction during agent execution.Source: §1 · p. 2

2 Problem Analysis

The problem analysis asks whether agent trajectories contain reducible waste, whether an LLM can reduce it, and how to control reduction timing and overhead.

2 Problem Analysis

  • The analysis finds substantial waste in agent trajectories, including useless, redundant, and expired information.This finding motivates the prototype reduction approach.Source: §2 · p. 3
  • An LLM can be used to identify and reduce trajectory waste, according to the section’s summary of its investigation.The paper describes this as a prototype approach.Source: §2 · p. 3
  • Because LLMs do not determine when to reduce trajectories on their own, the approach enforces reduction through a separate reflection module.Source: §2 · p. 3
  • Calling an LLM for reduction is not inherently efficient, but the section reports that simple optimizations keep the overhead under control.Source: §2 · p. 3

2.1 Preliminaries

An LLM agent repeatedly adds assistant and tool messages to a trajectory that supplies context for later predictions. This growing history raises computational and memory costs, though removing irrelevant content may preserve or improve performance.

2.1 Preliminaries

  • Agent trajectories concatenate assistant tool calls and tool outputs, retaining them until task completion for subsequent LLM predictions.The loop ends when the LLM finishes or the agent reaches its step limit.Source: §2.1 · pp. 3–4
  • 40 steps in an average SWE-bench Verified issue trajectory contain 48.4K tokens, and repeated inclusion raises accumulated input usage to 1.0M tokens per issue.Tool messages account for 30.4K tokens, assistant messages for 13.7K, and system/user messages for 4.4K.Source: §2.1 · p. 4
  • KV Cache reduces repeated computation but leaves remaining costs and consumes VRAM and I/O bandwidth; changing earlier tokens also invalidates later cache entries.Source: §2.1 · p. 4
  • Removing trajectory content need not lower agent performance because long contexts can degrade LLM performance, and waste can be removed with minimal information loss.The paper says this has the potential to maintain or slightly improve performance.Source: §2.1 · p. 4

2.2 Waste in Trajectories

A pilot inspection of 100 Trae Agent trajectories from SWE-bench Verified found widespread waste, grouped into useless, redundant, and expired information.

2.2 Waste in Trajectories

  • 100 trajectories from Trae Agent on SWE-bench Verified were manually inspected, and waste appeared in almost all of them.The study used qualitative inspection to identify three typical waste scenarios.Source: §2.2 · p. 4

2.2.1 Useless Information.

  • Repository listings and verbose build or test output can contain useless information even when output truncation limits logs to 16KB.Examples include cache and resource files and GNU make directory messages.Source: §2.2.1 · p. 4

2.2.2 Redundant Information.

  • File-editing calls can repeat content: replacement text appears in both arguments and results, can overlap the old text, and may repeat code already retrieved.Source: §2.2.2 · p. 5

2.2.3 Expired Information.

  • After an agent identifies the faulty file during a search, most content from other opened files may no longer be necessary.The paper presents this as information that expires after a local diagnostic step.Source: §2.2.3 · p. 5

2.3 A Prototype Approach to Trajectory Reduction

The prototype uses an LLM to identify and compress useless, redundant, or expired trajectory information, then integrates reduction through a separate reflection module. A sliding window limits the module’s context and constrains which step it can change.

2.3.1 Reducing the Waste.

  • AgentDiet prompts an LLM to remove useless, redundant, and expired information while preserving important structure and replacing removed text with a short takeaway.The prompt’s examples come from identified trajectory waste, and its safeguards were refined against cases of missed waste or excessive deletion.Source: §2.3.1 · p. 5
  • In the pytest case, the verbose passed-test list accounted for 1,714 of 1,995 tokens; replacing it with a placeholder preserved the failing test in a 259-token step.The figure illustrates this reduction in a serialized trajectory step.Source: §2.3.1 · p. 6, Figure 2 · p. 6

2.3.2 Agent Integration.

  • An agent with an erase tool often continued its original task instead of reducing its trajectory, even when explicitly prompted.The authors suggest that autonomous reduction may require careful fine-tuning, which they describe as resource-intensive, error-prone, and often impossible for proprietary LLMs.Source: §2.3.2 · pp. 6–7
  • A separate reflection module is called by the outer system when reduction is estimated to help, leaving the agent unaware of the change.It uses a cost-efficient LLM and a sliding window: at step s, it edits step s−a using context from s−a−b through s, and skips reflection when the target step is below threshold θ.Source: §2.3.2 · p. 7, §2.3.3 · pp. 7–8

2.3.3 Controlling the Overhead.

  • The sliding window caps each reduction step’s input at a+1+b steps and preserves earlier KV Cache entries by modifying only step s−a.It also prevents the module from erasing the most recent step or all steps at once.Source: §2.3.3 · p. 8

3 Algorithm

Algorithm 1 adds AgentDiet’s reflection step to a typical tool-using LLM agent. It reduces a selected trajectory step using bounded context and applies the replacement only when the token savings exceed a threshold.

3 Algorithm

  • AgentDiet’s design combines prompting a cost-efficient LLM about three types of waste, explicit reflection invocation, and a sliding window to balance overhead and reduction effectiveness.The reflection model may be more cost-efficient than the model used for agent steps.Source: §3 · pp. 8–9
  • The algorithm takes a task instruction and environment, runs agent steps and tool calls, and returns a completion result while modifying the environment in place.Its standard agent loop initializes the trajectory, appends each assistant/tool exchange, and stops when the task is done or the step limit is reached.Source: §3 · pp. 8–9
  • The reflection module uses LLMreflect to reduce step s−a from a bounded context window and replaces it only when the token-length reduction exceeds threshold θ.Reflection is skipped when the target step is no longer than θ; a and b determine the context window.Source: §3 · p. 9

4 Implementation

AgentDiet is integrated by adding a reflection step to an agent loop, and its hyperparameters are selected through iterative comparison of efficiency and performance. The reported results select GPT-5 mini as the reflection model and identify settings that balance efficiency with agent performance.

4.1 Integration in Agents.

  • AgentDiet can be added to a tool-using agent by calling its reflection module after each step in the agent loop.It was integrated into Trae Agent; the authors did not further integrate or experimentally evaluate it in other agents.Source: §4.1 · p. 10

4.2.1 The Process.

  • Hyperparameter tuning used hill climbing: vary one setting at a time, retain variants with better overall results, and repeat until reaching a local optimum.Efficiency and performance were compared using variants that changed one hyperparameter while keeping the others at their defaults.Source: §4.2.1 · p. 10

4.2.2 Benchmark.

  • The tuning benchmark was a random subset of 100 SWE-bench Verified instances, reserved separately from the later evaluation instances.The same subset was used for the manual analysis in Section 2.2.Source: §4.2.2 · p. 10

4.2.3 The Search Space.

  • The search tested five reflection LLMs, four sanity-check baselines, and variants of θ, a, and b.The tested a and b ranges were capped at 3 and 2 because higher values would increase overhead and could improve performance; the authors expected little benefit beyond a=2 and b=1.Source: §4.2.3 · pp. 10–11, Table 1 · p. 11

4.2.4 Metrics.

  • Efficiency was measured with Keep%, input and output tokens, and costs for agent and reflection steps; performance used Pass%, Step, and PStep.Step and PStep are average total steps across all instances and successful instances, respectively; lower values are better.Source: §4.2.4 · pp. 11–12

4.2.5 Results.

  • The reflection-model variants keep 14.4%–29.0% of processed-step tokens, with accumulated input tokens at 47.3%–72.2% of Original and reflection overhead of 5.2%–14.8%.Pass% is at least 58% across the variants and baselines, versus 65% for Original; Delete and Random increase Step and PStep by more than 13%.Source: §4.2.5 · p. 12, Table 2 · p. 12
  • GPT-5 mini maintains the same Pass% as Original and is the only model with decreases in Step and PStep; therefore, it was chosen as AgentDiet’s final LLMreflect.The a=2, b=1 setting is the minimum choice reported to have negligible performance impact while improving efficiency by more than 22%.Source: §4.2.5 · pp. 12–13
  • At θ=500, AgentDiet balances input-token savings, reflection overhead, and performance close to Original.Higher θ increases agent input tokens but reduces reflection overhead; a=2 and b=1 delay reduction and give the reflection module more context, improving performance at reduced efficiency.Source: §4.2.5 · p. 13, Table 3 · p. 13

5 Evaluation

The evaluation tests AgentDiet in Trae Agent with two LLMs, using two benchmarks and metrics for both efficiency and task-solving performance. AgentDiet’s hyperparameters follow the settings selected in the preceding experiments.

5 Evaluation

  • The evaluation asks whether AgentDiet improves trajectory efficiency without harming task performance, and whether results generalize across benchmarks and LLMs.These are framed as research questions on efficiency improvement, performance impact, and generalization.Source: §5 · p. 13

5.1.1 Experiment Subjects.

  • The study evaluates AgentDiet integrated into Trae Agent with Claude 4 Sonnet and Gemini 2.5 Pro, comparing it with the unmodified Original baseline.The reflection step is skipped in the Original baseline.Source: §5.1.1 · pp. 13–14

5.1.2 Metrics.

  • Efficiency is assessed with Keep%, I, O, $, and $+; performance is assessed with Pass%, Step, and PStep.The stated criteria are lower Keep%, I/O and costs for improved efficiency, and no decrease in Pass% or increase in Step/PStep for no performance harm.Source: §5.1.2 · p. 14

5.1.3 Benchmarks.

  • The evaluation uses 200 randomly selected SWE-bench Verified instances and all 300 Multi-SWE-bench Flash instances.SWE-bench Verified tasks are Python issues; Multi-SWE-bench Flash covers seven other programming languages and tasks are generally harder, often requiring agents to determine how to build the project.Source: §5.1.3 · p. 14

5.1.4 Hyperparameters.

  • AgentDiet uses a=2, b=1, θ=500, and GPT-5 mini as LLMreflect; the step limit is 50 for SWE-bench Verified and 100 for Multi-SWE-bench Flash.The higher Multi-SWE-bench Flash limit reflects that the benchmark requires more steps to complete.Source: §5.1.4 · p. 14

5.2 Results

The results evaluate AgentDiet’s efficiency, performance impact, and generalization. AgentDiet reduces input tokens and final computational cost while maintaining comparable resolution performance across the evaluated benchmarks and LLMs.

5.2.1 RQ1. Efficiency Improvement.

  • AgentDiet’s stepwise token-reduction histogram uses a logarithmic x-axis; gray marks unreduced steps and red/blue squares show original and reduced step sizes.Reduction is skipped for steps below θ=500 tokens or when a proposed reduction is not applied, limiting reflection overhead and KV-cache invalidation.Source: §5.2.1 · p. 15, Figure 4 · p. 15
  • 39.9%–59.7% fewer input tokens and 21.1%–35.9% lower final computational cost than Original, after accounting for AgentDiet’s overhead.The reflection module removes 69.2%–77.4% of processed content, but only processes steps above θ=500 tokens and applies reductions after a=2 steps.Source: §5.2.1 · p. 15

5.2.2 RQ2. Performance Impact.

  • On Multi-SWE-bench Flash with Gemini 2.5 Pro, AgentDiet reduced average steps from 57.20 to 43.90 and instances reaching the 100-step limit from 66 to 26.The authors attribute this case to reducing the long context associated with repeated invalid tool calls.Source: §5.2.2 · p. 16, Figure 5 · p. 16
  • AgentDiet’s resolution rate is comparable to Original (-1.0%–+2.0%) and it does not increase the steps required to resolve tasks.The authors report this pattern across two benchmarks and two LLMs.Source: §5.2.2 · p. 16

5.2.3 RQ3. Generalization.

  • AgentDiet generalizes across SWE-bench Verified and Multi-SWE-bench Flash, Claude 4 Sonnet and Gemini 2.5 Pro, and seven programming languages.The implementation’s prompts and hyperparameters were based on a few test cases and a 100-instance dataset.Source: §5.2.3 · pp. 16–17, Table 5 · p. 17

6 Discussion

The discussion examines AgentDiet’s latency and overhead trade-offs, then considers the scope and validity of its evaluation. It identifies constraints involving the agent setting, proprietary models, and test-based patch correctness.

6.1.1 Improving the Latency of Agents.

  • AgentDiet adds a reflection step and latency; running reflection and agent steps in parallel could remove that increase but reduces the reflection module’s context by one step.The authors do not quantitatively compare latency because commercial API latency is unstable.Source: §6.1.1 · p. 17

6.1.2 Exploring Different Designs for Trajectory Reduction.

  • The current design uses off-the-shelf reflection models, adding 5%–10% cost overhead; customized models are proposed as a way to reduce it.For agents routing tasks of varying difficulty to different LLMs, the authors also suggest automatically adjusting AgentDiet’s hyperparameters to balance efficiency and performance.Source: §6.1.2 · p. 18

6.2.1 Generalization Across Agents.

  • Evaluation uses two benchmarks and two LLMs but implements AgentDiet only for Trae Agent, constraining direct evidence about generalization across agents.The authors cite effort and cost constraints, including approximately US$ 2000 in existing LLM costs, and note that agent systems generally use similar prompts and tools.Source: §6.2.1 · p. 18

6.2.2 Data Leakage in LLMs.

  • Because the experiments use proprietary LLMs, the authors cannot control whether benchmark instances appeared in their training data; they judge this threat not to significantly affect validity.They included Multi-SWE-bench Flash, a third-party benchmark not mentioned in the evaluated LLMs’ technical reports.Source: §6.2.2 · p. 18

6.2.3 Correctness of Patches.

  • Passing all developer-written tests makes a patch plausible, but does not establish semantic equivalence to the developer’s ground-truth patch.Both benchmarks include at least one held-out test case, which the authors say mitigates overfitting; they also compare AgentDiet and the Original agent under the same overfitting threat.Source: §6.2.3 · p. 18

7 Related Work

Prior prompt-reduction methods target initial inputs rather than the iterative timing of agent trajectories, while existing context-management mechanisms are often ad hoc. AgentDiet instead targets aggressive trajectory reduction and does not require white-box access to the LLM.

7.1 Prompt Reduction

  • Prompt-reduction methods target initial inputs, not iterative agent trajectories; code-oriented compression may also fail to preserve code and command-output structure.Trajectory reduction timing can affect agent efficiency and performance, so these methods cannot be directly applied without considering when to reduce prompts.Source: §7.1 · p. 19
  • Lindenbauer et al. [18] delay deleting tool messages by 10 steps; the paper reports the Delete baseline has 7% lower Pass% and 14% higher Step and PStep than all LLM variants.The paper also notes that a=10 would cause significant KV Cache invalidation overhead.Source: §7.1 · p. 19

7.2 Context Management in Agent Systems

  • Cursor, Claude Code, Trae Agent [11], and SWE-agent [43] use context-management mechanisms that are often ad hoc and applied sparingly, whereas AgentDiet targets aggressive trajectory reduction for overall efficiency.The paper says existing work does not report or evaluate context management as a first-class research question.Source: §7.2 · p. 19

7.3 Efficiency Improvement for White-Box LLM Systems

  • Methods that modify LLM training or inference require white-box access, while AgentDiet does not depend on the model’s internal mechanisms and can apply to a wider range of agent systems.Source: §7.3 · p. 20

8 Conclusion

The paper studies trajectory reduction for coding agents and evaluates AgentDiet across two benchmarks and two LLMs. It reports lower input-token usage and computational cost while maintaining performance parity with the Original agent.

8 Conclusion

  • The study analyzes SWE-bench Verified trajectories, proposes AgentDiet, and evaluates it in a coding agent on SWE-bench Verified and Multi-SWE-bench Flash using Claude 4 Sonnet and Gemini 2.5 Pro.Source: §8 · p. 20
  • 39.9%–59.7% lower input token usage and 21.1%–35.9% computational cost savings accompany performance parity with the Original agent, ranging from -1.0% to +2.0%.Source: §8 · p. 20
  • The authors conclude that agent costs can be reduced without compromising performance and describe the approach as beneficial to practical agent use and future research.Source: §8 · p. 20
Loading 2509.23586v2…