PaperlayerView exact revision on arXiv

Source-linked AI summary

The Hidden Costs of Forgetting: A Study of Trajectory Deletion in Software Engineering Agents

Fernando Vallecillos-Ruiz, Islem Bouzenia, Michael Pradel, Leon Moonen

arXiv:2610.35496v1cs.SEcs.AI

TL;DR

Software-engineering agents resend growing trajectories, but evidence is limited on how deletion choices affect effectiveness, behavior, and cost. This study evaluates 42 deletion configurations across agents and tasks. Delayed deletion often preserves task-solving effectiveness, while re-fetching and caching-aware pricing can erase estimated cost savings.

  • Problem

    Prior work shows agents can tolerate trajectory reduction, but how deletion design choices affect effectiveness, behavior, and execution cost remains unclear.

    Source: §I · pp. 1–2

  • Method

    The study evaluates 42 trajectory-deletion configurations across two agents and 100 SWE-bench Verified tasks, measuring effectiveness, behavior, and estimated costs.

    Source: §I · p. 2, §IX · p. 10

  • Results

    With a two-turn delay, 37 of 42 evaluations show no significant decrease in tasks solved, but none of the delayed evaluations are cheaper under caching-aware pricing.

    Source: §I · p. 2, §IX · p. 10

  • Takeaways & Limitations

    Evaluations should consider execution length, retrieval behavior, and realistic pricing, since unchanged task success and lower token counts do not ensure lower costs.

    Source: §IX · p. 10

    Findings are bounded by one open-weight model, two agents, and a 100-task SWE-bench Verified subset, and the cost model assumes matching prefixes are cached.

    Source: §VII · p. 10

Abstract

from arXiv · show

Software engineering agents resend their entire trajectory to an LLM at every turn, motivating online trajectory reduction. Prior work shows that randomly deleting 75% of tokens in old turns may preserve agent effectiveness, but it remains unclear why, how agents compensate for missing content, and whether it actually reduces costs. This paper presents an empirical study of 42 trajectory-reduction configurations, focusing on effectiveness, agent behavior, and cost. We systematically explore deleting content from old turns by varying the scope, granularity, amount, and timing of deletion across 9,400 runs with two state-of-the-art agents on SWE-bench Verified. We find deleting large amounts of trajectory content (e.g., up to 100% of reasoning text) can preserve overall effectiveness. Importantly, such deletion must be applied with a delay, as immediate deletion tends to significantly degrade effectiveness. Agents often compensate for deleted content by re-fetching it, increasing both the number of turns taken and the input/output tokens. Whether trajectory reduction reduces costs depends on the pricing model: Under a flat pricing model, estimated costs fall in 26 of 42 evaluations. Surprisingly, under a caching-aware pricing model, zero evaluations yield lower costs, since reducing the trajectory breaks the cached prefix billed at a lower rate. Our findings have three implications: First, token counts alone are misleading; trajectory reduction techniques must consider overall costs under a realistic pricing model. Second, analyzing trajectories is crucial to understand agent behavior, as end-to-end effectiveness hides differences such as excessive re-fetching. Finally, these results highlight the need for future work on memory management to reduce costs without sacrificing effectiveness.

I. INTRODUCTION

The paper studies how trajectory-deletion choices affect software-engineering agents’ effectiveness, behavior, and costs. It finds that delayed deletion often preserves task-solving effectiveness, while re-fetching and pricing-model effects can offset token savings.

I. INTRODUCTION

  • Prior evidence that agents tolerate substantial trajectory reduction leaves unclear how individual deletion choices affect them or how to evaluate resulting behavior and costs.The paper argues for assessing task effectiveness, agent behavior, and execution cost together.Source: §I · p. 1
  • The study varies deletion scope, granularity, amount, and timing across 42 configurations, two agents, 100 SWE-bench Verified tasks, and 9,400 runs.It measures effectiveness, subsequent agent behavior, and execution costs under flat and caching-aware pricing.Source: §I · p. 1
  • With a two-turn delay, 37 of 42 evaluations do not significantly reduce tasks solved; immediate deletion significantly reduces tasks solved in 19 of 42.Deleting reasoning usually does not affect effectiveness, whereas deleting tool output is more harmful.Source: §I · pp. 1–2
  • Deleting tool outputs raises re-fetch rates to as much as 7.5 times baseline with delayed deletion and 10.6 times with immediate deletion.Agents often compensate for deleted content by retrieving it again.Source: §I · p. 2
  • Under flat pricing, 26 of 42 delayed-deletion evaluations reduce estimated costs; under caching-aware pricing, none do, and 26 are significantly more expensive.Rewriting a trajectory can invalidate the cached prompt prefix, so token reductions do not guarantee cost savings.Source: §I · pp. 1–2

II. BACKGROUND

Agent trajectories accumulate reasoning, tool calls, and outputs across repeated model interactions. Reduction approaches rewrite earlier turns, while prefix caching can make those rewrites costly.

A. Agents and Their Trajectories

  • A trajectory is the sequence of turns containing model reasoning, tool calls, and resulting outputs; retaining it makes subsequent prompts grow.Tool outputs such as file contents, search results, and test logs contribute to that growth.Source: A. Agents and Their Trajectories · p. 2

B. Trajectory Reduction

  • Trajectory-reduction approaches mask, select, or summarize earlier content; AgentDiet’s random baseline deletes 75% of eligible tokens and solves almost as many tasks as its model-based approach.The paper builds on this prior result to study agent compensation and execution costs.Source: B. Trajectory Reduction · p. 2

C. Prefix Caching and Its Pricing

  • With prefix caching, rewriting an old turn ends cache reuse from the first changed token, so later prompt tokens are billed at the full input price until prefixes match again.The figure illustrates a modified T2 causing T2 and later turns to be rebilled, before a later call matches the prefix again.Source: C. Prefix Caching and Its Pricing · p. 2, Figure 1 · p. 2

III. STUDY DESIGN

The study evaluates two agents on a fixed SWE-bench Verified subset and varies four dimensions of trajectory deletion. Its configurations distinguish what is deleted, how deletion is applied, how much is removed, and when rewriting occurs.

A. Agents and Tasks

  • The study uses Trae agent and Mini-SWE-agent with the same model, evaluating a fixed 100-task SWE-bench Verified subset previously used by AgentDiet.The agents contrast specialized tools with a single-shell interface.Source: A. Agents and Tasks · p. 3

B. Deletion Configurations

  • The 42 configurations vary deletion scope, granularity, amount, and timing, and are evaluated on both agents.Scope covers reasoning, tool output, or both; turn-level deletion removes scoped content as a whole, while token-level deletion removes a random share and retains the rest.Source: B. Deletion Configurations · p. 3
  • Delayed deletion keeps the last two turns intact, whereas immediate deletion rewrites eligible turns before the next model call.The figure contrasts deleting whole earlier turns with modifying parts of a turn.Source: B. Deletion Configurations · p. 3, Figure 2 · p. 3

C. Research Questions and Metrics

The study measures task solving and execution length, analyzes agents’ re-fetching behavior, and estimates costs under flat and caching-aware pricing. Its comparisons use paired task-level resampling to quantify uncertainty.

C. Research Questions and Metrics

  • Effectiveness is measured by tasks solved, while run length is summarized as median turns; runs can also be capped at 200 turns or classified as looping after ten repeated identical tool calls.Configurations are compared with uncompressed runs and, for selected dimensions, against one another.Source: C. Research Questions and Metrics · pp. 3–4
  • A read is labeled a re-fetch when at least 80% of its distinct lines appeared in an earlier read, and re-fetch rates are normalized to uncompressed runs.A rule-based labeler classifies tool calls, with 851 of 855 labels verified as correct in sampled logs.Source: C. Research Questions and Metrics · p. 4
  • The cost analysis compares flat pricing with caching-aware pricing, where only the longest matching prompt prefix receives the cached input price.The model assumes matching prefixes are cached and omits cache-write premiums, making its estimates a lower bound on realistic costs.Source: C. Research Questions and Metrics · p. 4

D. Statistical Analysis

Delayed deletion usually preserves task-solving results, but can lengthen runs, especially when tool outputs are deleted. The section also describes the study’s comparison methods and implementation, and details how deletion scope, granularity, and amount shape outcomes.

D. Statistical Analysis

  • Comparisons use five uncompressed runs of the same agent as the reference and paired bootstrap confidence intervals over 100 tasks, with 10,000 resamples.A change is significant when its interval excludes zero for tasks solved or one for ratios.Source: D. Statistical Analysis · p. 4

E. Implementation and Data

  • Experiments ran in Apptainer with vLLM on NVIDIA GH200 GPUs, using an approximately 262,000-token context window and temperature zero.The replication package contains all 9,400 runs, logs, and analysis code.Source: E. Implementation and Data · p. 4

A. Delayed Deletion: Scope, Granularity, and Amount

  • 37 of 42 delayed evaluations show no statistically significant decrease in tasks solved, while 21 have significantly longer runs, all deleting tool output.Table I reports tasks solved and median run-length change relative to the uncompressed baseline; three of the five significant losses exceed ten tasks.Source: A. Delayed Deletion: Scope,… · pp. 4, 6, §IV · p. 4, Table I · p. 5
  • Across 14 delayed reasoning-deletion evaluations, tasks solved range from five fewer to four more than baseline, with no statistically significant decrease or increase in run length.At a fixed deletion percentage, reasoning deletion removes fewer tokens than deleting tool output or both, so the scope effect is not isolated from deletion amount.Source: A. Delayed Deletion: Scope,… · p. 5
  • Token-level deletion lengthens runs more than turn-level deletion in 10 of 12 tool-output pairs, while only one paired comparison differs significantly in tasks solved.For Trae agent deleting both scopes at 75%, turn-level deletion solves 69 tasks versus 60 with token-level deletion.Source: A. Delayed Deletion: Scope,… · p. 5
  • At 100% turn-level deletion of both scopes, Trae agent’s tasks solved fall from a baseline mean of 74.6 to 43, and capped runs rise from four to 92.By contrast, with 100% turn-level deletion of tool output, it solves 70 tasks while median run length rises 73% and 29 runs reach the cap.Source: A. Delayed Deletion: Scope,… · p. 5
  • Compared with AgentDiet, this study reports 15 and 32 fewer tasks solved for its two corresponding Trae agent configurations, versus AgentDiet’s one and seven.Direct comparison is limited by differences in which turns are rewritten, the model, and the turn limit.Source: A. Delayed Deletion: Scope,… · pp. 5–6
  • Future evaluations should measure tasks solved, execution length, and budget exhaustion, and compare policies at matched token-removal amounts.The authors also motivate memory-management policies that treat reasoning and tool outputs differently.Source: A. Delayed Deletion: Scope,… · p. 6

B. Immediate Deletion: The Effect of Timing

Immediate deletion generally harms task solving and lengthens runs more than delayed deletion. Effects vary by deletion granularity and content: tool-output deletion can trigger loops or missing patches, while reasoning deletion has more modest effects on task solving.

B. Immediate Deletion: The Effect of Timing

  • At 100% immediate deletion of tool output or both scopes, Trae agent solves no tasks, while Mini-SWE-agent solves 14 with tool output deleted and two with both deleted.Trae agent repeats a directory listing until the cap in every run when both scopes are deleted; Mini-SWE-agent’s 14 successful patches occur without observing file contents.Source: B. Immediate Deletion: The Effect of… · p. 6
  • Immediate token-level tool-output deletion reduces Trae agent’s tasks solved at every percentage, while Mini-SWE-agent solves 27 tasks at 75% deletion.The authors suggest, without establishing causation, that remaining fragments can look like repository corruption and prompt repair attempts.Source: B. Immediate Deletion: The Effect of… · p. 6
  • Immediate and delayed reasoning deletion yield similar task counts in 13 of 14 evaluations, but seven immediate configurations have significantly longer runs.At 100% turn-level reasoning deletion, runs lengthen 35% for Trae agent and 105% for Mini-SWE-agent.Source: B. Immediate Deletion: The Effect of… · p. 6
  • 19 of 42 immediate-deletion evaluations significantly reduce tasks solved, and 34 significantly lengthen runs; no immediate configuration significantly shortens runs.Paired comparisons show significantly fewer tasks solved with immediate deletion in 16 of 42 timing comparisons.Source: B. Immediate Deletion: The Effect of… · p. 6
  • Future memory-management studies should vary retention delay and deletion scope together, measuring task solving, execution length, failure behavior, and budget exhaustion.The paper proposes testing whether reasoning and tool outputs require different protection periods.Source: B. Immediate Deletion: The Effect of… · p. 6

V. RQ2: HOW DOES THE AGENT COMPENSATE FOR DELETED CONTENT?

Agents often re-fetch deleted content rather than simply proceeding without it. Re-fetching increases with tool-output deletion and can add turns, with retrieval reaching farther back than the study’s two-turn delay.

V. RQ2: HOW DOES THE AGENT COMPENSATE FOR DELETED CONTENT?

  • At matched amounts and granularities, deleting both scopes yields higher re-fetch ratios than deleting tool output in all Trae and most Mini-SWE-agent comparisons; token-level deletion exceeds turn-level in 21 of 24 pairs, with three ties.Under delayed turn-level deletion, Trae agent’s ratio for both scopes grows from 1.7× at 25% to 7.5× at 100%.Source: §V · p. 7
  • Immediate deletion produces significantly higher re-fetch rates than delayed deletion in 25 of 42 comparisons, with the largest values in token-level rows.The highest reported ratio is 10.6× for Trae agent under immediate turn-level deletion of both scopes at 100%.Source: §V · p. 7
  • Delayed reasoning deletion has no consistent re-fetch effect, but immediate deletion of the latest reasoning reaches 1.9× under turn-level deletion and 1.5× under token-level deletion.Removing reasoning older than two turns does not produce the same increase.Source: §V · p. 7
  • Deleted outputs are re-fetched 1.5 to five times as often as kept ones, with a significant difference in all but one of 24 evaluations.The configurations do not distinguish outputs the agent still needs from those it does not.Source: §V · p. 7
  • Tool-output deletion raises re-fetch rates up to 7.5× with delayed deletion and 10.6× without it; retained-content re-fetching stays near baseline.The extra re-fetches are of deleted content, and delayed reasoning deletion produces no consistent increase.Source: §V · p. 8
  • Additional re-fetches primarily appear as extra turns, and re-fetches reach a median six to twelve turns back, beyond the two-turn delay.Future work should jointly evaluate retention and retrieval, including the effort required to recover deleted information.Source: §V · p. 8

VI. RQ3: HOW DOES DELETION CHANGE COST?

The section’s cost table reports estimated cost ratios for delayed-deletion configurations under flat and caching-aware pricing, relative to uncompressed runs.

VI. RQ3: HOW DOES DELETION CHANGE COST?

  • Table II reports delayed-configuration estimated costs as ratios to uncompressed runs under flat and caching-aware pricing.Ratios of 0.80 and 1.20 indicate costs 20% below and above baseline, respectively.Source: §VI · p. 8
  • The table focuses on delayed configurations because most preserve tasks solved without a statistically significant decrease; immediate configurations produce larger decreases that complicate per-run cost interpretation.The passage notes that immediate-configuration cost results are reported separately.Source: §VI · p. 8
  • Cost per task solved remains future work.Source: §VI · p. 8

A. Cost Under Flat and Caching-Aware Pricing

Estimated cost outcomes differ sharply by pricing model, and immediate deletion usually raises costs. Scope and granularity also shape the cost differences.

A. Cost Under Flat and Caching-Aware Pricing

  • Under flat pricing, 26 of 42 evaluations fall below baseline, with six significant savings and three significant increases; under caching-aware pricing, all exceed baseline, 26 significantly.The caching-aware configurations closest to parity remain no more than 5% above baseline.Source: A. Cost Under Flat and Caching-Aware… · p. 8
  • Four significant flat-pricing savings come from reasoning deletion, all four for Trae agent, with estimated costs up to 26% below baseline.Under caching-aware pricing, all 14 reasoning-deletion ratios exceed one, six significantly.Source: A. Cost Under Flat and Caching-Aware… · p. 8
  • In 17 of 18 matched pairs, turn-level deletion costs no more than token-level deletion under caching-aware pricing.The remaining pair differs by 0.02 in cost ratio.Source: A. Cost Under Flat and Caching-Aware… · p. 8
  • Without delay, 29 of 42 evaluations cost significantly more under flat pricing and 32 of 42 under caching-aware pricing.Across immediate configurations, one evaluation is cheaper under flat pricing and none under caching-aware pricing; no immediate reasoning-deletion evaluation is cheaper under either model.Source: A. Cost Under Flat and Caching-Aware… · pp. 8–9

B. Source of the Cost Difference

Rewriting a trajectory re-bills its changed suffix, while longer runs can add further input and output costs. As a result, token reductions and flat-pricing estimates can overstate savings under caching-aware pricing.

B. Source of the Cost Difference

  • In 31 of 42 delayed evaluations, token volume falls below baseline by up to one third, but full-price input tokens rise in all 42, from about 1.25 to nearly 10 times baseline.Every rewrite re-bills the prompt from its first changed token onward; caching-aware pricing captures lost cache discounts and additional tokens from longer runs.Source: B. Source of the Cost Difference · p. 9
  • At baseline, cached, new-input, and output tokens account for 50%, 14%, and 36% of Mini-SWE-agent’s cost and 52%, 14%, and 34% of Trae agent’s.Neither baseline has re-billed cost.Source: B. Source of the Cost Difference · p. 9
  • Re-billed input accompanies new input, and under token-level deletion at 25%, its cost component is about five times that of turn-level deletion.Under turn-level deletion, re-billed cost grows with deletion percentage because only selected turns invalidate the prefix.Source: B. Source of the Cost Difference · p. 9
  • Re-billing contributes to every configuration’s estimated cost increase, while cached-input reductions provide limited or no offset.Longer runs under tool-output deletion further increase token costs; delayed reasoning deletion can still incur re-billing without consistent increases in re-fetching or turns.Source: B. Source of the Cost Difference · p. 9
  • Token counts fall in 31 of 42 delayed evaluations and flat-pricing ratios fall below one in 26, but none of the delayed caching-aware evaluations costs less than baseline.Twenty-six delayed evaluations are significantly more expensive under caching-aware pricing; replacing an old turn invalidates the prefix from that point.Source: B. Source of the Cost Difference · p. 9
  • Six of 42 delayed configurations are significantly cheaper under flat pricing, compared with none under caching-aware pricing, where 26 are significantly more expensive.Re-billing offsets savings from delayed reasoning deletion, while extra turns and completions add cost under tool-output deletion.Source: B. Source of the Cost Difference · p. 9
  • Future evaluations should compare against an uncompressed baseline with the same cache assumptions and account for retrieval turns, generated reasoning, cache invalidation, and compression overhead.The paper also recommends reporting tasks solved alongside cost, potentially including cost per task solved.Source: B. Source of the Cost Difference · p. 9

VII. THREATS TO VALIDITY

The study’s findings are bounded by run variability, experimental settings, and the cost model, and may not generalize to other agents or compression approaches. The related work spans context management, trajectory reduction, and agent behavior; this study complements it by varying deletion choices and examining re-fetching and cache invalidation.

VII. THREATS TO VALIDITY

  • Each configuration runs once per agent and task with one random deletion draw, so confidence intervals cover task-sampling uncertainty but not intervention-run or deletion-draw variability.Holm correction leaves fewer individually significant results, although the reported grid trends hold.Source: §VII · p. 9
  • The findings use default prompts and tools, a 200-turn cap, a two-turn delay, one open-weight model, two agents, and 100 SWE-bench Verified tasks.Public repositories and fixes may permit model recall; reasoning deletion also removes fewer tokens than tool-output or combined-scope deletion at the same percentage.Source: §VII · p. 10
  • The caching-aware cost model assumes every matching prefix is available and omits cache-write premiums, minimum prefix lengths, expiry, and eviction.These simplifications can underestimate absolute costs and do not establish provider-independent relative cost increases.Source: §VII · p. 10

VIII. RELATED WORK

  • Context-management work ranges from masking observations to model-selected removal and trained context management; prompt compression chooses tokens by information content rather than randomly.This study provides a baseline by varying random deletion’s scope, granularity, amount, and timing.Source: §VIII · p. 10
  • Related trajectory-reduction studies examine compression primitives, reacquisition costs, and cached-prefix invalidation; this study links re-fetches to deleted versus retained outputs and reconstructs prefix invalidation from prompt logs.Prior work includes a report that removing 38% of tool-output tokens coincided with a 6.8% higher cost.Source: §VIII · p. 10
  • Prior agent-behavior studies examine common actions and misbehavior patterns, while this work inspects how agents behave after trajectory reduction.Source: §VIII · p. 10

IX. CONCLUSION

The conclusion finds that delayed deletion can preserve task success, but may increase agent work and does not guarantee cost savings. It recommends retaining recent tool outputs and evaluating compression against uncompressed agents under caching-aware pricing.

IX. CONCLUSION

  • With a two-turn delay, 37 of 42 evaluations show no significant decrease in tasks solved, but 21 have significantly longer runs.Additional re-fetches primarily recover deleted tool output; immediate tool-output deletion causes larger effectiveness declines, particularly at token-level granularity.Source: §IX · p. 10
  • The tested agents rely on recent tool outputs, and when removal is necessary, turn-level deletion generally preserves more tasks solved than token-level deletion in this setting.Token counts and flat pricing overstate the savings observed here; comparisons should use an uncompressed baseline under caching-aware pricing.Source: §IX · p. 10
  • Future work should evaluate longer retention delays, selection based on observed reuse, and less frequent history rewriting.Batched rewriting still invalidates the prefix, so net savings require evaluation rather than inference from rewrite frequency alone.Source: §IX · pp. 10–11

Related papers

Loading 2610.35496v1…