The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management

1 Introduction

• Environment observations make up around 84% of an average SWE-agent turn, while growing contexts raise token costs and quadratic attention costs.Large contexts also make relevant information harder to use, and individual SE tool outputs can run to thousands of tokens.Source: §1 · pp. 1–2
• Unchecked context growth can more than double agent costs, making context management an economic necessity.The authors report that the discussed management strategies consistently reduce costs and often improve performance.Source: §1 · p. 2
• The study systematically compares Observation Masking and LLM-Summary for cost and solve rate in SWE-agent, then probes generality with OpenHands.Experiments span model families, sizes, licenses, and reasoning regimes.Source: §1 · p. 2
• Both strategies more than halve costs versus the Raw Agent, and LLM-Summary does not consistently or significantly outperform Observation Masking across model configurations.The authors also report initial evidence of generalization to OpenHands and a hybrid approach with further cost reductions.Source: §1 · p. 2

2 Related Work

• SE-agent research has focused on improving solve rates through scaling, selection, critics, and search, often at increased inference cost.Efficient context management has received comparatively little attention.Source: §2 · pp. 2–3
• Figure 2 compares solve rate with average trajectory cost and reports Observation Masking on the most efficient frontier across tested models.The caption describes its solve rates as competitive with, and sometimes superior to, LLM-Summary.Source: Figure 2 · p. 3
• Prior context-size studies find that LLMs increasingly struggle to use provided context as it grows, while MEM1 does not compare against omission-based approaches.The cited MEM1 benchmarks use relatively short trajectories compared with SE-agent trajectories.Source: §2 · p. 3
• Xiao et al.’s Delete baseline removes full turns rather than summarizing them, and the authors report it is more efficient than LLM-Summary at comparable performance.The paper says this baseline does not exploit the prevalence of environment observations and breaks the reasoning trace across turns.Source: §2 · p. 3
• Figure 3 contrasts growing raw histories, summaries that retain recent turns, and masking of observations older than a fixed window.The diagram’s box heights indicate token counts.Source: Figure 3 · p. 4

3 Experimental Configuration

• The comparison uses rolling-window Observation Masking and prompt-based LLM-Summary implementations, with the latter adapted for controlled evaluation in SWE-agent.The study also probes generality with OpenHands.Source: §3 · p. 4, §3.2 · p. 5

3.1 Context Management Strategies

• A raw trajectory is a sequence of reasoning, action, and observation turns, and the policy conditions on its history to generate the next reasoning-action pair.Without management, the history grows with each turn, increasing costs and eventually reaching context limits.Source: §3.1 · p. 4
• Observation Masking replaces environment observations older than window M with placeholders while preserving the agent’s full reasoning and action history.It slows token growth but does not eliminate indefinite trajectory growth.Source: §3.1 · p. 5
• LLM-Summary prompts a summarizer on the prior summary and eligible turns, then recursively folds older interactions into an evolving summary while retaining recent turns.N controls how many turns accumulate at once, and M controls how many trailing turns remain unaltered.Source: §3.1 · p. 5

3.2 Experimental Configuration

• Experiments span model families, sizes, licenses, and reasoning regimes on SWE-bench Verified, using a 250-turn limit unless otherwise specified.The main settings use M=10 for Observation Masking and N=21, M=10 for LLM-Summary.Source: §3.2 · pp. 5–6

4.1 The Universal Benefit of Context Management

• In most configurations with trajectories long enough to benefit, both strategies cut cost per instance by more than 50%; in three of five setups, the most efficient strategy also beat Raw Agent solve rate.The Raw Agent was consistently the most expensive strategy.Source: §4.1 · p. 6

4.2 Observation Masking: Dominant Efficiency with Minimal Complexity

• Observation Masking had the lowest cost in four of five setups, reducing environment-observation tokens without a separate summarization call.It requires 10 warm-up turns versus 31 for LLM-Summary; a $0.03 saving per instance for Qwen3-Coder 480B amounts to $15 across the benchmark.Source: §4.2 · pp. 6–7

4 Main Results

• Both Observation Masking and LLM-Summary significantly reduce costs without significantly reducing solve rates, and LLM-Summary does not consistently or significantly outperform masking.Table 1 compares solve rate and instance cost across the tested SWE-agent configurations.Source: §4 · p. 6, Table 1 · p. 7

4.3 Challenging the Need for Complex Summaries

• Observation Masking matched or exceeded LLM-Summary on solve rate in the reported comparisons, including 54.8% versus 53.8% for Qwen3-Coder 480B.For Gemini 2.5 Flash, the reported LLM-Summary advantage was 36.0% versus 35.6%; the authors conclude recent context may often suffice.Source: §4.3 · p. 7

4.4 The Trajectory Elongation Effect

• LLM-Summary produced longer mean trajectories for Gemini 2.5 Flash and Qwen3-Coder 480B, which the authors say can diminish its efficiency gains.Gemini 2.5 Flash averaged 52 turns with LLM-Summary, versus 44 with Observation Masking and 50 with Raw Agent; Qwen3-Coder 480B trajectory length rose 15% versus Raw Agent and 13% versus Observation Masking. The authors attribute this partly to summaries encouraging agents to continue.Source: §4.4 · pp. 7–8, Figure 4 · p. 8

5.1 Probing for Generality With OpenHands

• On OpenHands, tuned Observation Masking matches LLM-Summary on cost and solve rate; the authors present this as initial evidence of generality when context skews toward environment observations.The rolling-window size is scaffold-specific: reusing SWE-agent’s optimal setting degraded performance, while tuning reproduced the result.Source: §5.1 · p. 8, Figure 5 · p. 10

5.2 The Costs of Summarization

• Summary-generation API calls account for up to 7.2% of total instance cost, and subtracting them makes the cost-efficiency gap largely disappear in most experiments.Each call processes a unique sequence of turns, limiting cache reuse to the LLM-Summary system prompt; the caption identifies Table 2 as per-model mean summary costs.Source: §5.2 · p. 9, Table 2 · p. 9

5.3 Hybrid: Combined Observation Masking and LLM-Summary

• The hybrid applies Observation Masking during turn accumulation, then defers LLM-Summary as a last resort to bound long trajectories.The tested configuration used N = 43 and M = W = 10, with unmasked context passed to the summarizer.Source: §5.3 · p. 9
• 7% and 11% lower costs than Observation Masking and LLM-Summary, respectively, with a 2.6-percentage-point solve-rate gain over Raw Agent, in the hybrid evaluation.The full-benchmark savings are reported as $20 versus Observation Masking and $35 versus LLM-Summary.Source: §5.3 · p. 9, Figure 5 · p. 10
• Reusing the individual strategies’ hyperparameters (N = 21, M = W = 10) degraded hybrid cost efficiency through compounding KV-cache inefficiencies and summarization overhead.The hybrid evaluation used Qwen3-Coder 480B with SWE-agent on SWE-bench Verified-50.Source: §5.3 · pp. 9–10

6 Limitations

• The study uses SWE-bench in the SE domain, where verbose tool outputs favor Observation Masking, so its apparent superiority may not generalize to domains with succinct interactions.The masking window and summary schedule are fixed heuristics that ignore observation relevance, staleness, semantic boundaries, and agent subgoals; scaffold generality also needs broader study.Source: §6 · p. 10

7 Conclusion

• Efficient context management consistently and significantly reduces system costs by > 50% without significantly reducing downstream performance.The paper reports results across diverse model configurations and agent scaffolds.Source: §7 · p. 10
• LLM-Summary does not consistently or significantly outperform the simple Observation Masking baseline.Source: §7 · p. 10
• The hybrid reduces costs by 7% and 11% versus Observation Masking and LLM-Summary, respectively, while improving downstream task performance.The authors connect these findings to economically feasible and environmentally sustainable LLM-agent deployment.Source: §7 · p. 10

A Experimental Configuration

• Gemini experiments accessed through Vertex AI use the cost returned by that API, while Qwen experiments use observed per-turn tokens priced post hoc.The Qwen models were self-hosted.Source: A Experimental Configuration · p. 14
• Qwen3-32B costs are inflated because Alibaba’s official pricing does not distinguish cache-hit from cache-miss input tokens.Source: A Experimental Configuration · p. 14
• The authors release experimental data and code through their HuggingFace dataset and repository.Source: A Experimental Configuration · p. 14

B Short Trajectories in Qwen3-32B (thinking): Hypotheses and Implications

• The Qwen3-32B (thinking) configuration has an around 50% shorter median trajectory than Qwen3-32B, with enabling thinking as the only configuration change.The authors report this as a surprising outlier and investigate its implications.Source: B Short Trajectories in Qwen3-32B… · p. 14
• Configuration and dataset checks found no errors or exit-status distribution differences beyond slightly improved performance under thinking.The qualitative review covered 20 trajectories; both configurations sometimes struggled with function-calling format, without suspicious exit patterns.Source: B Short Trajectories in Qwen3-32B… · p. 14
• At a median of 15 turns, trajectories fall short of LLM-Summary’s 31-turn warm-up; Observation Masking starts after 10 turns and still reduces cost by ≈10%.Both strategies maintain stable downstream performance in this setting.Source: B Short Trajectories in Qwen3-32B… · pp. 14–15

C Detailed Main Results

• Table 1’s symmetric confidence intervals are derived by averaging the asymmetric bounds, such as 17.0+3.4/−3.2 becoming 17.0 ± 3.3.The section points to Table 3 for the full asymmetric intervals underlying the main results.Source: C Detailed Main Results · p. 15
• Table 3 compares Solve Rate and Instance Cost for context-management strategies against the Raw Agent across models on SWE-bench Verified.It reports asymmetric 95% bootstrap confidence intervals and marks significance relative to the Raw Agent.Source: Table 3 · p. 15
• Table 3 reports each strategy’s change relative to the Raw Agent baseline and bolds the best-performing strategy for each metric.A dagger marks significance compared with the Raw Agent.Source: Table 3 · p. 15

C.1 Statistical Analysis

• Paired nonparametric bootstrap uses 10,000 replicates and 500 common instances per model to estimate strategy-minus-Raw-Agent differences, 95% percentile intervals, and significance at p < 0.05.Reported p-values of 0.0000 mean no sign crossing among replicates, at resolution ≤10^-4.Source: §C.1 · p. 15, Table 4 · p. 16

D Additional Studies

• The additional studies use SWE-bench Lite-50 for the critic-enhanced prompt and a random 150-instance SWE-bench Verified subset for configuration tests, all with GPT-4.1-mini.The subset-based sensitivity studies are released with the code.Source: D Additional Studies · p. 16

D.1 Observation Masking Configuration

• M = 10 yields the highest observed Observation Masking performance; performance falls when the window increases to M = 20.This configuration is used in the main experiments.Source: §D.1 · p. 16, Figure 7 · p. 17

D.2 LLM-Summary Configuration

• Using tail turns M > 0 improves LLM-Summary performance, and summarizing more turns at once improves solve rate; the main configuration uses N = 21 and M = 10.This differs from OpenHands’ 50-50 split between turns summarized and tail turns.Source: §D.2 · p. 17

D.3 Critic-Enhanced LLM-Summary

• Critic-enhanced LLM-Summary showed no solve-rate improvement over standard LLM-Summary and produced longer trajectories, increasing cost without improving downstream performance.The approach asks the LLM to generate a summary and execution-free critique, guided to assess looping, alignment with the problem, and the high-level solution approach.Source: §D.3 · pp. 17–18

D.4 Behavior of the Covered Context Management Strategies Across Turns

• Observation Masking performed equally or better than LLM-Summary on SWE-bench Lite-50, while fixed rolling-window masking was competitive on cost and context compression.Figure 8 reports means across three experiments with standard-deviation error bars.Source: §D.4 · p. 18, Figure 8 · p. 18
• Simulated trajectories closely matched observed trends; simulations repeated mean token counts by type from raw-agent turns to examine strategy behavior over longer trajectories.The empirical trends use three experiments per strategy, totaling 150 trajectories.Source: §D.4 · pp. 18–19, Figure 9 · p. 19
• Environment observations account for ≈84% of simulated turn tokens, helping explain why fixed rolling-window masking is a strong baseline; simulations suggest LLM-Summary may outperform it on longer trajectories.At lower turn counts, fixed rolling-window masking offers greater compression, while costs are similar; the LLM-based approach bounds maximum context size in a fuzzy manner.Source: §D.4 · p. 19
• M = 10 yields optimal downstream performance across context-window sizes in a single experiment on a random 150-sample SWE-bench Verified subset.Source: Figure 10 · p. 20

E LLM Summary Prompts

• The SWE-agent summary prompt adapts OpenHands’ prompt for software engineering and supplies the previous summary—or problem statement initially—plus turns to summarize; critic variants add execution-free critique.The prompt templates are shown in Figures 11–14.Source: E LLM Summary Prompts · p. 20, Figure 11 · p. 21, Figure 12 · p. 22

NeurIPS Paper Checklist

• The checklist justification says abstract and introduction claims are discussed in Sections 4–5 and supported by experiments spanning diverse configurations.Source: NeurIPS Paper Checklist · p. 25

• The authors state that Section 6 documents the study’s limitations and threats to validity.Source: §2 · p. 25
• The experimental setup documents the models, hyperparameters, and benchmark, with configuration details in Sections D.4 and D.Source: §2 · pp. 26–27
• The authors released their SWE-agent LLM-summary implementation and made main experiment results available through HuggingFace.Source: §2 · p. 27