PaperlayerView exact revision on arXiv

Source-linked AI summary

CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents

Yujiang Li, Zhenyu Hou, Yi Jing, Jie Tang, Yuxiao Dong

arXiv:2607.05378v2cs.LG

TL;DR

Long-horizon agents can exceed finite context windows, while compaction's role in RL training remains underexplored. CompactionRL jointly trains execution and summary generation with token-level loss normalization and cross-segment GAE. It improves performance over base models with inference-time compaction on two coding benchmarks, though its gains are designed for compaction-enabled inference.

  • Problem

    Long-horizon interaction histories can exceed finite context windows, and prior compaction has mostly been an inference-time heuristic or external-memory operation rather than jointly trained in RL.

    Source: §1 · p. 1

  • Method

    CompactionRL trains execution and summary generation under a shared task reward, using token-level loss normalization and cross-segment GAE for compacted rollouts.

    Source: §1 · p. 2

  • Results

    Against base models with inference-time compaction, CompactionRL gains 5.5 and 6.7 points for GLM-4.7-Flash and 6.6 and 4.9 points for GLM-4.5-Air on SWE-bench Verified and Terminal-Bench 2.0, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CompactionRL improves performance under a fixed peak context budget, supporting context compaction as part of RL training rather than solely an inference-time heuristic.

    Source: §5 · pp. 9–10

    CompactionRL is designed for compaction during both training and inference; whether its benefits transfer to single-window inference requires further study.

    Source: §5 · p. 10

Abstract

from arXiv · show

Long-horizon agentic LLMs are increasingly limited by finite context windows, as extended interaction trajectories can exceed the maximum context length before a task is completed. Context compaction offers a natural solution by summarizing previous interaction states and continuing the rollout under a compressed context, but incorporating compaction into reinforcement learning remains underexplored. We propose CompactionRL, a reinforcement learning strategy to train long-horizon agentic LLMs with context compaction. Our approach jointly optimizes task execution and summary generation with token-level loss normalization and cross-segment generalized advantage estimation. This design enables the LLM agents to learn from compacted long-horizon trajectories. We train CompactionRL on top of open models and observe consistent performance gains on agentic coding tasks. CompactionRL enables the open GLM-4.5-Air model (106B-A12B) to achieve Pass@1 scores of 66.4% on SWE-bench Verified and 26.2% on Terminal-Bench 2.0, exceeding the base model under inference-time compaction by 6.6 and 4.9 points, respectively. Built upon GLM-4.7-Flash (30B-A3B), CompactionRL improves Pass@1 by 5.5 and 6.7 points against the base model, reaching 56.0% on SWE-bench Verified and 20.2% on Terminal-Bench 2.0. CompactionRL is thus deployed in the RL pipeline for training the open GLM-5.2 model (750B-A40B).

1 INTRODUCTION

Long-horizon agents can exhaust finite context windows, and compaction makes training summaries consequential because later actions depend on them. CompactionRL trains execution and summarization together, reporting gains on two coding benchmarks over base models with inference-time compaction.

1 INTRODUCTION

  • Long interaction histories can exceed finite context windows; simply scaling context is costly and does not fully solve degraded utilization over long sequences.Long-horizon tasks involve repeated reasoning, actions, environment feedback, and plan revision.Source: §1 · p. 1
  • Compaction replaces earlier history with a summary that determines information available to subsequent actions, so task success depends on summary quality as well as execution.The paper describes prior compaction uses mainly as inference-time heuristics or external-memory operations.Source: §1 · p. 1
  • Compaction-aware asynchronous RL faces rollout-time variation and policy lag, summary/execution length imbalance, and distorted credit assignment when GAE is computed separately by segment.The introduction also notes single-rollout PPO and a learned critic for token-level advantages as ways to address rollout-group waiting and advantage estimation.Source: §1 · pp. 1–2
  • CompactionRL jointly trains task execution and summarization under the same final task reward, treating summarization as a learned policy rather than an inference-time heuristic.The framework incorporates compaction into rollout sampling and continues from a summary when the context budget is exhausted.Source: §1 · p. 2
  • Token-level loss normalization reduces length-induced weighting bias, while cross-segment GAE preserves temporal credit assignment across compaction boundaries.These methods address unequal execution and summary lengths and reward-credit distortion across segments.Source: §1 · p. 2
  • Against base models with inference-time compaction, CompactionRL gains 5.5 and 6.7 points for GLM-4.7-Flash and 6.6 and 4.9 for GLM-4.5-Air on SWE-bench Verified and Terminal-Bench 2.0, respectively.The benchmark order pairs the first gain with SWE-bench Verified and the second with Terminal-Bench 2.0 for each model.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses long-horizon agents through context management and memory, and RL work spans group-relative and critic-based approaches. Joint training of coding-agent execution and compaction segments remains underexplored.

2 RELATED WORK

  • Context-management RL methods include SUPO and Memex(RL), which use GRPO without accounting for context-state differences across compaction segments in advantage calculation; ReSum segments trajectories without training the summarizer.Other cited work trains only the context manager, while joint training of execution and compaction segments for long-horizon coding agents remains underexplored.Source: §2 · p. 3
  • RL for language models ranges from PPO with value functions to critic-free or group-relative objectives, while VC-PPO, VAPO, and SAO motivate critic-based optimization and single-rollout sampling for agentic RL.The cited work supports state-dependent advantage estimation and training without group-relative baselines.Source: §2 · p. 3

3 THE COMPACTIONRL METHOD

CompactionRL trains long-horizon agents to generate summaries during rollouts and optimize those summaries alongside task execution. Its optimization handles token-length imbalance and credit assignment across compacted segments.

3.1 TRAINABLE CONTEXT COMPACTION

  • Assistant-observation pairs remain atomic, while execution and summary segments are jointly trained using the same final task reward, without a separate summary-quality reward.Summary tokens are sampled from the trainable policy and included in the RL objective.Source: §3.1 · pp. 3–4
  • At compaction, policy-generated summaries reconstruct history alongside recent turns, preserving long-range information while retaining the latest environment state exactly.The default recent-turn tail is two turns, reduced if needed to fit the context budget.Source: §3 · p. 3, §3.1 · p. 4
  • 55.5 vs 49.0 SWE-Verified accuracy (pass@1) when only the summary agent changes; the best summarizer also triggers slightly fewer compactions per trace.The execution agent is fixed to GLM-4.7-Flash.Source: §3.1 · p. 4, Table 1 · p. 5

3.2 OPTIMIZATION UNDER COMPACTED ROLLOUTS

  • Single-rollout PPO avoids waiting for the slowest rollout in a group, reducing associated policy lag and off-policy effects; a learned critic estimates context-dependent token advantages.Resumed segments condition on different compacted histories.Source: §3.2 · p. 5
  • Token-level loss normalization gives each trainable token equal weight, avoiding segment-count bias when rollouts have different numbers and lengths of segments.Segment-level averaging could otherwise give more weight to rollouts with more compaction events despite equal task rewards.Source: §3.2 · p. 5
  • Cross-segment GAE corrects local advantages so terminal reward reflects its distance across the concatenated rollout, rather than seeming close at every segment end.The trajectory-position correction approximates full-trajectory GAE; exact recursive propagation across segments would preclude parallel batch processing.Source: §3.2 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate CompactionRL on coding benchmarks across two model scales, finding the strongest performance for the full method. Ablations and behavioral analyses examine its optimization components and training dynamics.

4.1 EXPERIMENTAL SETUP

  • Experiments train GLM-4.7-Flash and GLM-4.5-Air-SFT, then report Pass@1 on 200 sampled SWE-bench Verified tasks and the full Terminal-Bench 2.0 set.Evaluation allows up to three compactions and 250 interaction turns; SWE-bench public baselines use the full benchmark and may use different scaffolds.Source: §4.1 · p. 6, §4.2 · p. 7

4.2 MAIN RESULTS

  • CompactionRL achieves the best performance across both model scales and benchmarks, with gains over the variant that masks summary-response tokens.The comparison covers SWE-bench Verified and Terminal-Bench 2.0 under single-window and compacted evaluation settings.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 ABLATION STUDIES

  • CompactionRL outperforms the SUPO GRPO baseline by 4.6 points on SWE-bench Verified and 4.9 points on Terminal-Bench 2.0.SUPO assigns rollout-level group-relative advantages uniformly across compaction segments, whereas CompactionRL uses a learned critic for token-level advantages.Source: §4.3 · pp. 7–8, Table 3 · p. 8
  • Removing either token-level loss normalization or cross-segment GAE degrades performance, with a larger degradation when loss normalization is removed.The ablation attributes particular importance to correcting optimization bias from variable segment counts and lengths.Source: §4.3 · p. 8, Table 3 · p. 8
  • CompactionRL outperforms standard no-compaction RL with a 160k context under both compacted and non-compacted evaluation.The 160k baseline is intended to retain the full trajectory without compaction-induced information loss.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 ANALYSIS

  • CompactionRL uses fewer compactions and tool calls than the base model and the no-summary-loss variant, but more than standard no-compaction RL.It also achieves the highest Pass@1 on tasks that trigger compaction.Source: §4.4 · p. 8, Figure 3 · p. 9
  • With summary responses in the RL objective, summaries grow longer and more detailed, while reasoning tokens per turn increase and policy entropy rises more slowly.Without summary loss, summary length decreases during training; reasoning tokens also decrease for that variant and standard no-compaction RL.Source: §4.4 · p. 9, Figure 4 · p. 9

5 CONCLUSION

CompactionRL trains long-horizon agents by optimizing execution and summary tokens under a shared task reward, with token-level loss normalization and cross-segment GAE. It improves performance under a fixed peak context budget, while its gains have limits in single-window inference and its credit assignment remains approximate.

5 CONCLUSION

  • CompactionRL improves SWE-bench Verified and Terminal-Bench 2.0 performance under a fixed peak context budget, using token-level loss normalization and cross-segment GAE.Summary training produces more detailed compacted states and is important for the gains.Source: §5 · p. 9
  • The authors conclude that incorporating context compaction into RL training is effective for long-horizon agents, rather than treating compaction solely as an inference-time heuristic.Source: §5 · p. 10
  • CompactionRL is designed for settings with compaction during both training and inference; transfer to single-window inference is inconsistent, and cross-segment GAE approximates full-trajectory credit assignment.Source: §5 · p. 10

A.1 SWE-BENCH VERIFIED SUBSET

SWE-bench Verified experiments use a common 200-task subset spanning 10 repositories.

A.1 SWE-BENCH VERIFIED SUBSET

  • All SWE-bench Verified results in the experiments use the same 200-task subset spanning 10 repositories.Source: §A.1 · p. 13
  • Table 5 lists the complete evaluation subset grouped by repository, with repository prefixes and numeric instance IDs.For example, astropy__astropy and instance ID 7671 identify astropy__astropy-7671.Source: §A.1 · p. 13

A.2 TRAINING AND COMPUTE DETAILS

The appendix describes model and compute configurations, and reports similar CompactionRL performance across two random seeds under the same configuration.

A.2 TRAINING AND COMPUTE DETAILS

  • CompactionRL training was repeated with two random seeds under the same configuration, with similar performance across seeds.Source: §A.2 · p. 13
  • The policy and critic share a backbone initialization, with the critic augmented by a scalar value head.Source: §A.2 · p. 13
  • Table 6 summarizes training configurations and compute resources, while Table 7 compares CompactionRL and GRPO training-resource use under GLM-4.5-Air (106B-A12B).Source: §A.2 · p. 13, Table 6 · p. 14, Table 7 · p. 14

A.3 COMPACTION PROMPT TEMPLATES

The compaction prompts preserve task context and recent interaction details, while directing the agent to resume from a summary and retained messages.

A.3 COMPACTION PROMPT TEMPLATES

  • The fixed summary-generation template asks for a detailed account of the conversation, including the original task and completed work.It says to preserve exact commands, outputs, code behavior, and error text when those details matter for continuing the work.Source: §A.3 · p. 15
  • The summary covers only the earlier conversation portion because recent messages remain verbatim, and it is instructed not to repeat those messages unless continuity requires it.Source: §A.3 · p. 15
  • The summarization prompt requires exactly the specified tags and lists continuity details such as prior work, decisions, errors, and pending tasks.Source: §A.3 · p. 15
  • The resume prompt presents the summary and recent messages as context, treats recent messages as the latest state, and directs continuation without further questions.The prompts also instruct the agent to work only from the conversation and not assume access to external files or tools.Source: §A.3 · p. 15

B ANALYSIS ON HYPERPARAMETER SENSITIVITY

CompactionRL maintains stable performance within practical inference-time settings, with only a minor drop when the recent-history tail is removed.

B ANALYSIS ON HYPERPARAMETER SENSITIVITY

C SUMMARY QUALITY ANALYSIS

CompactionRL summaries score higher than summaries from RL without compaction training, and the analysis examines how summary quality relates to task continuation.

C SUMMARY QUALITY ANALYSIS

  • The summary-quality comparison covered 79 samples from a 200-task SWE-bench Verified subset where at least one method compacted, with scores repeated three times.Multiple summaries within a trajectory were averaged first.Source: C SUMMARY QUALITY ANALYSIS · p. 16
  • CompactionRL’s mean summary-quality advantage was +0.22 (95% bootstrap CI [0.020, 0.420]); scores of 0–2 fell from 22.2% to 11.9%.Source: Table 9 · p. 16, C SUMMARY QUALITY ANALYSIS · p. 16
  • On the subset where both methods compacted, task success was 38.10% for CompactionRL versus 28.57% for RL without compaction training.Source: C SUMMARY QUALITY ANALYSIS · p. 16
  • The rubric assesses requirement fidelity, current-state fidelity, failure-history fidelity, next-step validity, and continuation efficiency and progress.Source: C SUMMARY QUALITY ANALYSIS · p. 17
  • Three recurring summary failures were misdiagnosed failures, stale repository state, and premature completion; detailed state helps the post-compaction agent recover, finish, and verify.Source: C SUMMARY QUALITY ANALYSIS · pp. 17–18

Related papers

Loading 2607.05378v2…