PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management

Miao Lu, Weiwei Sun, Weihua Du, Zhan Ling, Xuesong Yao, Kang Liu, Jiecao Chen

arXiv:2510.06727v1cs.CLcs.AIcs.LG

TL;DR

Long-horizon LLM tool-use RL faces rapidly growing interaction context. SUPO integrates learned summaries into RL training, and experiments report higher scores than GRPO on CodeGym and BrowseComp-Plus.

  • Problem

    In long-horizon LLM tool-use RL, dozens to hundreds of calls make prompts, outputs, tool observations, and reasoning traces grow rapidly, creating a central context-length challenge.

    Source: §1 · pp. 1–2

  • Method

    SUPO periodically replaces tool-use history with agent-generated task-relevant summaries and uses a policy-gradient formulation to jointly optimize summarization and tool-use behavior.

    Source: §1.1 · p. 2, §6 · p. 13

  • Results

    Across CodeGym and BrowseComp-Plus, SUPO scores higher than GRPO; on BrowseComp-Plus, it reaches 53%, a 14% improvement over GRPO with 64K working context.

    Source: §5.2.1 · p. 10

  • Takeaways & Limitations

    The findings support summarization-based context management as a scalable approach to RL fine-tuning beyond fixed context-length constraints.

    Source: §1.1 · p. 2, §6 · p. 13

    The policy-gradient treatment is demonstrated for this context-management technique; deriving correct gradients for more general agentic workflows remains future work.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

We study reinforcement learning (RL) fine-tuning of large language model (LLM) agents for long-horizon multi-turn tool use, where context length quickly becomes a fundamental bottleneck. Existing RL pipelines can suffer from degraded instruction following, excessive rollout costs, and most importantly, strict context limits. To address these challenges, we introduce summarization-based context management to training. In specific, it periodically compresses the tool using history by LLM-generated summaries that retain task-relevant information to keep a compact context while enabling the agent to scale beyond the fixed context window. Building on this formulation, we derive a policy gradient representation that seamlessly enables standard LLM RL infrastructures to optimize both tool-use behaviors as well as summarization strategies in an end-to-end fashion. We instantiate this framework with \underline{SU}mmarization augmented \underline{P}olicy \underline{O}ptimization (\texttt{SUPO}), an LLM RL algorithm that enables long-horizon training beyond a fixed context limit. Experiments on interactive function calling and searching tasks demonstrate that \texttt{SUPO} significantly improves the success rate while maintaining the same or even lower working context length compared to baselines. We also demonstrate that for complex searching tasks, \texttt{SUPO} can further improve the evaluation performance when scaling test-time maximum round of summarization beyond that of training time. Our results establish summarization-based context management as a principled and scalable approach for training RL agents beyond a fixed context length limit.

1 Introduction

Long-horizon multi-turn RL is constrained by rapidly growing context, which can impair instruction following, increase rollout costs, and exceed model context limits. The paper proposes jointly optimized summarization-based context management and instantiates it in SUPO.

1 Introduction

  • Long-horizon tool-use RL faces degraded instruction following, higher rollout costs, and context-window limits that prevent training on tasks requiring more calls than fit in one window.Accumulated prompts, outputs, tool observations, and reasoning traces can grow over dozens or hundreds of rounds, creating a scalability barrier without explicit context management.Source: §1 · pp. 1–2

1.1 Our Approach and Contributions

  • The proposed method periodically replaces tool-use history with agent-generated, task-relevant summaries, resetting the working context to the initial prompt plus a compact history representation.The agent jointly optimizes summary generation, learning what information to preserve, abstract, or discard.Source: §1.1 · p. 2
  • The summarization-augmented MDP integrates summary steps into state transitions, and its policy-gradient decomposition lets existing RL infrastructure optimize summarized sub-trajectories.The framework decomposes a long-horizon rollout’s gradient into gradients from several summarized sub-trajectories.Source: §1.1 · p. 2
  • SUPO jointly optimizes tool use and summarization, using trajectory management, group-relative advantage estimation, and overlong-trajectory masking to stabilize optimization and encourage more tool use.Ablations validate the advantage-calculation and overlong-mask designs.Source: §1.1 · p. 2
  • SUPO reports absolute success-rate gains of +3.2% and +14.0% with the same or shorter context, while test-time summary-round scaling improves BrowseComp-Plus performance by up to 7.0%.The experiments cover CodeGym and BrowseComp-Plus; the passage does not assign the two absolute gains to individual environments.Source: §1.1 · p. 2

2 Related Works

Prior RL tool-use methods are bounded by fixed model context windows, while this work frames end-to-end learned summarization as a way to train beyond that limit. It positions the approach among context-compression methods and RL-trained agent memory systems.

2.1 Reinforcement Learning for LLM Multi-Turn Tool-Use

  • Prior multi-turn tool-use RL improves LLM planning and task decomposition but remains bounded by fixed context length; this work uses end-to-end summarization to extend training beyond that limit.The cited prior work’s solvable task difficulty is bounded by the model’s context length.Source: §2.1 · pp. 2–3

2.2 Context Management and Memory in Long-context LLM Agents

  • Long-context agents use working-context compression [9] [10] [19] [22] [26] [28] or external memory [2] [13] [17] [23] [25] [35]; this work uses LLM summarization for compression.Prior methods can discard irrelevant information or condense critical information, but are largely heuristic and not task-specifically fine-tuned or end-to-end optimized.Source: §2.2 · p. 3

2.3 Reinforcement Learning for Agent Memory

  • RL-trained memory systems are the closest comparators, differing in task scope, memory design, and whether training is shown to scale beyond fixed context limits.MemAgent [31] updates working memory by overwriting it with new input segments for long-context question answering, a special case of this framework; MEM1 [36] learns constant-size internal state for question answering and web navigation but concatenates full histories during policy optimization, leaving scaling beyond reliable context windows unknown. Memory-R1 [27] uses a separate RL-trained memory manager and answer agent with external memory, without summarization or demonstrated multi-turn tool-use scaling.Source: §2.3 · p. 3

3 Preliminaries

The section formalizes multi-turn LLM tool-use RL as an MDP, then extends it with summarization-based context management and an end-to-end training objective. It also derives a policy-gradient view intended to fit existing RL infrastructure.

3.1 Standard Modeling of RL Fine-tuning of LLM Multi-turn Tool Use

  • The standard MDP treats accumulated tokens as states, LLM outputs as actions, tool results as observations, and uses a binary verifier reward for the final response.The reward follows RL with verifiable rewards [5].Source: §3.1 · p. 4

3.2 Scaling RL Training via Summarization-based Context Management

  • Summarization triggers when context exceeds threshold L, compressing history into a summary while retaining the initial prompt; working context is bounded by L + 2LA + LO + |vsum|.Here LA and LO are the maximum token counts for one LLM call and tool call, respectively.Source: §3.2 · p. 5
  • The end-to-end objective jointly trains task completion through reasoning and tool use, and task-specific summarization that preserves relevant information while discarding irrelevant context.The rollout can end when the LLM returns a final response, reaches the maximum step H, or reaches the maximum number of summaries S.Source: §3.2 · p. 5
  • The policy gradient sums contributions from complete trajectories, allowing each to be calculated with existing multi-turn RL infrastructure using the initial prompt and previous summary.The framework addresses context management in this setting; deriving correct policy gradients for more general agentic workflows is left as future work [30].Source: §3.2 · p. 6

4 End-to-end RL Training of Agent with Summarization

SUPO adapts GRPO-style policy optimization to train LLM agents beyond their working context limit by managing rollouts through summarization. Its design splits rollouts into trajectories, uses rollout-level advantage estimates, masks overlong rollouts, and controls context growth by discarding the last action-observation pair before summarization.

4.1 Overall Algorithm: SUPO

  • SUPO is a GRPO-style [18] policy optimization algorithm that uses summarization-based context management to train beyond the LLM working context limit.It treats each rollout as multiple complete trajectories so standard RL infrastructure can be reused.Source: §4.1 · p. 6, §4.2 · p. 7

4.2 Algorithm Design Details

  • Each rollout is split into trajectories beginning with the task prompt and the previous trajectory’s summary, and ending with the current trajectory’s summary.This lets SUPO treat the split trajectories as complete trajectories supported by existing RL infrastructure such as VeRL [20].Source: §4.2 · p. 7
  • SUPO assigns the same rollout reward and rollout-group relative advantage to tokens across its split trajectories, improving consistently over trajectory-group relative advantage [15].The shared advantage estimator is calculated from relative advantages within the rollout group.Source: §4.2 · p. 8
  • Overlong masking excludes rollouts that fail to produce a final response before the step limit H or summarization limit S.Without masking, the objective could suppress long rollouts with good summarization strategies and cause summarization patterns to collapse on long-horizon tasks.Source: §4.2 · p. 8
  • Discarding the last action-observation pair when context exceeds threshold L controls the post-summary context to within L+|vsum|+LA.This prevents long observations from making summaries exceed the RL training context length; LA is the maximum action sequence length and is typically much smaller than LRL.Source: §4.2 · p. 8

5 Experiments

Across CodeGym and BrowseComp-Plus, SUPO improves evaluation performance while enabling summarization-based context management; training dynamics and sample summaries show that it supports longer tool-use rollouts and retains task-relevant information. On BrowseComp-Plus, SUPO also reaches its highest reported accuracy when test-time summarization rounds exceed training-time rounds.

5.1 Experiment Setups

  • SUPO uses a 4K working context and 32K effective context on CodeGym, and 64K working context and 192K effective context on BrowseComp-Plus.The CodeGym setup uses 12,800 training problems and 128 evaluation problems; BrowseComp-Plus uses 730 training questions and 100 held-out evaluation questions.Source: §5.1 · p. 9

5.2.1 Training and Evaluation Results of SUPO

  • 53% is SUPO’s highest BrowseComp-Plus evaluation score, a 14% improvement over GRPO; on CodeGym, SUPO scores higher than GRPO at the same 32K effective context length.Table 1 covers GRPO, SUPO, and ablations on both tasks; the prose reports these comparisons.Source: Table 1 · p. 10, §5.2.1 · p. 10

5.2.2 Further Analysis of SUPO

  • Sample summaries show SUPO retaining continuation-critical information: CodeGym array indices and BrowseComp-Plus answer details from article 18432.Before SUPO, the CodeGym summary retained comparison counts but not the current index; in the BrowseComp-Plus example, only the post-SUPO summary retained key information for the answer.Source: §5.2.2 · p. 12, Table 2
  • During training, CodeGym’s summarization rate rises and BrowseComp-Plus’s stays near 1, while conditional success on summarized rollouts increases on both tasks.Together, these dynamics indicate joint improvement in tool calling and summarization.Source: §5.2.2 · p. 10, Figure 3 · p. 11
  • Without overlong masking, summarization collapses on both tasks: more rollouts finish in one trajectory and conditional success falls to 0 during training.The authors identify overlong masking as effective in preventing this failure mode.Source: §5.2.2 · pp. 11–12, Figure 3 · p. 11
  • SUPO permits and incentivizes up to 3× as many tool calls as GRPO on BrowseComp-Plus, while GRPO’s average tool calls decline and SUPO without overlong masking drops quickly.The comparison concerns training with 64K working context for GRPO and 64K working / 192K effective context for SUPO variants.Source: Figure 4 · p. 12, §5.2.2 · p. 12
  • The standard advantage estimator (3) yields consistently better results than estimator (4), which the authors conjecture weakens the relative advantage of long, successful trajectories.They also report worse test-time summarization-round scaling for models trained with estimator (4).Source: §5.2.2 · p. 12

5.3 Scaling beyond Trajectory Number during Training

  • 60.0% is the highest final BrowseComp-Plus accuracy among the evaluated algorithms when scaling test-time summarization rounds.The evaluation varies S across {1, 2, 5, 11, 23}; rollout with summarization-based context management also improves accuracy without end-to-end summarization training.Source: §5.3 · pp. 12–13, Figure 5 · p. 12

6 Conclusions and Future Works · Appendix

The paper presents SUPO, an RL framework that integrates summarization-based context management to train LLMs beyond context-length constraints. It reports strong performance against vanilla multi-turn RL on CodeGym and BrowseComp-Plus and outlines several future research directions.

6 Conclusions and Future Works

  • SUPO demonstrates strong empirical performance on CodeGym and BrowseComp-Plus compared with a vanilla multi-turn RL baseline.The passage provides no numerical results.Source: §6 · p. 13
  • The framework integrates summarization as a component of RL training for fine-tuning LLMs.Source: §6 · p. 13
  • Formulating summarization-based context management as an MDP yields a policy-gradient formulation that lets standard RL infrastructure scale beyond context-length constraints.Source: §6 · p. 13
  • Future work includes refining advantage estimation with critics.Source: §6 · p. 13
  • The authors identify integrating external memory modules as a future direction.Source: §6 · p. 13
  • They also propose jointly optimizing summarization strategies across diverse domains.Source: §6 · p. 13

A Proofs for Section 3 · A.1 Proof of Theorem 3.2

The proof of Theorem 3.2 derives a policy-gradient expression by representing the rollout through its terminal reward and decomposing it at states that trigger summarization. It concludes with a gradient form that includes the policy action taken with the generated summary in context.

A.1 Proof of Theorem 3.2

  • The proof sets the horizon to H and pads any rollout that ends early by repeating its terminal state.This ensures the reward can be represented at step H.Source: §A.1 · p. 16
  • The rollout probability is paired with the terminal reward R(sH, aH) in the objective’s trajectory expression.Source: §A.1 · p. 16
  • Differentiating J yields a score-function term involving the derivative of the log rollout probability, weighted by the terminal reward.Source: §A.1 · p. 17
  • The rollout probability is factored beginning with the initial-state distribution μ(s1).Source: §A.1 · p. 17
  • For each rollout, the proof identifies indices hi where the context is overlong and contains the summary, marking states to be summarized.These indices are defined by the summarization thresholds and transition kernel.Source: §A.1 · p. 17
  • The summarization indices divide the rollout into I + 1 complete trajectories, whose states share prefixes that are prefixes of the last state shi.Source: §A.1 · p. 17
  • The resulting gradient expression includes the policy-gradient term for the action at summarization index hi, conditioned on the context containing vsum.The proof then concludes Theorem 3.2.Source: §A.1 · p. 17

B More Algorithm and Experiment Details · B.1 Sample Problems · B.2 Summarization Instructions

The appendix presents sample CodeGym and BrowseComp-Plus problems and gives separate summarization instructions for each task type.

B More Algorithm and Experiment Details

  • The appendix’s examples cover function-calling and web-research tasks, with system prompts used across the paper’s experiments.It presents samples for both CodeGym and BrowseComp-Plus.Source: §B.1 · pp. 18, 20

B.1 Sample Problems

  • In CodeGym’s first example, the agent must count student pairs with increasing heights using provided functions, without writing code or calling more than one function per step.The task requires submitting the inferred count through the done function.Source: §B.1 · pp. 18–19
  • The second CodeGym example asks the agent to find the largest positive day-to-day visitor increase, returning zero if no adjacent-day increase occurs.The provided functions calculate adjacent-day changes and find the maximum change.Source: §B.1 · pp. 19–20
  • BrowseComp-Plus prompts frame the agent as a rigorous research assistant that searches and opens sources, then submits a concise, evidence-supported answer.The instructions emphasize objectivity, thoroughness, interleaved tool use, verification, and citing document IDs in the final explanation.Source: §B.1 · pp. 20, 22
  • One BrowseComp-Plus sample asks the agent to identify a person by connecting clues about a 2019 mass-spectrometry article, a Ghanaian university role, and aid to four universities.The prompt directs the agent to deconstruct the query, plan searches, verify claims, and construct a supported final report.Source: §B.1 · pp. 21–22
  • A second BrowseComp-Plus sample asks for a historical place matching specified construction dates, wall thickness, location, acquisition, and damage-history constraints.The prompt also constrains the acquisition-era president’s birth dates.Source: §B.1 · p. 23

B.2 Summarization Instructions

  • The CodeGym summarization prompt asks for a history summary that retains important information needed to continue solving the task and prohibits function calls during summarization.The generated summary is placed inside <summary> tags.Source: §B.2 · p. 23
  • The BrowseComp-Plus summarization template preserves the original query, verification-checklist status, sourced findings, discrepancies, leads, dead ends, and the next action.The resulting summary is intended to be the sole context for continuing the task.Source: §B.2 · p. 23

C More Experiment Results · C.1 Summarization Patterns

The section presents CodeGym and BrowseComp-Plus rollouts to illustrate summarization patterns learned through end-to-end training and how they interact with tool-use history.

C.1 Summarization Patterns

  • CodeGym and BrowseComp-Plus rollouts showcase learned summarization patterns and their interplay with tool-use history.Source: §C.1 · p. 24

C.1.1 CodeGym

In the CodeGym example, the agent’s score rises from 0 before training to 1 after training, and the post-training trajectories show summaries carrying task progress forward with a shorter token budget.

C.1.1 CodeGym

  • The agent scores 0 before training and 1 after training on CodeGym Sample Problem 1.The passage also calls attention to the summarization pattern highlighted in orange.Source: §C.1.1 · p. 24
  • The displayed trajectories use 4K/3K tokens before training and 4K/2K tokens after training.Source: §C.1.1 · pp. 24, 27
  • After training, a trajectory summary records the task, array, and progress at count 41, identifies (5, 7) as the next pair, and the trajectory ends with answer 57.The corresponding pre-training trajectory summary records a count of 28 and later ends with answer 43.Source: §C.1.1 · pp. 25–28

C.1.2 BrowseComp-Plus

This section presents BrowseComp-Plus sample problems and trajectories, showing score changes before and after training and how summaries preserve search progress while noting unresolved information.

C.1.2 BrowseComp-Plus

  • Both sample problems moved from score 0 before training to score 1 afterward.The section presents rollout trajectories from before and after training for Sample Problems 1 and 2.Source: §C.1.2 · p. 29
  • At training step 0, a generated summary retained the article, department-head, and university-support clues, while verification checks remained pending.The summary described the World Bank Group as having 189 member countries in 2012, versus the query’s 187-country clue.Source: §C.1.2 · pp. 30–31
  • At training step 115, a trajectory identified a May 2019 Analytical Chemistry article on MALDI MS imaging, but the department-head connection remained unresolved.The summary names the article and authors, while listing the Ghanaian university role as a lead to investigate.Source: §C.1.2 · p. 35
  • The example compares one 48K-token trajectory before training with three trajectories of 60K, 50K, and 41K tokens after training.The passages do not establish these as a controlled context-length comparison.Source: §C.1.2 · pp. 37, 39
  • The carried summary recorded Ahsan Manzil’s location, construction dates, acquisition, and disaster damage as verified, while leaving wall thickness pending.It also identified Hussain Muhammad Ershad’s 1920 birth year and directed the next search toward specific wall-thickness measurements.Source: §C.1.2 · pp. 40–42
  • At training step 115, the agent answered Ahsan Manzil with score 1, versus Fort St. George with score 0 at step 0.The final answer says Ahsan Manzil meets the criteria, but acknowledges that its wall thickness is not explicitly documented and relies on an architectural generalization.Source: §C.1.2 · pp. 37, 38, 39, 42

C.2 Experiments for Scaling Trajectory Number

This subsection presents evaluation results for scaling the test-time number of trajectories.

C.2 Experiments for Scaling Trajectory Number

  • The full results evaluate scaling the test-time number of trajectories.Table 3 presents the evaluation results for this experiment.Source: §C.2 · p. 43, Table 3 · p. 43

Related papers

Loading 2510.06727v1…