PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-Evolving Acting and Forecasting for Language Agents

Haofei Yu, Yixing Chen, Yining Zhao, Xuyan Ye, Yuxiang Liu, Zhigang Hua, Yan Xie, Shuang Yang, Jiaxuan You

arXiv:2610.20597v2cs.AIcs.LG

TL;DR

Long-horizon agent training is costly, and standard RL often leaves intermediate trajectory supervision unused; existing value and observation targets are difficult to construct effectively. CoPE forecasts multi-step actions from successful on-policy trajectories with a shared policy backbone, and the paper reports improvements across five benchmarks alongside greater sample efficiency on ScienceWorld.

  • Problem

    Long-horizon agent rollouts are costly, while terminal scalar rewards leave intermediate supervision underused and existing value or observation targets are difficult to construct effectively.

    Source: §1 · p. 1

  • Method

    CoPE jointly trains a shared policy backbone to act and forecast multi-step action chunks from successful on-policy trajectories.

    Source: §1 · p. 2

  • Results

    Across five benchmarks, CoPE improves aggregate performance over RL baselines, and on ScienceWorld is up to 1.9× more sample-efficient than ECHO and 2.0× than GRPO.

    Source: §1 · p. 2, §7 · p. 7, §10 · p. 10

  • Takeaways & Limitations

    CoPE combines action forecasting and policy optimization in one backbone, improving sample efficiency with little computational overhead and no inference cost over a GRPO-trained policy.

    Source: §10 · p. 10

    Success-conditioned episodes may over-represent easy tasks because success depends on both the task and the agent’s choices.

    Source: §8 · p. 9

Abstract

from arXiv · show

Training language agents is bottlenecked by the high cost of long-horizon rollouts, yet standard agentic reinforcement learning (RL) learns only from a terminal scalar reward, leaving the supervision in intermediate steps largely unused. Prior work exploits this supervision through value targets that estimate per-step success probabilities or observation targets that fit environment response. Both are hard to construct well: (1) value targets come from the same single scalar signal for each step in the trajectory, and (2) observation targets, such as web pages and terminal logs, are long and noisy and reach only one step ahead. We propose CoPE, which adds a third kind of multi-step action targets. On successful on-policy trajectories, CoPE trains the policy backbone to generate the action sequences that follows each history. Each action is also supervised when forecast from the preceding histories, forming a temporal ensemble that the shared backbone absorbs during training, and this enables acting and forecasting to co-evolve. The policy still acts one step at a time at inference, with no overhead. Across five benchmarks (ALFWorld, ScienceWorld, WebShop, $\tau^2$-bench, and AppWorld), CoPE (1) achieves stronger aggregate performance than GRPO and RL baselines with predictive auxiliary targets across simulated environments, while improving overall performance on both realistic applications; (2) delivers up to $1.9\times$ and $2.0\times$ higher sample efficiency than ECHO and GRPO, respectively, on ScienceWorld; and (3) achieves these gains with a single policy backbone, without additional models, separate pretraining, or inference overhead. The codebase is available at https://github.com/ulab-uiuc/CoPE.

1 INTRODUCTION

Long-horizon agent training is costly, while terminal rewards leave intermediate supervision underused. COPE adds successful-trajectory action forecasting, and the paper reports gains across five benchmarks.

1 INTRODUCTION

  • Long-horizon agent training is computationally expensive because rollouts require many interaction steps and tool calls, making limited interactions valuable.Standard agentic RL often learns only from a scalar reward at trajectory end.Source: §1 · p. 1
  • COPE predicts multi-step action sequences from successful on-policy trajectories, using step-specific, decision-relevant targets that implicitly ensemble forecasts across histories.The shared backbone still acts one step at a time at inference, without overhead.Source: §1 · p. 2, Figure 1 · p. 2
  • Across five benchmarks, COPE outperforms GRPO and observation-prediction baselines on aggregate simulated-environment metrics and improves overall metrics on both realistic applications.The paper also reports a negative result on the Airline domain of τ 2-bench.Source: §1 · p. 2

2 RELATED WORKS

  • Value-target methods reuse trajectory outcomes or add step-level credit assignment, while observation-target methods predict environment feedback, typically for one transition.COPE combines success filtering with action-space prediction rather than reconstructing observations.Source: §1 · p. 1, §2 · p. 3

3 PRELIMINARIES

  • In the described sparse-reward setting, GRPO assigns every action in a trajectory the same normalized advantage derived from its terminal outcome.Consequently, the intermediate decisions receive no additional supervision about how the trajectory unfolded.Source: §3 · p. 3

4 EXTENDING POLICY LEARNING ACROSS FUTURE HORIZONS

With binary outcome rewards, standard policy gradients supervise each history using only its immediate action. The section defines a successful-rollout action target and extends policy learning to forecast action chunks across future horizons.

4 EXTENDING POLICY LEARNING ACROSS FUTURE HORIZONS

  • Standard policy gradients supervise each history with only its immediate action, so they provide no direct signal to plan further ahead.The idealized GRPO update fits the single-step hindsight marginal, not the later actions realized in successful trajectories.Source: §4 · pp. 3–4
  • The hindsight action chunk is the distribution of the next K + 1 actions under successful rollouts, conditioned on history and marginalized over intermediate observations.At K = 0, it reduces to the single-step action distribution.Source: §4 · p. 4
  • Action forecasting trains an autoregressive forecaster to match the multi-step hindsight chunk, using action sequences observed after histories in successful rollouts as training targets.The shared parameters make this objective train the policy representation to encode the multi-step plan.Source: §4 · p. 4

5 COPE: CO-EVOLVING ACTING AND FORECASTING

COPE jointly trains a language agent to act and forecast future actions using one shared backbone. Forecasting uses successful on-policy action chunks, and forecasts from earlier histories are implicitly ensembled into the acting policy.

5.1 SHARING PARAMETERS FOR ACTING AND FORECASTING

  • The acting policy and forecaster share one LLM backbone, with a forecasting instruction distinguishing the action-chunk distribution from the next-action policy distribution.Sharing avoids a separate model and writes multi-step structure into the representation used for acting.Source: §5.1 · p. 5

5.2 MULTI-STEP ACTION FORECASTING LOSS

  • The forecaster is trained by maximum likelihood on action chunks from successful on-policy rollouts, which are samples from the success-conditioned hindsight target.Failed trajectories are excluded because fitting them would instead teach the unconditional rollout distribution.Source: §5.2 · p. 5

5.3 CO-EVOLUTION OF ACTING AND FORECASTING

  • Each successful action receives forecast supervision from up to K preceding histories, and the shared backbone implicitly ensembles those forecasts into the acting policy.Forecasting also depends on acting to collect successful rollouts, with its target depending on the current policy.Source: §5.3 · p. 6

5 COPE: CO-EVOLVING ACTING AND FORECASTING

  • COPE jointly optimizes GRPO and action forecasting losses, then uses the acting policy alone to produce one action per step at inference.The forecasting objective adds no inference cost over a GRPO-trained policy.Source: §5 · p. 5, §5.3 · p. 6

6 EXPERIMENTAL SETTINGS

The experiments evaluate CoPE across five long-horizon benchmarks, comparing it with prompting and RL baselines, including methods with auxiliary prediction losses. CoPE improves aggregate performance and sample efficiency, with a modest added training pass on ScienceWorld.

6 EXPERIMENTAL SETTINGS

  • The five benchmarks cover household tasks, science experiments, online shopping, customer service, and API-call composition across consumer apps.The experiments use Qwen2.5-7B-Instruct for the simulated environments and τ 2-bench, and Qwen2.5-14B-Instruct for AppWorld.Source: §6 · p. 6
  • Simulated-environment comparisons include prompting, RL without auxiliary losses, and RL with next-state or next-observation prediction losses.On τ 2-bench and AppWorld, the study also includes published baseline numbers; its direct AppWorld comparison is GRPO and CoPE under a shared setup.Source: §6 · p. 6

7 EXPERIMENTAL RESULTS

  • CoPE improves benchmark-level aggregate metrics over GRPO across simulated environments and realistic applications.The reported benchmarks are ALFWorld, ScienceWorld, WebShop, τ 2-bench, and AppWorld.Source: §7 · pp. 6–7
  • COPE raises ALFWorld performance by 5.0 points over GRPO, WebShop success by 12.5 points to 77.0%, and ScienceWorld success by 8.0 points to 72.5%.COPE also achieves the best performance among RL-based methods on ScienceWorld.Source: §7 · p. 7
  • On τ 2-bench, COPE improves overall success from 27.5% to 28.7%, driven by an approximately 12-point Retail gain despite an Airline drop.On AppWorld, success increases from 41.67% to 55.36% on the normal split and from 18.71% to 22.54% on the challenge split.Source: §7 · p. 7, Table 1 · p. 7, Table 2 · p. 7
  • COPE exceeds the stronger observation-based baseline by 1.0 point on ALFWorld, 4.4 average-score points and 5.5 success-rate points on WebShop, and 5.0 success-rate points on ScienceWorld.These gains are not uniform across ALFWorld subtasks: ECHO or RWML performs better on Cool and Pick2.Source: §7 · p. 7
  • COPE is 1.6× and 2.0× more sample-efficient than GRPO on AppWorld and ScienceWorld, respectively.On ScienceWorld, ECHO requires 53 and 114 steps to reach equivalent performance milestones, making CoPE 1.7× to 1.9× faster.Source: §7 · p. 7
  • On ScienceWorld, the supervised pass adds 11% time per training step on average, while shorter rollouts reduce end-to-end step time by 9% versus GRPO.The direct added cost scales with the batch success rate.Source: §7 · p. 7

8 DISCUSSIONS

The discussion examines how forecasting can improve acting through shared parameters and how multi-step delays compare with single-step targets. It also identifies success-conditioned over-representation as a boundary on performance in τ 2-bench.

Evidence for implicit temporal ensembling

  • Explicit forecasting improves GRPO but adds nothing to COPE, whose policy already contains information that GRPO gains only with a separate forecaster and extra test-time forward passes.This supports the forecasting-to-acting half of co-evolution through implicit ensembling via shared parameters.Source: §8 · pp. 7–8

A single delay versus an ensemble over delays

  • The discussion frames CoPE’s delay-based forecasting as related to action chunking, where delayed predictors can provide benefits and ensembles over randomized delays match chunking.The cited work describes a single delay as helping but falling short on harder tasks.Source: §8 · p. 8

Delay 0 carries little information about success

  • Delay-0 targets can preserve detours that later recovery makes compatible with success, so direct delay-0 supervision speeds early training but later hurts it.The recovery that determines success appears only in later-delay targets.Source: §8 · p. 8

A single delay versus an ensemble over delays

  • COPE trains each action at every delay up to K and gains 8.0 success-rate points over GRPO, compared with +2.0 for K = 0 and +5.0 for invalid-action filtering.The invalid-action filter uses an environment-specific signal and covers only detours the environment rejects; COPE needs no such signal.Source: §8 · p. 8, Table 5 · p. 9

Why action forecasting loss is conditioned on success

  • Success conditioning aligns forecasting with the RL objective, whereas unconditioned imitation of average or failed trajectories creates objective conflict and leads to zero reward after the first few training steps.The forecaster learns behavior in episodes that eventually succeed, which may over-represent easy tasks.Source: §8 · pp. 8–9

Mixed results on τ 2-bench

  • On τ 2-bench, COPE gains on Retail but drops on Airline because successful episodes over-represent easy refusal tasks, whose histories resemble tasks requiring action.The database remains unchanged in 38% of Airline tasks versus 4.4% in Retail; early success is 52% for refusal tasks versus 8.9% for tasks requiring database changes.Source: §8 · p. 9

9 ABLATION STUDIES

The ablations show COPE’s gains across two backbones and identify a two-step forecast horizon as the strongest tested setting. Forecasting-loss weights improve over GRPO, while normalization reduces gradient spikes.

Ablation on forecasting horizon K

  • K = 2 performs best among the tested forecasting horizons, while K = 4 declines slightly as noise from distant predictions begins to dominate.Every K ≥1 setting outperforms GRPO and the K = 0 baseline.Source: §9 · p. 9, Figure 6 · p. 10

Ablation on forecasting coefficient λ

  • Every tested forecasting-loss coefficient improves on GRPO, with λ = 10−2 achieving the highest score.Larger coefficients accelerate early gains; smaller ones follow a similar but delayed trajectory.Source: §9 · p. 9, Figure 7 · p. 10

Ablation on group normalization

  • Averaging the forecasting loss over successful rollouts removes its direct dependence on success count and reduces gradient-norm spikes.Without normalization, the forecasting gradient scales with the number of successful rollouts, shifting its balance with the RL objective.Source: §9 · pp. 9–10, Figure 8 · p. 10

Ablation on model backbone

  • COPE improves score and success rate on WebShop and ScienceWorld under both Qwen2.5-7B-Instruct and Qwen3-8B backbones.The results indicate that the gains are not tied to a particular model generation.Source: §9 · p. 10

10 CONCLUSION

COPE trains a language agent to forecast its own future actions alongside acting, using success-filtered action chunks across five benchmarks. The paper reports improved aggregate results over RL baselines, with little computational overhead, and frames the work as a research algorithm rather than a deployed system.

10 CONCLUSION

  • COPE trains a language agent to forecast its own future actions alongside acting, using success-filtered action chunks up to K steps ahead.Across five benchmarks, it improves aggregate results over RL baselines and sample efficiency with little computational overhead.Source: §10 · p. 10

REPRODUCIBILITY STATEMENT

  • The manuscript describes the algorithm, tasks, models, baselines, evaluation protocols, licenses, dependencies, and documented configurations, and says complete configurations and training code will be released upon publication.Evaluation metrics are defined in Appendix B.4.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

  • COPE is a training algorithm, not a deployed system, and its societal impact is largely mediated by downstream practitioners.The authors note potential misuse of more capable autonomous agents and recommend safety evaluation, monitoring, and human oversight in higher-stakes domains.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

  • Generative AI tools assisted with method development, proof writing, experimental design feedback, result interpretation, code implementation, and manuscript preparation.The authors reviewed AI-assisted derivations, code, results, and citations and take responsibility for the final content.Source: AI USE STATEMENT · p. 11

A ARTIFACT DETAILS

The appendices document model licenses, software, benchmark splits, baselines, training settings, reward criteria, and evaluation metrics. Together, they specify the experimental configurations and the scope of the reported measurements.

A.1 MODEL LICENSE

  • The evaluated models include proprietary GPT and Gemini variants and Apache 2.0-licensed Qwen2.5 and Qwen3 models.The listed models are GPT-5-mini, GPT-5.5, GPT-4o-mini, Gemini-2.5-flash-lite, Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, and Qwen3-8B.Source: §A.1 · p. 16

A.2 SOFTWARE VERSIONS

  • The simulated-environment setup builds on AgentGym-RL1, adds adapters for AppWorld and τ 2-bench, and uses verl2 v0.2.0.post2.AgentGym-RL1 provides unified task configurations for ALFWorld, WebShop, and SciWorld.Source: §A.2 · p. 16

A.3 DATASET STATISTICS

  • Measured splits include 200-task test sets for ALFWorld, WebShop, and ScienceWorld; AppWorld uses separate test_normal (168-task) and test_challenge (417-task) splits.WebShop has 6,910 generated instructions, with 3,930 used for training and 200 for testing; the sets are disjoint and cover 59.8% of the pool. AppWorld’s challenge split measures transfer to unseen app combinations.Source: Table 7 · p. 16, §A.3 · pp. 16–17

B.1 BASELINE METHODS

  • The baselines span prompting, rejection-sampling fine-tuning, online RL, world-model-based methods, observation prediction, and entropy-modulated policy gradients.Published G2PO AppWorld numbers are not controlled comparisons because their task splits and evaluation and training setups differ.Source: §B.1 · pp. 17–18

B.2 TRAINING HYPERPARAMETERS

  • COPE uses GRPO settings with an additional action-forecasting objective, while τ 2 targets are extracted from assistant replies and placed in separate assistant turns.A sequence is discarded under skip_no_call only when none of its K actions contains a tool call.Source: Table 9 · p. 18, §B.2 · pp. 18–19, Table 10 · p. 19

B.3 TRAINING REWARDS AND SUCCESS LABELS

  • The binary terminal reward Y = 1 determines which trajectories enter the action-forecasting loss, with task-specific completion criteria for each benchmark.For example, WebShop requires a raw score of exactly 1.0, while AppWorld requires all state-based task tests to pass.Source: §B.3 · p. 20

B.4 EVALUATION METRICS

  • The policy and action-forecasting objectives use binary success, while graded environment scores are reported separately alongside success rates.Overall results are count-weighted averages except for τ 2-Bench, where retail, airline, and telecom domain results are averaged equally.Source: §B.4 · p. 20

C ENVIRONMENT PROMPTS

The appendix specifies how the environment prompts and chat histories are arranged for the agent. It gives standing instructions and action formats for ALFWorld, WebShop, and SciWorld, with each environment’s first observation arriving within the conversation.

C ENVIRONMENT PROMPTS

  • Each episode is one chat sequence: a standing instruction and fixed assistant acknowledgement precede the first observation, after which observation and action turns alternate.Completed action-observation pairs are appended in place to the history context.Source: C ENVIRONMENT PROMPTS · pp. 20–21

C.1 ALFWORLD

  • ALFWorld asks the agent to complete a household task by choosing from available actions and formatting output as Thought/Action or Action.The prompt says an action outside the available list is illegal, and “Nothing happened” means the previous action was invalid.Source: §C.1 · p. 21
  • ALFWorld’s fixed assistant acknowledgement is “OK. I'll follow your instructions and try my best to solve the task.”Source: §C.1 · p. 21

C.2 WEBSHOP

  • WebShop instructs the agent to search or click available options and format actions as search[keywords] or click[value].The per-episode shopping goal is not part of the standing instruction; it arrives in the first observation.Source: §C.2 · p. 21, §C.3 · p. 22

C.3 SCIWORLD

  • SciWorld’s standing prompt lists available environment actions and asks the agent to respond with a Thought/Action format.The per-episode goal arrives in the first observation, rather than the standing instruction.Source: §C.3 · p. 22
  • SciWorld’s fixed assistant acknowledgement matches ALFWorld’s: “OK. I'll follow your instructions and try my best to solve the task.”Source: §C.3 · p. 22

C.4 APPWORLD

The AppWorld example shows an agent using documented app APIs in a multi-step Python REPL conversation to answer a Spotify playlist-count task. It retrieves all playlist pages, counts them, and returns 23.

C.4 APPWORLD

  • AppWorld’s standing instruction embeds a worked example, while the actual task arrives in the first observation.The instruction’s ASSISTANT and USER lines are static demonstration text, not dialogue history.Source: §C.4 · pp. 22–23
  • The agent uses a multi-step Python REPL conversation to interact with apps through their associated APIs.Each code execution returns output for use in subsequent calls.Source: §C.4 · pp. 22–23
  • The playlist-library API supports pagination, so the example loops through pages before counting the collected playlists.Its documented page limit defaults to five results per page.Source: §C.4 · pp. 25–27
  • The example answers the playlist-count task with 23 playlists.The agent counts the collected playlist records and returns the number as the task answer.Source: §C.4 · p. 27
  • The agent consults API documentation, retrieves Spotify credentials through the supervisor app, and logs in before requesting the playlist library.The example looks up the supervisor email and Spotify password rather than assuming login values.Source: §C.4 · pp. 25, 28

C.5 τ 2-BENCH

The τ²-bench prompt retains the domain-specific policy and native tool-calling format. The assistant either responds to the customer or emits tool calls, with only the first tool-call block executed.

C.5 τ 2-BENCH

  • The prompt supplies the domain policy and tools, and asks the agent to help the user by either sending a message or making a tool call.The instructions prohibit doing both in one turn.Source: §C.5 · pp. 28–29
  • The conversation begins with the customer’s message and includes the dialogue history in the user-message template.The assistant turn is represented as a customer message or an executable tool-call response.Source: §C.5 · p. 29
  • The assistant responds to the customer or emits a tool-call block, and only the first block is executed when several are emitted.Tool calls use a JSON object inside the specified XML tags.Source: §C.5 · p. 29

C.5.1 WHAT FILLS {POLICY} AND {T O O L S}

The τ²-bench setup inserts benchmark policy documents and advertises the native tools. For readability, the paper gives a compact retail-tool signature summary rather than reproducing the full native schemas.

C.5.1 WHAT FILLS {POLICY} AND {T O O L S}

  • The policy documents come from τ²-bench and are inserted unmodified into the evaluated domains.The passage introduces the measured domains but does not specify their values in the supplied text.Source: §C.5.1 · p. 29
  • The retail-tool list is summarized at the signature level for readability, rather than copied as a literal native-prompt schema.The actual system message renders one OpenAI JSON schema per tool.Source: §C.5.1 · p. 29
  • The retail tool examples include lookup, order-management, and user-address functions with explicit confirmation requirements for several changes.Examples include canceling a pending order, modifying an address, and returning delivered items.Source: §C.5.1 · pp. 29–30
  • Some tool operations have specific limits: cancellation applies only to pending orders, and an agent can return or exchange a delivered order only once.The cancellation tool also requires the agent to explain the change and obtain user confirmation.Source: §C.5.1 · p. 29

C.6 AUXILIARY PROMPTS (ACTION FORECASTING)

The action-forecasting objective uses a separate sequence that truncates the dialogue at the current history and appends a planning request. It targets the realized next actions, with serialization adapted for τ²-bench.

C.6 AUXILIARY PROMPTS (ACTION FORECASTING)

  • The forecasting sequence truncates the dialogue at the current history, appends a planning turn, and targets the realized next-K actions.The horizon is represented by {k}, and {history} denotes the dialogue up to the truncation point.Source: §C.6 · p. 30
  • For ALFWorld, WebShop, SciWorld, and AppWorld, the sequence retains the user instruction and fixed assistant acknowledgement.The planning request asks for the next actions, starting with the action taken now.Source: §C.6 · p. 30
  • τ²-bench retains its native system message and role-structured dialogue, without an acknowledgement turn or Action: field.Its target places each action in a separate assistant turn, separated by a fixed Next action? user message; loss is computed only on assistant action turns.Source: §C.6 · pp. 30–31

Related papers

Loading 2610.20597v2…