Source-linked AI summary
EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning
Zishan Xu, Zhiyuan Yao, Yuxin Chen, Yifu Guo, Zhengxi Lu, Yuquan Lu, Jinyang Huang, Yan Xu, Yasheng Wang, Weinan Zhang, Xingshan Zeng, Weiwen Liu
TL;DR
Training long-horizon tool-use agents depends on costly, difficult-to-verify environments or externally grounded simulators. EnvACE replaces external interaction during training with policy-internalized world rehearsal, outperforming environment-scaling baselines across diverse benchmarks and improving performance with private test-time rehearsal.
Problem
Existing agent training remains dependent on costly executable environments or potentially inaccurate simulators requiring real-environment grounding.
Method
EnvACE trains one shared policy to alternate between issuing tool calls and rehearsing their induced environment responses, jointly optimized with task-success rewards.
Results
EnvACE consistently outperforms strong baselines across BFCL-v4, τ 2-Bench, VitaBench, and FinMCP-Bench, achieving a 32.91% overall score across three benchmarks.
Takeaways & Limitations
World rehearsal internalizes environment dynamics in the policy and enables additional gains from private rehearsal before real execution without extra external interaction.
Takeaways & Limitations
Evaluation is limited to models up to 8B parameters and primarily tool-interactive tasks, leaving larger models and broader agentic settings untested.
Abstract
from arXiv · showhide
Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground. We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal. The policy alternates between acting and rehearsal: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response. Both roles are jointly optimized end-to-end using task-success rewards. Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision making. Across BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation. Controlled studies further show that world rehearsal consistently improves policy learning across model scales. At test time, the internalized world model enables private rehearsal before committed execution, yielding further gains under a moderate rehearsal budget without additional external interaction. Our findings establish world rehearsal as a new path toward scaling LLM agent training beyond the constraints of external environments. Our code is publicly available at https://github.com/Within-yao/EnvACE.
1 INTRODUCTION
EnvACE replaces externally supplied environment responses during agentic RL training with world rehearsal, jointly training one policy to act and generate the responses induced by its actions. This internalized world model improves transferable benchmark performance and enables additional gains from private rehearsal at test time.
- Motivation: Long-horizon tool-using agents must combine dialogue, tool use, environment feedback, and adaptation across extended interactions.
- Limitations of existing paradigms: External environment synthesis is costly and difficult to verify at scale, while external simulators leave environment modeling outside the acting policy.
- World rehearsal: World rehearsal internalizes environment dynamics within the policy without querying an external environment, supporting more generalizable decision making.
- EnvACE: EnvACE interleaves tool-call generation with rehearsal of the induced environment response, using one shared policy jointly optimized for acting and rehearsal with task-success rewards.
- Results: EnvACE consistently outperforms strong baselines across BFCL-v4, τ 2-Bench, VitaBench, and FinMCP-Bench, while controlled analyses show benefits across model scales and additional test-time gains.
2 RELATED WORK
Prior work has expanded long-horizon language-agent benchmarks and strengthened reinforcement-learning methods for reasoning, search, and tool use. Interactive-agent training nevertheless depends on environments that map actions to responses and outcomes, motivating executable-environment synthesis and language-model-based feedback simulation.
- Long-horizon benchmarks increasingly evaluate language agents through interactions with realistic real-world environments.
- Recent reinforcement learning improves language-model reasoning while extending toward efficient search and tool-augmented reasoning.
- Interactive agent training requires environments that map actions to responses and task outcomes, but fixed executable environments are costly to construct and scale.
- To reduce this burden, recent approaches synthesize executable tools, databases, tasks, and evaluators or simulate environment feedback with language models.
3 PRELIMINARIES
The paper formulates tool-interactive tasks as finite-horizon POMDPs in which policies generate actions and environments return observations and rewards. EnvACE revises this interaction boundary by assigning observation generation to a rehearsal role of the policy.
- Tool-interactive tasks are modeled as finite-horizon POMDPs with states, structured tool-call or answer actions, observations, transition dynamics, and rewards.
- At each step, the policy conditions on interaction history to generate an action, receives a tool output when applicable, and updates the history with the action-observation pair.Rollouts terminate as trajectories containing the instruction and successive actions and observations.
- Trajectory-level scalar rewards evaluate task performance, and Group Relative Policy Optimization estimates advantages by normalizing rewards against rollouts for the same instruction.
- EnvACE assigns observation generation to a rehearsal role of the policy instead of leaving subsequent-decision conditioning entirely to an external environment.
4 METHOD
EnvACE replaces external environment interaction during training with an act–rehearse loop in which a shared policy generates both actions and induced responses. Role-wise optimization internalizes action–response dynamics, while private rehearsal at test time supports refinement before committed execution.
- World rehearsal: During training, EnvACE alternates acting and rehearsal, appending each policy-generated environment response to the history so subsequent actions proceed without an external environment.The shared policy generates an environment-facing action, then produces the corresponding response conditioned on the history and action.
- World rehearsal: Repeated rehearsal absorbs action–response relationships into the shared policy parameters, enabling an internal agent world model that conditions later decisions on rehearsed responses.Acting and rehearsal continue until termination, with generated responses appended to the interaction history.
- Role-wise GRPO optimization: EnvACE computes separate reward baselines and role-wise advantages for ACT and REHEARSE outputs, while jointly updating the shared policy with trajectory-level rewards.Each output in a rollout inherits its trajectory reward, but advantages are computed relative to outputs from the same role across sampled rollouts.
- Test-time scaling: At inference, EnvACE performs N private rehearsal attempts from the same task context, evaluates each imagined trajectory, and obtains assessment and revision feedback.Attempts can run independently in parallel or sequentially, with later attempts observing prior trajectories and feedback in the sequential mode.
- Test-time scaling: EnvACE summarizes private rehearsals and self-evaluations into memory m_x, then conditions one committed external execution on that memory without altering the environment during rehearsal.The rehearsal attempts remain private to the agent and are used only before committed execution.
5 EXPERIMENTS
Experiments show that EnvACE performs strongly across diverse agentic benchmarks, improves policy learning through world rehearsal and parameter sharing, and scales with model capacity. At test time, private rehearsal further improves performance without additional external interaction.
- Main Results: EnvACE achieves a 32.91% Overall score across BFCL V4, τ 2-Bench, and VitaBench, surpassing EnvScaler-8B and AWM-14B by 0.99% and 0.37%.On BFCL V4, it reaches 46.04%, exceeding Qwen3-8B and AWM-8B by 2.00% and 1.75%.
- Main Results: EnvACE attains the best FinMCP-Bench TF1 score of 46.78%, outperforming EnvScaler-8B and AWM-8B by 3.10% and 4.28%.It also achieves the highest tool precision at 54.04%, despite not having the highest tool recall.
- World Rehearsal Improves Policy Learning: At 8B, EnvACE raises τ 2-Bench performance from 31.2% with standard GRPO to 36.7%, a gain of 5.5%.The improvement is especially relevant to stateful, multi-turn interactions because rehearsal helps anticipate environment responses.
- Internalizing Environment Dynamics: Sharing parameters between acting and rehearsal improves τ 2-Bench performance from 35.5% with Per-role Policy to 36.7%, a gain of 1.2%.Shared parameters allow rehearsal knowledge about action-conditioned responses to directly inform acting.
- Scaling and Training Dynamics: Scaling EnvACE from 1.7B to 8B improves BFCL V4 average performance from 31.81% to 46.04% and τ 2-Bench average performance from 15.3% to 36.7%.The corresponding gains are 14.23% and 21.4%, and EnvACE outperforms standard GRPO at both scales.
- World Rehearsal Improves Test-Time Performance: With test-time rehearsal budget N = 2, parallel rehearsal using EnvACE reaches a 40.9% Overall score, improving the Non-TTS result of 36.7% by 4.2%.The gains occur without additional external-environment interaction and exceed rehearsal with the base model across parallel and sequential modes.
6 CONCLUSION
EnvACE is an agentic reinforcement learning method centered on world rehearsal, with a shared policy jointly learning to act and generate environment responses. Repeated rehearsal internalizes environment dynamics as an agent world model that guides later decisions.
- 6 CONCLUSION: EnvACE assigns acting and rehearsal to one shared policy and optimizes both roles with role-wise GRPO.The policy learns both action selection and generation of the corresponding environment responses.
- 6 CONCLUSION: Repeated rehearsal absorbs environment-response knowledge into the policy parameters, forming an internal agent world model.This internalized model captures environment dynamics within the policy rather than relying only on external interaction.
- 6 CONCLUSION: The internalized environment dynamics guide the policy’s subsequent decisions.World rehearsal therefore supports decision making by conditioning later actions on learned environment responses.
7 LIMITATION
The study evaluates EnvACE only up to 8B models and primarily on tool-interactive tasks, leaving larger-scale effectiveness and broader agentic settings for future work.
- 7 LIMITATION: EnvACE is evaluated only up to the 8B scale, so its effectiveness on larger models remains uninvestigated.This limitation is attributed to computational constraints.
- 7 LIMITATION: The evaluation focuses primarily on tool-interactive tasks, leaving extension of world rehearsal to broader agentic settings as future work.
A CASE STUDY
The case studies show that EnvACE uses world rehearsal to anticipate tool-call failures before execution, repair invalid parameters, and avoid prohibited write operations. Compared with EnvScaler8B and Vanilla agents, this enables more reliable and efficient tool use with fewer recovery steps.
- Figure 7: EnvACE anticipates a potential tool-call failure and repairs invalid parameters before execution, whereas EnvScaler8B and Vanilla agents require additional recovery interactions.This example illustrates more reliable and efficient tool use through world rehearsal.
- Figure 8: EnvACE rehearses a planned tool call, predicts that a flight-reservation write operation will fail, and avoids executing it under the environment constraints.The user requested a basic-economy reservation change, but the write operation was disallowed.
- Figure 8: EnvACE proactively replaces the invalid write with a safe read-only query, while EnvScaler8B and Vanilla agents fail to anticipate the consequence and execute invalid writes.The intervention occurs before tool execution through internal world rehearsal.
World Rehearsal Prompt for Agentic Tool Execution
The world rehearsal prompt instructs a precise tool simulator to validate the RL model’s tool call against an allowed-tool whitelist and JSON schema, then generate the induced execution result. It supplies execution context and requires a tagged, valid-JSON tool-call format.
- Validation and execution: The simulator validates each RL model tool call against the candidate tools whitelist and the applicable JSON schema before generating its execution result.The whitelist contains the tool schemas allowed for the sample, while the current tool is provided with its name, schema, and parsed arguments.
- Inputs: Its inputs include previous tool calls and results, the current conversation history, and the RL model’s latest response containing tool-call tags.Previous calls and results provide few-shot ground-truth reference data and may be unavailable when no prior calls exist.
- Validation and execution: The latest response must wrap the call in <tool call>...</tool call> tags and contain valid JSON with name and arguments fields.The required structure is {"name": "...", "arguments": {...}}.
- Failure handling: Malformed JSON or a missing wrapper produces an INVALID_TOOL_CALL_FORMAT execution result with a concise reason.The format check treats absent tool-call tags and malformed JSON as failures.
2. Tool Whitelist Check
The tool name must be present in the candidate tools whitelist before simulation proceeds. If it is unavailable, the system returns a TOOL_NOT_AVAILABLE error instructing the caller to check the function call and use provided tools.
- Whitelist requirement: The simulated tool name must appear in the candidate tools whitelist.The whitelist may be an explicit list or the <tools> block in the caller’s system prompt.
- Whitelist requirement: The whitelist can be supplied either as an explicit list or through the caller’s system-prompt <tools> block.
- Failure handling: On failure, the system returns code TOOL_NOT_AVAILABLE and asks the caller to check the function call and use provided tools.The error message states that the tool name was not found.
3. Argument Schema Check … Previous Attempt History Prompt
The procedure validates tool arguments against a JSON schema, generates schema-consistent execution results, and uses simulated attempts, evaluations, and prior histories to produce one conservative final execution recommendation.
- 3. Argument Schema Check: Arguments must be JSON objects with required fields, matching primitive types, and valid enum values.These checks define the input schema requirements.
- 4. Tool Response Generation: After validation passes, the system reuses matching few-shot results or generates a factually consistent result satisfying the input schema.Examples may require correcting types or errors before reuse.
- Self-Evaluation Prompt: The supervisor judges whether each simulated action is correct and helpful, identifies weaknesses, and explains shortcomings using environment feedback.It receives tool documents, conversation history, and the simulated action with its returned result.
- Self-Evaluation Prompt: Incorrect actions receive a revised action or improved plan, while the evaluation output uses 1 for correct actions and 0 for incorrect actions.The evaluation result is returned within the specified result tags.
- Summarization Prompt for Final Execution Recommendation: The final advisor synthesizes multiple simulated attempts, tool returns, and optional feedback into the most reliable and safest execution strategy.It extracts lessons from failures and partial successes before determining the action plan.
- Important Rules: The output must be one conservative natural-language recommendation, avoiding repeated history and multiple alternative plans.Actions should be robust and verified in simulation.
- Output Format: The recommendation may include a step-by-step sequence only for reference, with the rationale for each step explained.The prescribed output format is a JSON object containing recommendation and rationale fields.
- Previous Attempt History Prompt: Previous attempts are provided as structured records containing each action, evaluation, and suggestion.These records support the final recommendation process.