Source-linked AI summary
Agent Learning via Early Experience
Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang, Yuting Ning, Zhaorun Chen, Xiaohan Fu, Jian Xie, Yuxuan Sun, Boyu Gou, Qi Qi, Zihang Meng, Jianwei Yang, Ning Zhang, Xian Li, Ashish Shah, Dat Huynh, Hengduo Li, Zi Yang, Sara Cao, Lawrence Jang, Shuyan Zhou, Jiacheng Zhu, Huan Sun, Jason Weston, Yu Su, Yifan Wu
TL;DR
Language-agent reinforcement learning is difficult when rewards are unavailable or long-horizon interaction is inefficient, while expert-data imitation exposes agents to limited action consequences. The paper introduces early experience, using agents’ own actions and resulting states as reward-free supervision through implicit world modeling and self-reflection. Across eight environments, it reports consistent gains in effectiveness and generalization, with early-experience checkpoints also strengthening downstream reinforcement learning.
Problem
Language-agent reinforcement learning is difficult in environments lacking reliable rewards, while imitation learning does not expose agents to the consequences of their own actions.
Method
Early experience converts agents’ own actions and resulting future states into reward-free supervision through implicit world modeling and self-reflection.
Results
Across eight diverse environments and multiple model families, early experience consistently improves effectiveness and out-of-domain generalization, while also strengthening reinforcement-learning warm starts.
Takeaways & Limitations
Early experience is presented as a practical and scalable bridge between imitation learning and reinforcement learning.
Takeaways & Limitations
The reduced data requirements could lower barriers to training agents for malicious purposes, motivating attention to deployment guidelines and access controls.
Abstract
from arXiv · showhide
A long-term goal of language agents is to learn and improve through their own experience, ultimately outperforming humans in complex, real-world tasks. However, training agents from experience data with reinforcement learning remains difficult in many environments, which either lack verifiable rewards (e.g., websites) or require inefficient long-horizon rollouts (e.g., multi-turn tool use). As a result, most current agents rely on supervised fine-tuning on expert data, which is challenging to scale and generalizes poorly. This limitation stems from the nature of expert demonstrations: they capture only a narrow range of scenarios, and expose the agent to limited environment diversity. We address this limitation with a middle-ground paradigm we call early experience: interaction data generated by the agent's own actions, where the resulting future states serve as supervision without reward signals. Within this paradigm, we study two strategies of using such data: (1) implicit world modeling, which uses collected states to ground the policy in environment dynamics; and (2) self-reflection, where the agent learns from its suboptimal actions to improve reasoning and decision-making. Evaluation across eight diverse environments and multiple model families shows that our approaches consistently improve effectiveness and out-of-domain generalization, highlighting the value of early experience. Moreover, in environments with verifiable rewards, our results provide promising signals that early experience offers a strong foundation for subsequent reinforcement learning, making it a practical bridge between imitation learning and fully experience-driven agents.
1. Introduction
The paper introduces early experience, in which agents learn from future states produced by their own actions without external rewards. Across diverse environments and model families, implicit world modeling and self-reflection improve effectiveness, generalization, and subsequent reinforcement-learning performance.
- Reinforcement learning is difficult for language agents because many real-world environments lack verifiable or dense reward signals.
- Supervised fine-tuning avoids reward requirements but limits agents to static expert demonstrations and prevents them from observing their own action outcomes.
- Early experience converts future states caused by an agent’s own actions into supervision without relying on external rewards.
- Implicit world modeling predicts collected future states to represent environment dynamics, while self-reflection identifies suboptimal decisions by comparing behavior with expert demonstrations.
- Across eight environments and multiple model families, both strategies consistently outperform purely imitation-learning baselines and improve out-of-domain generalization.
- Early experience also provides stronger reinforcement-learning warm starts than standard imitation-learning checkpoints in environments with verifiable rewards.
2. Related Work
Prior language-agent training commonly relies on static expert demonstrations, while reinforcement learning remains difficult in many language environments. The paper instead uses agents’ observed next states as supervision and integrates world-modeling ideas into the policy itself.
- Supervised fine-tuning trains language agents on human-annotated or synthesized expert trajectories, but its supervision remains static.
- Reinforcement learning trains agents through trial and error using long-term rewards, yet remains difficult to apply to language-agent settings.
- Unlike hindsight relabeling or on-policy expert correction, this approach treats observed next states themselves as supervision without goal relabeling or an interactive expert.
- Traditional world models predict future states and rewards as separate simulators, whereas this approach trains the policy to predict its own future states.
- The implicit world model internalizes coarse environment dynamics, offers a lightweight warm-up, and avoids standalone planning overhead.
- Self-reflection extends earlier prompting and episodic-memory approaches by using experience to support learning within the agent-training paradigm.
3. Preliminaries
The paper formulates language-agent decision-making as a Markov Decision Process and describes imitation learning as reward-free supervised policy learning. This setup highlights why missing rewards, distribution shift, and unobserved action consequences motivate alternative supervision.
- The language-agent decision problem is formalized as an MDP with states, actions, transitions, rewards, discounting, and an initial-state distribution.
- In language-agent environments, states may encode webpage contents, tool outputs, or textual environment descriptions, while actions include agent responses.
- Reward functions may be unknown or unverifiable during training, motivating supervision sources other than environmental rewards.
- Imitation learning minimizes supervised loss on expert state-action pairs rather than learning from the consequences of alternative actions.
- Distribution shift arises when the learned policy deviates from the expert and reaches states absent from training data, causing errors to compound.
- Because imitation learning does not observe outcomes of non-expert actions, agents have limited ability to recover from errors.
4. Early Experience
Early experience augments expert demonstrations with future states produced by the agent’s own alternative actions, providing reward-free supervision. The paper develops implicit world modeling and self-reflection to learn from these transitions and explanations.
- Notation for Early Experience: For each expert state, the method executes the expert action and multiple alternative actions, recording the resulting next states.Each rollout triple contains a state, an alternative action, and its resulting next state.
- Self-Reflection: Both approaches use alternative actions proposed by the initial LLM, but world modeling predicts their consequences while self-reflection explains action preferences.Together, they turn the agent’s actions and resulting future states into scalable supervision.
- Notation for Early Experience: Early experience collects future states from the agent’s own proposed actions and converts them into supervision without explicit rewards.The rollout records alternative actions and their resulting environment responses, including DOM changes, tool outputs, error messages, and task progression.
- Implicit World Modeling: Implicit world modeling trains the policy to predict natural-language next states from state–alternative-action pairs, thereby internalizing environment dynamics.The same model parameters are used for state prediction and action prediction, integrating predictive signals directly into policy learning.
- Implicit World Modeling: The implicit world-modeling objective exposes the policy to common transitions, side effects, and invalid-action outcomes without requiring a separate simulator.Its predictive training acts as a lightweight warm-up before policy optimization and exposes the agent to non-expert behaviors.
- Self-Reflection: Self-reflection compares expert actions with policy-sampled alternatives and generates explanations, grounded in their different resulting states, for why the expert action is preferable.The explanations are trained jointly with the expert action and are mixed with expert data during next-token training.
5. Experiments
Across eight diverse environments and multiple model families, early-experience methods consistently improve effectiveness and out-of-domain generalization over imitation learning. They also provide stronger starting points for subsequent reinforcement learning, while their benefits vary with action-space structure, observation complexity, and reasoning demands.
- Effectiveness: Early experience consistently outperforms imitation learning across eight diverse environments and multiple model families.The evaluation spans embodied navigation, science, planning, question answering, tool use, and web navigation tasks.
- Effectiveness: IWM yields steady gains in structured environments, while SR produces larger improvements on multi-step reasoning and constraint-satisfaction tasks.Reported gains include ALFWorld/ScienceWorld +2.3 to +5.5, WebShop +11.3 to +18.4, TravelPlanner +12.8 to +15.0, ScienceWorld +13.3, and BFCLv3 +8.0 on the 3B model.
- Action-Space Perspective: Early experience improves policies across closed, structured-large, and open action spaces by learning transition regularities, reducing tool misuse, and supplying dense signals from exploratory rollouts.IWM is especially useful for stable dynamics, while SR targets logical errors and long-horizon plan repair.
- Observation-Space Perspective: Early experience remains effective across observation complexities, with IWM helping predictable transitions and SR helping reasoning failures and long-horizon repairs.The methods turn agent actions and resulting states into supervision without rewards, including in noisy, fine-grained web states.
- Takeaway: Early experience strengthens policies across substantially different environments, action spaces, and observation complexities across two model sizes and three environment families.This supports the generalizable feasibility of the paradigm beyond expert demonstrations.
- Out-of-Domain Generalization: Out-of-domain scores drop across tasks, but early experience recovers a substantial portion of the gap and sometimes achieves larger relative gains than in-domain training.IWM helps most under stable dynamics, SR under shifts in tool availability or arguments, and both help under retrieval shifts.
- RL Following Early Experience: Early-experience checkpoints consistently reach higher post-RL ceilings than imitation-only starts under identical GRPO training recipes.The advantage sometimes grows during RL and otherwise narrows without reversing; raw pretrained-model starts perform worst and train unstably.
6. Discussion
Early experience retains advantages across supervision levels and model scales, while outperforming reasoning-focused and preference-based alternatives. Its benefits extend to constrained-compute settings and reflect distinct effects of branching factor and expert-data quantity.
- Data Scaling: 1/8 of WebShop demonstrations already surpasses full-dataset imitation learning, while ALFWorld reaches the same with 1/2 of demonstrations.Both implicit world modeling and self-reflection improve with more expert data, while retaining a margin over imitation learning.
- Branching Factor: Implicit world modeling improves steadily as branching factor K increases, whereas self-reflection shows diminishing returns at very large K.Larger K exposes implicit world modeling to richer transition regularities; self-reflection can lose contrast when alternatives include other successful actions.
- Comparison to Baselines: Early experience consistently outperforms Long CoT, STaR-style training, and DPO across the reported benchmark comparisons.Long CoT provides modest gains, STaR rationales remain ungrounded, and DPO training collapses within tens of optimization steps on both benchmarks.
- Model Scaling: Early experience outperforms imitation learning at every tested model scale, including 70B, while absolute performance also rises with scale.Both methods remain effective with LoRA-only updates under constrained compute, and similar trends appear for Qwen models.
7. Conclusion
The paper presents early experience as reward-free learning from agents’ own actions and resulting states. Across diverse environments, its two methods improve effectiveness and out-of-domain robustness while retaining advantages for reinforcement-learning warm starts.
- Conclusion: Early experience converts agents’ own actions and resulting states into supervision without external rewards, bridging human-data training and experience-driven learning.The paradigm includes implicit world modeling and self-reflection.
- Conclusion: Across eight diverse environments, implicit world modeling and self-reflection consistently improve in-domain effectiveness and out-of-domain robustness.The environments span embodied navigation, scientific experimentation, long-horizon planning, multi-turn tool use, and web navigation.
- Conclusion: Early experience retains its advantage when used to warm-start reinforcement learning.
Impact Statement
The work enables language agents to learn from environmental interactions without explicit rewards and reduces the data needed to train capable agents. This may also lower barriers to training agents for malicious purposes, motivating deployment safeguards.
- Impact Statement: Learning from environmental interactions without explicit rewards reduces the data needed to train capable language agents.
- Impact Statement: Reduced data requirements could lower barriers to training agents for malicious purposes.The statement suggests careful deployment guidelines and access controls for general agents in open-ended environments such as the web.
Overview
The supplementary material covers contributions, implementation details, and full results and examples, alongside author-specific project, experimentation, and advisory roles.
- Supplementary Contents: Section A provides the contribution statement, Section B gives implementation details, and Section C contains full results and examples.
- Contributions: Kai Zhang led the project, designed both early experience methods, implemented the training pipeline, and validated results on ALFWorld and WebArena.
- Contributions: Individual environment experiments were led by designated authors across WebShop, TravelPlanner, SearchQA, BFCLv3, ScienceWorld, and Tau-Bench.
- Contributions: Senior authors advised the work, while Yu Su co-conceived the project and served as its primary scientific advisor.
B. Implementation Details
Self-reflection training presents a situation, goal, expert action, expected outcome, and alternatives, then asks the model to reason about why the expert action is preferable.
- The input specifies a situation and goal that require choosing among multiple possible actions.
- The prompt compares alternative actions by explaining why each may be less optimal.
- The model justifies the expert action using its expected outcome and relevant situational constraints.
- Instructions require reasoning grounded strictly in the provided information and focused on logical decision-making.
- The output is a direct self-reflection monologue without extra headings, disclaimers, or external notes.
B.2. Detailed Experiment Setup
The experiments cover diverse language-agent environments and model families, while using fixed, fair training and evaluation procedures across settings.
- Environments: The study evaluates eight environments spanning embodied navigation, web navigation, multi-turn tool use, and long-horizon planning.
- Models and Expert Trajectories: The evaluated demonstrations include optimal trajectories, potentially suboptimal human- or model-collected trajectories, and synthesized training data.
- Training and Evaluation: Training uses consistent prompt formatting and decoding strategies, with the imitation baseline’s selected step budget fixed for all methods.
- Training and Evaluation: Implicit World Modeling uses one world-modeling epoch followed by supervised updates within the imitation step budget, while Self-Reflection matches imitation’s epoch count.
- Training and Evaluation: Evaluation reports each benchmark’s native metric and follows its official validator.
C. Full Results and Examples
The paper details environment-specific data construction and training examples for ALFWorld, WebShop, BFCLv3, and WebArena. These procedures augment expert data with rollout states or reflective comparisons.
- C.1. ALFWorld: ALFWorld provides 21,031 expert state–action pairs from optimal trajectories for imitation learning.
- C.1. ALFWorld: Implicit World Modeling augments each ALFWorld state with eight non-expert actions plus the expert action, producing 189,279 triplets.
- Self-Reflection: Self-Reflection constructs prompts that compare the expert action with alternative actions and justify the expert action given the state and available tools.
- C.2. WebShop: WebShop world-modeling data combines reformatted expert steps with executed non-expert actions and next-state summaries, yielding 122,954 triplets.
- C.2. WebShop: WebShop reflection data retains trajectories completable in fewer than 15 steps, resulting in 6,235 reflection examples.
- C.8. WebArena: The WebArena example selects REPORTS because the task requires best-selling products, whereas catalog filtering lacks sales rankings.