Source-linked AI summary
Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, Bill Yuchen Lin
TL;DR
Open LLM agents are commonly trained from expert trajectories, but this reliance can produce sub-optimal policies because target-environment exploration is limited. ETO instead iterates between collecting failure trajectories and training on contrastive preference pairs with DPO, and it outperforms SFT and other strong baselines across three tasks, including a 22% improvement over SFT on ScienceWorld’s out-of-distribution test set.
Problem
Expert-only SFT behavioral cloning may yield sub-optimal, less generalizable open-agent policies because it inadequately explores target environments.
Method
ETO iteratively explores environments to collect failure trajectories, pairs them contrastively with successful trajectories, and updates the policy using DPO.
Results
Across three agent datasets, ETO outperformed SFT behavioral cloning and strong baselines by a large margin, including a 22% improvement over SFT on ScienceWorld’s out-of-distribution test set.
Takeaways & Limitations
ETO supports learning from exploration failures, with reported gains in out-of-distribution generalization, action efficiency, and self-play without expert trajectories.
Takeaways & Limitations
ETO assumes incorrect actions occur from the beginning of a trajectory, while identifying intermediate bad actions and constructing action-wise rewards remains difficult in current environments.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have become integral components in various autonomous agent systems. In this study, we present an exploration-based trajectory optimization approach, referred to as ETO. This learning method is designed to enhance the performance of open LLM agents. Contrary to previous studies that exclusively train on successful expert trajectories, our method allows agents to learn from their exploration failures. This leads to improved performance through an iterative optimization framework. During the exploration phase, the agent interacts with the environment while completing given tasks, gathering failure trajectories to create contrastive trajectory pairs. In the subsequent training phase, the agent utilizes these trajectory preference pairs to update its policy using contrastive learning methods like DPO. This iterative cycle of exploration and training fosters continued improvement in the agents. Our experiments on three complex tasks demonstrate that ETO consistently surpasses baseline performance by a large margin. Furthermore, an examination of task-solving efficiency and potential in scenarios lacking expert trajectory underscores the effectiveness of our approach.
1 Introduction
ETO addresses the limitations of expert-only imitation learning by incorporating exploration failures into iterative policy optimization. Across three complex interactive tasks, it improves on SFT behavioral cloning and other strong baselines, with analyses covering generalization, efficiency, and settings without expert trajectories.
- Open LLM agents are less effective than GPT-4, motivating methods that improve their construction and performance.
- Expert-only SFT behavioral cloning may produce sub-optimal, less generalizable policies because it inadequately explores target environments.
- ETO learns from exploration failures alongside successful trajectories rather than relying solely on successful demonstrations.
- Across WebShop, ScienceWorld, and ALFWorld, ETO consistently outperformed SFT behavioral cloning and other robust baselines by a significant margin.The evaluations covered web navigation, simulated science experiments, and embodied household tasks.
- ETO improved performance by 22% over SFT on ScienceWorld’s challenging out-of-distribution test set.The analysis also reported higher rewards with fewer action steps and promising self-play performance without expert trajectories.
- ETO iteratively collects failure trajectories during environment interaction and refines the agent policy through contrastive learning.The method starts from an SFT-based base agent, pairs failed agent trajectories with previously collected expert trajectories, and applies DPO loss.
2 Task Formulation
The agent–environment interaction is modeled as a POMDP in which natural-language instructions, actions, and observations define the task interface. The policy generates actions from the instruction and interaction history until completion or a step limit, after which a reward is computed.
- The task environment is formalized as a POMDP with instruction, state, action, observation, transition, and reward spaces.In the LLM-agent setting, instruction, action, and observation spaces are subsets of natural-language space.
- At each step, the LLM policy generates an action from the instruction and prior actions and observations, producing environment feedback for the next decision.
3 Method
ETO starts with a behavioral-cloned base agent, then iteratively explores the environment, forms contrastive trajectory pairs, and updates the policy through trajectory-level learning. Its training objective increases the likelihood of higher-reward trajectories while constraining deviation from a reference policy.
- Overview: ETO initializes a base agent through behavioral cloning and continually improves its policy through trial-and-error exploration.The framework is organized as an iterative exploration-training process.
- Behavioral Cloning: Behavioral cloning uses ReAct-style trajectories that generate Chain-of-Thought rationales before each action.The CoT and action are generated together in the ReAct framework.
- Exploration Phase: During exploration, the base agent performs training tasks, receives trajectory rewards, and is paired with expert trajectories when their rewards differ.Pairs with equal rewards are discarded, including cases where both trajectories successfully complete the task.
- Training Phase: ETO models failure-success preferences with contrastive learning, using DPO-style optimization to update the agent policy.The policy is trained on trajectory-pair data rather than directly applying online reinforcement-learning optimization.
- Training Phase: The contrastive objective raises the likelihood of higher-reward trajectories and lowers the likelihood of lower-reward trajectories while preserving basic agent capabilities.A KL-weighted constraint controls deviation from the base reference policy.
4 Experiments
ETO is evaluated across three interactive agent datasets, where it improves baseline performance, generalizes to unseen task variations, and can solve tasks with fewer actions. Iteration and contrastive-data analyses also identify settings where gains diminish or become unstable.
- Experimental settings: ETO is evaluated on WebShop, ScienceWorld, and ALFWorld, covering web navigation, science experiments, and household tasks with seen and unseen test scenarios.WebShop and ScienceWorld provide dense final rewards, whereas ALFWorld provides binary task-completion rewards.
- Overall results: ETO increases average reward over SFT by 8% on WebShop and 9.5% on ScienceWorld, while outperforming all other baselines across the three datasets.The comparison includes SFT behavioral cloning and post-imitation baselines such as Best-of-N, RFT, and PPO.
- Generalization: 20% performance improvement on ScienceWorld-Unseen shows ETO’s stronger advantage on out-of-domain task variations, while ALFWorld results outperform RFT and PPO despite their degradation.The paper attributes this pattern to learning from trial-and-error exploration failures.
- Different base LLMs: ETO consistently improves performance across Llama-2-13B-Chat and Mistral-7B, with relatively larger gains for the weaker Llama-2-7B agent.The reported comparison covers WebShop and ScienceWorld.
- Efficiency: ETO reaches higher ScienceWorld rewards in fewer action steps than SFT and sometimes reaches score 100 earlier than the oracle agent.This efficiency analysis uses reward trajectories over time steps on ScienceWorld-Seen tasks.
- Iteration and ablation: ETO improves WebShop and ScienceWorld performance during the first two iterations, but later iterations decline; on ALFWorld, only the first iteration improves over SFT.The fixed expert set and repeated exploration constrain contrastive-data diversity, encouraging later overfitting; binary rewards further hinder ALFWorld iteration gains.
- Contrastive-data ablation: Trajectory-wise contrastive learning performs best, while step-wise contrastive modeling is less stable and requires a lower learning rate and higher β.The paper links this instability to estimating action quality from final rewards when constructing step-wise pairs.
- Expert-trajectory availability: ETO alone does not improve WebShop without behavioral cloning, whereas combining RFT with ETO produces further enhancement without relying solely on expert trajectories.RFT can improve capabilities without expert trajectories in this experiment, but ETO’s standalone result does not.
5 Related Work
ETO builds on imitation learning, preference-based policy learning, and failure-aware agent training, while differing from approaches that require additional expert data.
- Imitation Learning: Behavioral cloning learns a direct state-to-action policy from expert trajectories, but ETO instead learns from contrastive failure-success trajectory pairs.ETO shares the failure-learning spirit of DAgger without gathering additional expert trajectories for failed cases.
- LLM Agents: LLM agents commonly use language models as core controllers for web browsing, embodied tasks, multimodal reasoning, and complex question answering.The related work includes systems such as AutoGPT, BabyAGI, and RestGPT.
- LLM Policy Learning: RLHF trains a reward model before policy optimization, whereas DPO directly models preferences to improve training efficiency and stability.ETO uses DPO-style preference learning for trajectory optimization.
- ETO’s Position: ETO can leverage exploration failures for policy optimization with high computational efficiency and robustness.The comparison is summarized in the cited table caption.
6 Conclusion
ETO improves LLM agents through iterative exploration and training that turns failure trajectories into preference data. Experiments on three agent datasets report large-margin gains over behavioral cloning and strong baselines, alongside efficiency and promise without expert trajectories.
- Conclusion: ETO iteratively explores environments, collects failure trajectories, constructs preference pairs, and trains the policy with DPO loss.The framework improves a behavioral-cloning base agent through repeated exploration-training cycles.
- Conclusion: ETO outperforms behavioral cloning and strong baselines by a large margin across three agent datasets.The paper also reports remarkable task-solving efficiency and potential when expert trajectories are unavailable.
Limitations
The paper identifies limitations in ETO’s failure-action assumptions and task scope, especially for fine-grained correction and generalized agents.
- Limitations: ETO assumes incorrect actions occur from the beginning, while realistic failures may arise at an intermediate step.Most current environments lack the information needed to identify bad actions and support action-wise or process-level reward modeling.
- Limitations: ETO primarily develops specialized agents for individual tasks, with limited exploration of strong generalized agents.Future work is proposed on policy transferability and multitask training.
A Datasets
ETO is evaluated in WebShop, ScienceWorld, and ALFWorld, covering web navigation, science experiments, and embodied household tasks. The datasets provide different reward structures, evaluation splits, and sources of trajectory supervision.
- WebShop: WebShop is an online shopping environment where agents navigate a website and receive a final reward based on product-attribute and price matching.The reward is issued after the agent selects the buy action.
- ScienceWorld: ScienceWorld is a text-based environment for elementary science experiments spanning ten task types, with rewards based on optional subgoal achievement.Tasks include areas such as thermodynamics and electrical circuits.
- ScienceWorld: ScienceWorld’s original test set evaluates generalization to unseen task variations, while the development set is used for seen-scenario testing.Task-9 and Task-10 are excluded because their trajectories are excessively long.
- ALFWorld: ALFWorld uses interactive TextWorld environments paralleling embodied worlds, requiring agents to complete high-level household instructions.Its seen and unseen sets assess in-distribution and out-of-distribution generalization, respectively.
- Trajectory Supervision: Trajectory supervision combines human annotations, GPT-4-generated WebShop trajectories, ScienceWorld heuristic searches, and GPT-4-generated rationales.WebShop trajectories are selected with reward greater than 0.7, while ScienceWorld golden trajectories cover each subtask.
B Success Rate
This section defines success-rate measurement across the three tasks and describes a step-wise ETO variation that learns from contrastive action outcomes using DPO.
- Success rates use task-specific definitions: reward-1.0 instances for WebShop, latent-state completion for ScienceWorld, and average final reward for ALFWorld.ScienceWorld success does not require the reward to equal 1.0.
- The predicted trajectory is generated by teacher forcing through step t − 1, then letting the agent act from step t onward.The resulting trajectory is evaluated by the environment, which returns its reward.
- ETO constructs good-bad action pairs by comparing expert and agent-predicted actions after a shared teacher-forced prefix.The higher- and lower-reward actions are selected from the expert action and the predicted action at step t.
- Table 7 reports success rates for different methods on three agent datasets.For ALFWorld, the reported success rate equals average final reward.
- The resulting contrastive action relation is used in a DPO loss to improve the policy.
D Case Study
The case studies contrast SFT behavioral cloning with ETO on WebShop and ScienceWorld. ETO corrects action-selection failures that lead SFT to sub-optimal or unsuccessful trajectories.
- Figures 5 and 6 illustrate the WebShop and ScienceWorld case studies, respectively.
- WebShop: In WebShop, SFT failed to select the “3pc” color option, whereas ETO chose the appropriate attribute after learning from past failures.
- ScienceWorld: In ScienceWorld, SFT repeatedly focused on a non-living object instead of finding an animal, while ETO successfully completed the task.
- ScienceWorld: The ScienceWorld example also shows SFT selecting an action inconsistent with its correct chain-of-thought rationale.ETO’s successful behavior is presented as evidence of trial-and-error effectiveness in this case study.
E Prompt for Evaluation
The evaluation prompts specify how agents interact with WebShop, ScienceWorld, and ALFWorld. They provide environment observations, task instructions, and constrained action formats or action vocabularies.
- The evaluation includes instruction prompts for WebShop, ScienceWorld, and ALFWorld, shown in Figures 7, 8, and 9.
- WebShop: WebShop requires the agent to respond to each observation with an available search or click action matching the specified structure.The prompt formats actions as search[keywords] or click[value].
- ScienceWorld: ScienceWorld instructs the agent to explore a multi-room environment, obtain needed items, and complete scientific experiments.The prompt allows teleportation between rooms and states that containers are already open.
- ScienceWorld: ScienceWorld provides actions for manipulating objects, devices, electrical components, and room observations.The listed actions include opening, activating, connecting, examining, moving, and picking up objects.
- ALFWorld: ALFWorld frames evaluation as household interaction in which the agent observes the environment, plans future actions, and acts toward a task goal.