PaperlayerView exact revision on arXiv

Source-linked AI summary

Look Before You Leap: Autonomous Exploration for LLM Agents

Ziang Ye, Wentao Shi, Yuxin Liu, Yu Wang, Zhengzhou Cai, Yaorui Shi, Qi Gu, Xunliang Cai, Fuli Feng

arXiv:2605.16143v1cs.AIcs.CL

TL;DR

LLM agents often lack autonomous exploration, limiting their ability to acquire environment-specific knowledge before acting. The paper introduces ECC, trains exploration with interleaved task and exploration rollouts, and proposes Explore-then-Act; explicitly trained agents improve downstream performance across diverse interactive environments.

  • Problem

    Task-oriented RL provides limited incentive for autonomous exploration, leaving agents with insufficient ability to acquire knowledge online in unfamiliar environments.

  • Method

    The paper introduces ECC and interleaves task-execution and exploration rollouts, then uses Explore-then-Act to separate information gathering from task execution.

  • Results

    Explicit exploration training substantially improves downstream task performance and helps agents convert an initial interaction budget into useful environment knowledge across diverse environments.

  • Takeaways & Limitations

    Autonomous exploration is an essential trainable capability for agents that must build grounded environmental knowledge before task execution.

  • Takeaways & Limitations

    The framework studies exploration mainly as an initial phase in text-based interactive environments, whereas real-world settings may require dynamic, task-conditioned exploration.

Abstract

from arXiv · show

Large language model based agents often fail in unfamiliar environments due to premature exploitation: a tendency to act on prior knowledge before acquiring sufficient environment-specific information. We identify autonomous exploration as a critical yet underexplored capability for building adaptive agents. To formalize and quantify this capability, we introduce Exploration Checkpoint Coverage, a verifiable metric that measures how broadly an agent discovers key states, objects, and affordances. Our systematic evaluation reveals that agents trained with standard task-oriented reinforcement learning consistently exhibit narrow and repetitive behaviors that impede downstream performance. To address this limitation, we develop a training strategy that interleaves task-execution rollouts and exploration rollouts, with each type of rollout optimized by its corresponding verifiable reward. Building on this training strategy, we propose the Explore-then-Act paradigm, which decouples information-gathering from task execution: agents first utilize an interaction budget to acquire grounded environmental knowledge, then leverage it for task resolution. Our results demonstrate that learning to systematically explore is imperative for building generalizable and real-world-ready agents.

1 Introduction

The paper frames autonomous exploration as a distinct, measurable capability that current task-oriented RLVR agents often lack in unfamiliar environments. It introduces ECC, trains exploration explicitly alongside task execution, and proposes Explore-then-Act to improve downstream task performance.

  • Motivation: Current RLVR approaches primarily optimize task-completion rewards, leaving autonomous exploration underexplored in complex environments.This gap concerns agents’ ability to acquire environment-specific information before acting.
  • Motivation: Premature exploitation causes agents to follow training-time priors instead of systematically uncovering hidden constraints and available tools in unfamiliar environments.The resulting failures include lacking a clear starting point and insufficient environment understanding.
  • Method: Exploration Checkpoint Coverage (ECC) quantifies how extensively an agent discovers key states, objects, and affordances in an unfamiliar environment.The metric formalizes exploration as an independent, verifiable capability for evaluating models and training paradigms.
  • Method: Interleaved GRPO alternates task-execution and exploration rollouts, optimizing them with task-completion and ECC rewards, respectively.This strategy explicitly trains exploration rather than relying on task-oriented training to produce it incidentally.
  • Results: Explicit exploration training develops meaningful exploration and substantially improves downstream task performance across diverse interactive environments.The paper evaluates ALFWorld, SciWorld, TextCraft, and a challenging ALFWorld variant, and proposes Explore-then-Act to separate information gathering from task resolution.

2 Related Work

LLM-based agents have progressed from prompt engineering to supervised fine-tuning and reinforcement learning, but narrow training data can limit generalization to novel settings. Existing work also commonly addresses unfamiliar environments through offline, heuristic, or code-driven environment modeling.

  • LLMs serve as foundational components in agent systems because of their instruction following, planning, and broad generalization capabilities.
  • Agent development has progressed from prompt engineering to supervised fine-tuning on curated trajectories, while narrow training data constrains generalization to novel settings.
  • Existing work often treats unfamiliar-environment modeling as an offline engineering or pre-compilation task rather than an online capability.
  • Heuristic and code-driven pipelines such as Wall-E, WESE, and AutoManual construct external knowledge bases using traditional search algorithms including BFS and DFS.

3 Methodology

The methodology defines autonomous exploration as a goal-free information-gathering capability, measures it with Exploration Checkpoint Coverage (ECC), and optimizes it alongside task execution. Explore-then-Act then separates fixed-budget exploration from goal-conditioned acting using a grounded knowledge summary.

  • Autonomous Exploration: Autonomous exploration independently probes an unfamiliar environment to build an internal model of transition dynamics, the state space, and action semantics.Unlike task execution, exploration operates without an assigned task goal.
  • Exploration Checkpoint Coverage: Exploration Checkpoint Coverage (ECC) measures the fraction of environment-specific facts or affordances reached, observed, or verified during an exploration trajectory.Checkpoints include reachable locations, important objects, valid interaction targets, functional states, action-relevant affordances, and environment-specific constraints.
  • Interleaved Training Schedule: An interleaved training schedule alternates ECC-based exploration optimization with standard task-focused GRPO optimization rewarded by downstream task completion.This schedule is designed to develop both exploration and task-solving abilities.
  • Optimizing for Exploration via GRPO: The training method uses ECC as a verifiable reward in GRPO, directly encouraging discovery of more checkpoints without requiring a subjective open-ended language judge.Policy updates increase the likelihood of trajectories with higher relative ECC while applying a KL penalty for stability.
  • Explore-then-Act: Explore-then-Act allocates a fixed budget of N exploration steps, summarizes the resulting trajectory as grounded knowledge K, and conditions subsequent task actions on K, the history, and goal g.The preliminary exploration phase is goal-free, while the subsequent acting phase is goal-conditioned.

4 Experiments

Experiments evaluate autonomous exploration across diverse LLM backbones and interactive environments using task-free exploration, ECC, and downstream performance gain. Results show exploration deficits in existing and task-trained models, while exploration-aware training improves execution, robustness, and information use.

  • Experimental Setup: Experiments cover Qwen2.5-7B, Qwen3-4B, LLaMA3.1-8B, GPT-4.1, and Claude-Opus-4.5 across ALFWorld, ScienceWorld, and TextCraft.Agents explore without task instructions under a maximum interaction budget of 100 steps.
  • Experimental Setup: Exploration quality is measured by Steps and ECC (%), while downstream utility is measured by ∆Task = E-t-A −Dir.ECC captures discovered critical states, key objects, and distinct locations during free exploration.
  • Autonomous Exploration Capability: Qwen2.5-7B and Qwen3-4B achieve average ECC scores of 22.2% and 28.5%, respectively, while task-oriented GRPO lowers Qwen3-4B’s average ECC from 28.5% to 18.8%.The models often repeat loops or terminate exploration prematurely.
  • Exploration-Aware Training: GRPO (Interleaved) consistently outperforms GRPO (Task-Only) in Direct Execution and E-t-A, while GRPO (Explore-Only) also improves over the base model.These results indicate that exploration-centric rewards enhance exploratory skills and task execution.
  • Explore-then-Act: GRPO (Interleaved) and GRPO (Explore-only) achieve positive E-t-A gains across all three environments and both backbone models.Exploration-focused training helps agents convert an initial exploration budget into actionable information.
  • Behavioral Diagnostics and Robustness: GRPO (Interleaved) reduces repetitive behaviors while increasing information-seeking and error-recovery actions, conditioning agents to verify states, adapt to feedback, and pursue alternatives.Additional experiments examine robustness under ALFWorld perturbations and exploration efficiency using a fixed Qwen3-4B executor.

5 Conclusion

The paper identifies autonomous environment exploration as an essential capability missing from many LLM agents optimized primarily for task completion. It formalizes exploration as an independent trainable objective and introduces Exploration Checkpoint Coverage (ECC) to quantify agents’ discovery of critical environmental elements.

  • 5 Conclusion: Task-focused LLM agents often exhibit premature exploitation, making autonomous environment exploration a missing but essential capability.The passage characterizes exploration as necessary for agents beyond primary task-completion optimization.
  • 5 Conclusion: The paper formalizes environment exploration as an independent and trainable objective.This framing enables systematic study of exploration as a capability distinct from task completion.
  • 5 Conclusion: Exploration Checkpoint Coverage (ECC) is introduced as a verifiable metric for quantifying discovery of critical states, objects, and affordances.ECC measures the extent to which agents discover these elements within an environment.

A Limitations and Future work. · B Boarder Impact

The paper presents autonomous exploration as an initial, measurable capability for LLM agents while acknowledging limitations in upfront exploration and outlining potential benefits for applications in unfamiliar or changing environments.

  • A Limitations and Future work.: The framework studies exploration primarily as an initial phase before task execution.This provides a clean and controllable setting for isolating, measuring, and training exploration ability.
  • A Limitations and Future work.: Real-world environments may be too large and complex to fully explore upfront.The passage identifies this mismatch as a limitation of the current framing.
  • A Limitations and Future work.: The work identifies extending the framework beyond upfront exploration as future work.The supplied passage begins describing this direction but is truncated before specifying the proposed extension.
  • B Boarder Impact: The work formalizes autonomous exploration as a measurable capability for LLM agents.It also introduces training strategies intended to improve this capability.
  • B Boarder Impact: The methods enable agents to acquire grounded environment knowledge online.This capability may benefit systems operating in unfamiliar or changing environments.
  • B Boarder Impact: Potential application areas include virtual assistants, web automation, educational tools, and embodied AI systems.These applications are presented as possible beneficiaries rather than direct deployments.
  • B Boarder Impact: The work is methodological and does not directly deploy real-world agents.The passage frames stronger exploration ability as a potential benefit rather than a demonstrated deployment outcome.

C Addtional Experimental Details

The experiments use interleaved GRPO training, combining task-execution and exploration rollouts with their respective rewards, and run on a single 8× NVIDIA H800 node.

  • GRPO training: Interleaved GRPO batches combine task-execution rollouts rewarded by binary task success with exploration rollouts rewarded by ECC, using a default 5:1 ratio.The ratio is task-execution to exploration rollouts per training batch.
  • Training Resources: All experiments run on a single node equipped with 8× NVIDIA H800 GPUs.This describes the hardware configuration used across the experiments.
  • Training Resources: GRPO training requires approximately 192 GPU-hours because of online rollout generation.The reported compute requirement reflects the online rollout generation process.

D Sensitivity to the Task-Exploration Ratio

The study varies the task-to-exploration rollout ratio during GRPO training and evaluates each policy with Direct Execution and Explore-then-Act. Mixed training outperforms endpoint strategies when task rollouts are sufficiently represented, with 5:1 yielding the best performance under both evaluation modes.

  • Setup: GRPO training varies only the task:exploration rollout ratio across Task-Only, Explore-Only, and six mixed settings on Qwen3-4B with ALFWorld.All runs use the same training budget and hyperparameters and are evaluated under Direct Execution and Explore-then-Act.
  • Analysis: Task-Only training preserves strong Direct Execution performance but provides almost no benefit from an additional exploration phase.
  • Analysis: Explore-Only training substantially underperforms because it lacks sufficient task-completion signal.
  • Analysis: Mixed training improves over both endpoints once task rollouts are sufficiently represented, with performance rising toward task-heavy ratios.
  • Analysis: The 5:1 task:exploration ratio achieves the best Direct Execution and Explore-then-Act success rates.

E Construction of Environment Checkpoints

Environment Checkpoint Coverage constructs verifiable checkpoint sets from each environment engine’s internal state, covering locations, objects, and affordances. Coverage is then determined deterministically from textual observations and successful actions without model-generated annotations or LLM judgment.

  • Checkpoint Construction: ECC checkpoints are derived from the environment engine’s internal state representation, avoiding model-generated annotations.The construction procedure produces verifiable, ground-truth checkpoints for each environment instance.
  • Checkpoint Construction: Each checkpoint set includes distinct navigable locations, key interactable objects, and valid object- or location-associated affordances.Examples include opening objects and activating devices.
  • Checkpoint Construction: The extraction pipeline gathers reachable states, extracts locations, objects, and affordances, then deduplicates and filters checkpoints by relevance.Algorithm 1 applies these operations before returning the checkpoint set.
  • Verification Mechanism: Coverage is marked when an observation or action unambiguously demonstrates awareness of a checkpoint.Verification uses deterministic string matching against textual observations and requires no LLM-based judgment.
  • Verification Mechanism: Location checkpoints trigger on corresponding room descriptions, object checkpoints after observed interaction or examination, and affordance checkpoints after successful associated actions.In ALFWorld, checkpoints derive from PDDL game states; in ScienceWorld, they come from the object tree and action space.

F Detailed Construction of ALFWorld Variants · G Prompt for Exploration · H Case Study

The paper evaluates exploration under controlled ALFWorld environment shifts, specifies a grounded exploration prompt, and uses qualitative case studies to compare exploration-aware with task-oriented agents. The variants isolate object placement, interaction preconditions, and distractor ambiguity while preserving task structure.

  • F Detailed Construction of ALFWorld Variants: F Detailed Construction of ALFWorld Variants: Three ALFWorld variants perturb one environment axis while preserving task structure across the same 274 test instances.This design attributes performance changes to specific perturbations rather than fundamentally different tasks.
  • F Detailed Construction of ALFWorld Variants: F Detailed Construction of ALFWorld Variants: Variant 1 randomly relocates task-relevant objects and receptacles while keeping all necessary objects reachable and tasks solvable.For example, a mug may move from a kitchen counter to a bedroom drawer.
  • F Detailed Construction of ALFWorld Variants: F Detailed Construction of ALFWorld Variants: Variant 2 changes interaction preconditions, requiring altered action sequences such as explicitly opening containers or clearing receptacles.The spatial layout remains unchanged while valid interaction sequences change.
  • F Detailed Construction of ALFWorld Variants: F Detailed Construction of ALFWorld Variants: Variant 3 injects visually or semantically similar distractors, increasing ambiguity and testing target discrimination.For instance, several additional books may be placed in different locations when one specific book is required.
  • G Prompt for Exploration: G Prompt for Exploration: The general exploration prompt instructs agents to understand environments through systematic exploration of actions, state information, patterns, and clues.Its strategy includes testing state transitions, mapping information, tracking reversibility, identifying decision points, and exploring paths and branches.
  • G Prompt for Exploration: G Prompt for Exploration: The prompt requires findings to come from direct interaction, instructing agents to execute actions, trust observations over expectations, and update their understanding.Agents are told not to rely on prior knowledge, skip seemingly obvious exploration, or assume action behavior.
  • H Case Study: H Case Study: Two qualitative ALFWorld case studies use real model rollouts from identical environment instances to illustrate behavioral differences between exploration-aware and task-oriented agents.The case studies are presented as behavioral comparisons within the same environment conditions.

H.1 Task-Oriented Training VS Exploration-Aware Training

In the same ALFWorld bedroom with identical goal-free instructions and a 50-step budget, the task-oriented model terminated after one step with ECC 0, while the exploration-aware model used 49 steps to achieve ECC 87%. The exploration-aware trajectory discovered objects, action semantics, and environmental constraints, whereas the task-oriented model retained only generic, ungrounded knowledge.

  • Task-Oriented Training VS Exploration-Aware Training: ECC: 0 and Steps: 1 for GRPO Task-Only, which immediately terminated exploration after its initial observation.It failed to interact with objects or discover environment dynamics because no actionable goal or task reward signal was present.
  • Task-Oriented Training VS Exploration-Aware Training: The task-oriented model generated only generic prior knowledge, with objects unnamed, action syntax unknown, and constraints unverified.Although it acknowledged the need to explore, it had already terminated.
  • Task-Oriented Training VS Exploration-Aware Training: The exploration-aware model verified action semantics and constraints, including single-item inventory, proximity requirements, container states, and holding requirements for examination.Invalid actions returned “Nothing happens,” providing a weak error signal.
  • Task-Oriented Training VS Exploration-Aware Training: The exploration-aware model discovered a grounded inventory spanning the bed, dining table, drawer 1, and sidetables, and produced a fully grounded world model.The assessment cites correct syntax, discovered constraints, and complete object inventory.

H.2 Low-ECC vs. High-ECC Grounded Task Execution

In the same ALFWorld kitchen task, high-ECC grounded execution succeeds in 7 steps by using exploration-derived knowledge, whereas zero-ECC task-only execution fails after 100 steps. The contrasting trajectories show that grounded exploration supports efficient, constraint-aware action while shallow task routines lead to exhaustive search and perseveration.

  • H.2 Low-ECC vs. High-ECC Grounded Task Execution: High-ECC grounded execution completes the mug-cooling and placement task successfully in 7 steps.The agent directly locates the mug, cools it with the fridge, and places it on the coffeemachine.
  • H.2 Low-ECC vs. High-ECC Grounded Task Execution: The exploration-aware agent knows the exact cooling syntax, respects the one-object constraint, and finds the mug on its first receptacle visit.Its trajectory verifies constraints while moving directly between the relevant objects and devices.
  • H.2 Low-ECC vs. High-ECC Grounded Task Execution: Zero-ECC grounded execution fails after 100 steps on the same task and environment.The task-only agent exhaustively searches cabinets, drawers, and the sink basin, then fails to cool the mug because of incorrect sequencing.
  • H.2 Low-ECC vs. High-ECC Grounded Task Execution: The failed agent repeatedly attempts cooling while away from the fridge, producing “Nothing happens” until its budget is exhausted.This reflects inefficient search, violation of the proximity constraint, and perseveration without adaptation.
  • H.2 Low-ECC vs. High-ECC Grounded Task Execution: The case studies associate exploration-aware training with systematic, hypothesis-driven exploration and comprehensive environment models, unlike shallow task-like routines.The reported behaviors include testing individual actions, verifying constraints, and discovering environment structure.
Loading 2605.16143v1…