PaperlayerView exact revision on arXiv

Source-linked AI summary

Behavioral Exploration: Learning to Explore via In-Context Adaptation

Andrew Wagenmaker, Zhiyuan Zhou, Sergey Levine

arXiv:2507.09041v1cs.LGcs.ROcs.SYeess.SY

TL;DR

Autonomous agents need to explore and adapt quickly, while existing approaches often rely on random exploration or slow gradient-based updates. The paper trains a long-context policy to explore expert behaviors using demonstration data and in-context history. It reports substantial gains over standard behavioral cloning and RL-based exploration approaches, including in a real-world robotic setting.

  • Problem

    Existing approaches to fast online exploration and adaptation often rely on random exploration and slow, gradient-based behavior updates.

    Source: §1 · p. 1

  • Method

    Behavioral exploration trains a long-context policy from demonstrations to condition expert-action prediction on past observations and an exploration measure, then adapts in-context using interaction history.

    Source: §1 · p. 1, §4.2 · p. 5

  • Results

    The authors report substantial gains over standard behavioral cloning and RL-based exploration approaches, with results spanning simulated benchmarks and real-world robotic manipulation.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    BE offers a policy-training approach for exploring behaviors contained in demonstration data with only a small modification to standard behavioral cloning.

    Source: §6 · p. 10

    BE explores behaviors present in the demonstrations, leaving open how to use their prior information when solving a task requires behavior beyond that space.

    Source: §6 · p. 10

Abstract

from arXiv · show

Developing autonomous agents that quickly explore an environment and adapt their behavior online is a canonical challenge in robotics and machine learning. While humans are able to achieve such fast online exploration and adaptation, often acquiring new information and skills in only a handful of interactions, existing algorithmic approaches tend to rely on random exploration and slow, gradient-based behavior updates. How can we endow autonomous agents with such capabilities on par with humans? Taking inspiration from recent progress on both in-context learning and large-scale behavioral cloning, in this work we propose behavioral exploration: training agents to internalize what it means to explore and adapt in-context over the space of ``expert'' behaviors. To achieve this, given access to a dataset of expert demonstrations, we train a long-context generative model to predict expert actions conditioned on a context of past observations and a measure of how ``exploratory'' the expert's behaviors are relative to this context. This enables the model to not only mimic the behavior of an expert, but also, by feeding its past history of interactions into its context, to select different expert behaviors than what have been previously selected, thereby allowing for fast online adaptation and targeted, ``expert-like'' exploration. We demonstrate the effectiveness of our method in both simulated locomotion and manipulation settings, as well as on real-world robotic manipulation tasks, illustrating its ability to learn adaptive, exploratory behavior.

1. Introduction

The paper targets fast online exploration and adaptation without relying on extensive trial and error or slow gradient-based updates. It proposes learning from expert demonstrations so a policy can adapt in-context while exploring behaviors represented in those demonstrations.

1. Introduction

  • Existing approaches often require extensive trial-and-error interactions or slow gradient-based behavior updates, unlike the fast adaptation the paper seeks.Source: §1 · p. 1
  • The authors motivate informed exploration with a robot choosing among relevant cups rather than acting randomly, trying an irrelevant plate, or repeating an attempted pickup.Source: §1 · p. 1
  • The long-context policy conditions on past observations and a measure of how exploratory an expert action is relative to that history, enabling it to select different expert behaviors online.The approach is trained by behavioral cloning and adapts by feeding past observations into the policy context.Source: §1 · p. 2, Figure 1 · p. 2
  • BE yields significant gains over standard behavioral cloning and reinforcement-learning approaches across simulated benchmarks and a real-world robotic setting.The paper reports results on simulated reinforcement-learning and imitation-learning benchmarks, as well as a real-world robotic setting.Source: §1 · p. 2

2. Related Work

The related work spans online exploration, meta-learning, demonstration-based control, and return-conditioned policies. The paper distinguishes its approach by learning exploration in-context from demonstrations rather than relying on online reward-driven updates or simply imitating history.

Exploration in decision-making

  • Most decision-making exploration methods operate fully online without prior information or offline data, while the paper learns online in-context rather than through gradient-based online RL.Offline-data approaches discussed include methods that warm-start RL or combine offline policies with online RL.Source: §2 · p. 2

Learning for fast online adaptation

  • Meta-RL methods generally require online access to training environments, while offline meta-RL and meta-imitation pursue different objectives from exploring beyond the observed history.Meta-RL maximizes a reward function; meta-imitation aims to imitate a small set of demonstrations.Source: §2 · pp. 2–3

Learning control policies from expert demonstrations

  • Unlike skill-learning and goal-conditioned behavioral-cloning work, this paper trains a single model from demonstrations to explore and adapt rather than directly mimic task-solving behavior.The cited Zhou et al. (2024) approach uses a multi-stage system and internet-scale data, whereas this work aims to learn from demonstration data alone.Source: §2 · p. 3

Decision-making with return-conditioned policies

  • Return-conditioned policies typically condition on desired reward-to-go or goals, whereas this work conditions on coverage and uses a more history-dependent structure to explore.The related work also notes modern transformer- and diffusion-based variants of return-conditioned policies.Source: §2 · p. 3

3. Preliminaries

The preliminaries frame decision-making as reward-free and define trajectories and policies conditioned on histories. Demonstration trajectories come from a behavior policy whose behaviors are assumed to provide a reasonable prior on useful behavior, without assuming optimality.

3. Preliminaries

  • A reward-free MDP is specified by state and action spaces, a transition kernel, and an initial-state distribution; a trajectory is a sequence of interactions of fixed length K.Source: §3 · p. 3
  • A policy maps a state and conditioning input to an action distribution, where the conditioning input can include the agent’s history of past interactions.Source: §3 · p. 3
  • Demonstration trajectories are collected by a behavior policy that may mix demonstrators; its behavior need not be optimal but is assumed to be a reasonable prior over useful behaviors.Source: §3 · p. 3

4. Learning to Explore from Demonstrations

The section formalizes exploration as targeted coverage of behaviors represented in expert demonstrations, then develops a history-conditioned policy for selecting less-explored expert behaviors. It also describes practical model and training choices for implementing this approach.

4. Learning to Explore from Demonstrations

  • The section frames the objective as online exploration over useful behaviors represented in demonstrations, rather than uniform exploration over all possible behaviors.The behavior policy need not be optimal, but its demonstrated behaviors are assumed to provide a reasonable prior over potentially useful behaviors.Source: §4 · p. 3

4.1. Exploration Over Demonstration Coverage

  • Demonstration-based coverage focuses exploration on features spanned by the behavior policy, rather than the entire state space.The proposed coverage measure uses the trace of an inverse feature matrix; in finite-state settings it reduces to total state coverage.Source: §4.1 · p. 4

4.2. Learning Exploratory Behaviors via Conditional Distribution Modeling

  • Behavioral exploration conditions behavior-policy actions on past observations and resulting coverage, increasing the likelihood of actions that add coverage.In the two-goal example, conditioning on higher coverage favors the less frequently selected action that reaches the previously unvisited goal; under stated conditions, this history-dependent distribution is optimal for Objective 4.1.Source: §4.2 · p. 5

4.3. Practical Implementation

  • A diffusion model with a transformer backbone models the continuous, multimodal action distribution using the current state, coverage-to-go, and state history as conditioning inputs.The training history distribution is chosen uniformly over demonstration trajectories, approximating histories the policy may encounter online; task labels can also condition behavior to focus exploration within a task.Source: §4.3 · p. 6

5. Experiments

The experiments test whether BE can learn directed exploration from offline data and adapt online across RL benchmarks, simulated imitation learning, and real-world robot tasks. They also examine how history conditioning shapes its exploration.

5. Experiments

  • The evaluations span reward-free online exploration from D4RL offline datasets, simulated and real-world imitation learning, and online deployment with prior observations in BE’s context.The D4RL setup shortens Antmaze episode horizons to test directed exploration; the imitation-learning experiments include Libero and WidowX tasks.Source: §5.1 · pp. 6–7, §5.2 · p. 8, §5.2.2 · p. 9

5.1. Fast Exploration in RL Benchmarks

  • On Antmaze, BE reaches goals more quickly and achieves higher coverage than the considered baselines, while focusing exploration on demonstrated walking behaviors.The reported comparison aggregates medium and large Antmaze results for goals reached and regions reached.Source: §5.1 · p. 7, Figure 3 · p. 7, Figure 4 · p. 7
  • On Kitchen, BE performs comparably to state-of-the-art skill-based RL approaches and improves over standard BC.The evaluation measures the percentage of tasks accomplished.Source: §5.1 · pp. 7–8, Figure 5 · p. 7
  • On Antmaze medium, conditioning deployment on different coverage values adjusts the number of regions BE visits.Figure 6 illustrates this calibration.Source: §5.1 · p. 7, Figure 6 · p. 7

5.2. Exploration in Imitation Learning

  • On Libero, BE needs 2–3× fewer attempts than standard BC for hidden tasks and beats BC on speed and final success when task-conditioned.Action noise improves BC’s hidden-task success but hurts its task-conditioned performance; naive history conditioning can also hurt BC, unlike BE’s history conditioning.Source: §5.2.1 · p. 8, Figure 7 · p. 9, Figure 8 · p. 9

5.3. Understanding Behavioral Exploration

  • BE’s visited regions shift away from conditioning trajectories, and conditioning on past states yields broader maze coverage than a fixed history or BC.Compared with BC, BE also covers more low-density regions of the demonstration data.Source: §5.3 · pp. 9–10, Figure 13 · p. 10, Figure 12 · p. 10

6. Discussion

Behavioral exploration (BE) trains policies to explore within the behaviors in demonstration data, and experiments report gains over behavioral cloning and RL-based exploration. The discussion identifies the demonstration-behavior boundary and language-based exploration as future directions.

6. Discussion

  • BE trains policies to explore over the behaviors contained in a demonstration dataset.Source: §6 · p. 10
  • BE yields substantial gains over standard BC and RL-based exploration, and WidowX results demonstrate scaling to real-world settings.The authors also identify combining BE with generalist robot policies as future work.Source: §6 · p. 10
  • Exploration may require behaviors beyond the demonstration data, a setting the authors leave as an open question.Source: §6 · p. 10
  • The authors propose exploring whether BE can be combined with LLMs for language-based exploration.Source: §6 · p. 10

A. Proof of Proposition 4.2

Under the stated assumptions, the proof shows that a history-conditioned policy reaches every terminal feature direction spanned by the behavior policy, making its visitation distribution optimal after nβ episodes. The proof uses nested induction to establish progress toward previously unreached terminal states.

A. Proof of Proposition 4.2

  • The result assumes a deterministic environment, one-hot features nonzero only at terminal states, finite actions, a shared start state, nβ rounds, and a sufficiently small ϵ.Source: A. Proof of Proposition 4.2 · p. 16
  • An outer induction establishes that each episode reaches a previously unreached terminal state in Cβ, reducing the remaining directions by one per episode.The proof tracks the remaining set of feature directions across episodes.Source: A. Proof of Proposition 4.2 · p. 16
  • The inner induction uses history-conditioned coverage maximization to select supported actions that can reach a terminal state with a previously unseen feature direction.The argument maintains a path using actions in the support of πβ and rules out choices that cannot reach a remaining terminal direction.Source: A. Proof of Proposition 4.2 · pp. 16–17
  • After nβ episodes, eπ reaches each direction in Cβ with probability 1/nβ, making it optimal for Objective 4.1.Source: A. Proof of Proposition 4.2 · pp. 16–17

B. Experiment Details

The experiments use a diffusion-policy architecture and action chunking for both BE and BC, with BE additionally requiring history and future-trajectory settings. Hyperparameter sweeps and history subsampling are also part of the experimental procedure.

B. Experiment Details

  • BE and BC share a diffusion-policy architecture and action chunking, with several future actions executed before the next prediction.Source: B. Experiment Details · p. 18, Table 1 · p. 18
  • BE additionally specifies a context length and, when trajectories are long, a future-trajectory length for coverage calculation.Source: B. Experiment Details · p. 18
  • BC uses the same architecture-size sweep as BE, and both methods report their best-performing settings; shorter-than-collected histories are randomly downsampled at deployment.Source: B. Experiment Details · p. 18

B.1. D4RL Antmaze and Kitchen Experiments

The D4RL experiments test Antmaze navigation and Kitchen manipulation, using success over a sequence of episodes and additional maze-traversal measures. The appendix describes shortened Antmaze horizons, baseline settings, and BE's feature construction.

B.1. D4RL Antmaze and Kitchen Experiments

  • The environment visualizations include Antmaze and Kitchen, with red squares marking regions used for region counts in Antmaze.Source: Figure 14 · p. 18, Figure 15 · p. 18
  • Antmaze episodes are shortened to 500 steps for medium and 750 for large to test directed exploration; success is counted across the full sequence, not within one episode.Kitchen success uses its default detector, while Antmaze goals use a specified distance threshold.Source: §B.1 · p. 18
  • RL baselines use provided implementations and checkpoints, with online warmup lengths swept and selected for best final performance per method and environment.Source: §B.1 · pp. 18–19
  • BE represents states with the final layer of a randomly initialized three-layer network, using 32-dimensional features and λ = 0.01.At deployment, the history is a randomly selected subset of previously visited states, resampled each episode.Source: §B.1 · p. 19
  • The appendix identifies average-goal results for Antmaze medium and large, final success-rate tables, and individual-goal plots; supplied figure captions specify goal, region, and individual-goal measures.Source: §B.1 · p. 19, Table 2 · p. 19, Figure 16 · p. 20

B.2. Libero Experiments

The Libero appendix reports results averaged across 90 tasks and provides final success rates and task-level plots, alongside evaluation and deployment details.

B.2. Libero Experiments

  • In hidden-task evaluation, training omits task conditioning but includes a 21-dimensional scene vector; each task runs for 300 steps with the built-in success detector.All methods receive timestep, proprioceptive state, and both camera views, with images encoded by ResNet-18.Source: §B.2 · p. 21
  • For task-conditioned evaluation, the task is represented by a one-hot encoding.At deployment, BE samples one state from each of the three most recent episodes and repeats each for one-third of the history length.Source: §B.2 · p. 21
  • BE and BC use swept hyperparameters, while SUPE uses default vision-based settings; BC comparison variants add action noise or condition on 100 steps of history.The action-noise variant adds Gaussian noise with covariance 0.15·I at every step.Source: §B.2 · p. 21
  • Results are averaged across all 90 tasks, with final success rates in Table 6 and individual outcomes in Figures 22–25.The supplied passages do not state the success-rate values.Source: §B.2 · p. 21, Table 6 · p. 21, Figure 22 · p. 23

B.3. WidowX Experiments

The WidowX appendix evaluates three manipulation tasks and reports that prompt changes shift OpenVLA’s choices somewhat but do not produce robust exploratory behavior.

B.3. WidowX Experiments

  • The robot uses delta end-effector control at 5 Hz, with a top-down RGB camera and 128×128 images as policy input.Source: §B.3 · p. 27
  • For BE and BC, text conditioning is omitted and the state feature uses only relative x, y, and z coordinates; reaching succeeds when the end-effector contacts the object.Source: §B.3 · p. 27
  • Because OpenVLA cannot condition on a history of past observations, each deployment episode is effectively independent from past episodes; the experiment tests whether behavior converges on one solution or diversifies.Source: §B.3 · p. 27
  • Prompt modification made OpenVLA reach for the left pot more often but retained a strong rightward bias and increased failures to reach either pot.The evaluation considers eight rollouts; the authors conclude prompt changes fall short of robust exploratory behavior.Source: §B.3 · pp. 27–28, Table 8 · p. 28
Loading 2507.09041v1…