Self-Supervised Exploration via Disagreement

Deepak Pathak; Dhiraj Gandhi; Abhinav Gupta

Source-linked AI summary

Self-Supervised Exploration via Disagreement

Deepak Pathak, Dhiraj Gandhi, Abhinav Gupta

arXiv:1906.04161v1Jun 10, 2019cs.LGcs.AIcs.CVcs.ROstat.ML

TL;DR

Efficient exploration remains difficult when sensorimotor environments are stochastic and existing intrinsic-reward methods are sample-inefficient. The paper trains an ensemble of dynamics models to maximize prediction disagreement and optimizes the policy through a differentiable objective, demonstrating the approach across benchmark environments and on a real robot.

Problem
Exploration is a major bottleneck, especially because stochastic dynamics challenge predictive models and existing intrinsic-reward formulations are sample-inefficient.
Method
The method trains an ensemble of forward dynamics models and selects actions maximizing disagreement, then optimizes the policy by direct likelihood maximization through a differentiable objective.
Results
The approach is demonstrated in stochastic Atari, MNIST, Mujoco, Unity, and a real robotic object-manipulation setup.
Takeaways & Limitations
Self-supervised disagreement-based exploration supports sample-efficient policy learning without requiring extrinsic rewards.
Takeaways & Limitations
The differentiable exploration results are limited to relatively short-horizon scenarios because training long-horizon models from high-dimensional inputs remains unsolved.

Abstract

from arXiv · show

Efficient exploration is a long-standing problem in sensorimotor learning. Major advances have been demonstrated in noise-free, non-stochastic domains such as video games and simulation. However, most of these formulations either get stuck in environments with stochastic dynamics or are too inefficient to be scalable to real robotics setups. In this paper, we propose a formulation for exploration inspired by the work in active learning literature. Specifically, we train an ensemble of dynamics models and incentivize the agent to explore such that the disagreement of those ensembles is maximized. This allows the agent to learn skills by exploring in a self-supervised manner without any external reward. Notably, we further leverage the disagreement objective to optimize the agent's policy in a differentiable manner, without using reinforcement learning, which results in a sample-efficient exploration. We demonstrate the efficacy of this formulation across a variety of benchmark environments including stochastic-Atari, Mujoco and Unity. Finally, we implement our differentiable exploration on a real robot which learns to interact with objects completely from scratch. Project videos and code are at https://pathak22.github.io/exploration-by-disagreement/

1. Introduction

The paper targets exploration bottlenecks caused by stochastic dynamics and poor sample efficiency in intrinsic-reward methods. It proposes ensemble disagreement as a self-supervised signal and uses a differentiable objective for direct policy optimization, demonstrating results across simulated and robotic settings.

Motivation: Stochastic observations, action noise, and stochastic action outcomes make predictive modeling difficult beyond noise-free simulations.Stochastic models remain challenging to scale to high-dimensional inputs, while deterministic feature-space models can fail when the agent itself causes the noise.
Motivation: Current intrinsic-reward methods are also sample-inefficient because their rewards depend on unknown environment dynamics and require high-variance policy-gradient estimators.Curiosity, for example, rewards disagreement between predictions and observations, but its action-dependent reward is treated as a black box for policy optimization.
Approach: The proposed approach trains an ensemble of forward dynamics models and incentivizes actions that maximize prediction disagreement or variance.This formulation is inspired by active learning and Query-by-Committee, where disagreement identifies informative samples without relying on ground-truth labels.
Approach: The disagreement objective is differentiable, enabling direct likelihood maximization for policy optimization instead of reinforcement learning.The resulting exploration policy is described as sample efficient and is evaluated in stochastic Atari, MNIST, Mujoco, Unity, and real-robot manipulation.

2. Exploration via Disagreement

The paper explores state spaces by training an ensemble of forward models and rewarding disagreement, including in stochastic environments. Because the disagreement reward omits the observed next state, it can support direct, differentiable policy optimization alongside self-supervised learning.

2. Exploration via Disagreement: The agent jointly trains a policy and an ensemble of forward models to explore efficiently without extrinsic rewards.Collected transitions train models that predict the next state from the current observation and action; policy optimization maximizes expected discounted intrinsic reward.
2. Exploration via Disagreement: Model disagreement serves as intrinsic reward because explored regions promote agreement, whereas novel regions produce divergent next-state predictions.The reward is defined from variance across ensemble outputs, using disagreement to guide policy exploration.
2. Exploration via Disagreement: Unlike conventional intrinsic-reward methods that treat rewards as black boxes, the proposed reward does not depend on the observed next state and can provide analytical policy gradients.The policy can optimize the differentiable disagreement objective directly rather than relying only on reinforcement-learning estimators.
2. Exploration via Disagreement: The formulation addresses stochasticity by allowing ensemble predictions to converge toward the mean of stochastic outcomes, reducing reward in stochastic local minima.This contrasts with prediction-error rewards, which can remain nonzero when outcomes are inherently noisy.
2. Exploration via Disagreement: The differentiable objective enables policy optimization through direct likelihood maximization, with policy and forward models trained alternately using supervised maximum likelihood.The paper presents short-horizon differentiable exploration and leaves multi-step scenarios for future work.

3. Implementation Details and Baselines

The implementation trains forward dynamics in feature space and compares disagreement-based variants with curiosity, prediction-error, and Bayesian disagreement baselines. A near-deterministic Atari sanity check evaluates whether the method preserves standard-environment performance.

3. Implementation Details and Baselines: Forward dynamics models are trained in feature space because feature-space prediction is reported to generalize better than raw pixel-space prediction.The formulation can use any representation φ by replacing observations with φ(x_t).
3. Implementation Details and Baselines: For continuous actions, direct policy gradients use deterministic actions with epsilon-greedy sampling; for discrete actions, the paper uses a straight-through estimator.These mechanisms back-propagate through the action-sampling process into the policy.
3. Implementation Details and Baselines: The comparisons include PPO-optimized disagreement, differentiable disagreement, curiosity, random-feature prediction error, prediction-error variance, and Bayesian disagreement.Bayesian disagreement is represented by a dropout neural network.
3. Implementation Details and Baselines: Figure 2 compares intrinsic-reward formulations on a near-deterministic Atari benchmark to test whether disagreement preserves accuracy in non-stochastic scenarios.The caption states that disagreement compares favorably with state-of-the-art approaches without losing accuracy.

4. Experiments

The experiments evaluate disagreement-based exploration across deterministic and stochastic environments, then test differentiable policy optimization for efficient exploration in simulation and on a real robot. The method matches or exceeds comparison approaches, handles stochasticity, and improves learning and object interaction efficiency.

Experimental scope: Experiments span Atari, 3D Unity navigation, MNIST, Mujoco object manipulation, and real-world Sawyer-arm manipulation.The evaluation covers non-stochastic environments, stochastic transition or observation settings, and differentiable policy optimization.
Non-stochastic environments: The disagreement formulation performs as well as or slightly better than state-of-the-art exploration methods in most standard scenarios.The comparison includes curiosity, random-feature curiosity, Bayesian uncertainty, and prediction-error variance.
Stochastic environments: In Noisy MNIST, disagreement converges to almost zero intrinsic reward for both low- and high-stochasticity states, unlike prediction-error reward.Prediction-error reward remains high for highly stochastic observations, which can keep the agent exploring them indefinitely.
Stochastic environments: In Unity navigation, the disagreement approach performs similarly without the TV and outperforms prediction-error curiosity when stochastic TV content is present.The result supports ensemble disagreement as an alternative for stochastic setups.
Stochastic environments: In sticky-action Atari, disagreement-based exploration is robust across scenarios and eventually achieves a higher Pong score than the comparison approach.The ensemble version outperforms previous state-of-the-art approaches, while Bayesian network-based disagreement performs less well.
Differentiable exploration: Differentiable policy optimization accelerates Enduro learning and explores more efficiently in Mujoco and real-robot object interaction tasks.The real robot learns to interact with objects in fewer than 1000 examples, while the final policy interacts with unseen objects approximately 67% of the time versus 17% for random exploration.

5. Related Work

Prior exploration work includes curiosity, uncertainty, visitation, diversity, adversarial, empowerment, and policy-perturbation methods, while this paper adapts disagreement-based selection from active learning to exploration.

Curiosity methods use prediction error or surprise to encourage exploration, while uncertainty-based methods target states and actions whose forward-model predictions are uncertain.
Other intrinsic-reward approaches seek infrequently visited states, diverse states or actions, or states expected to improve the prediction model.
Related exploration strategies include adversarial goal generation, empowerment, perturbations of learned policies, and value-based methods.
Active learning selects unlabeled samples using criteria such as entropy, uncertainty, expected informativeness, and disagreement; this work applies disagreement to exploration instead.