Source-linked AI summary
Unsupervised Control Through Non-Parametric Discriminative Rewards
David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, Volodymyr Mnih
TL;DR
Reward-free control remains difficult because agents typically rely on hand-crafted rewards or expert data. DISCERN jointly learns a goal-conditioned policy and discriminative achievement reward from observations and actions, and learns diverse visually specified goals across Atari, the DeepMind Control Suite, and DeepMind Lab.
Problem
Reward-free control lacks general methods for learning goal-conditioned behavior without hand-crafted rewards or expert demonstrations.
Method
DISCERN jointly learns a goal-conditioned policy and goal-achievement reward from streams of observations and actions using mutual-information-based discriminative learning.
Results
DISCERN learns diverse visually specified goals in an unsupervised manner across Atari, the DeepMind Control Suite, and DeepMind Lab.
Takeaways & Limitations
The learned reward reflects similarity in controllable aspects of environments rather than raw observation-space distance.
Takeaways & Limitations
The fixed episode length assumes sampled goals are approximately achievable within T steps and that policies need not finish earlier.
Abstract
from arXiv · showhide
Learning to control an environment without hand-crafted rewards or expert data remains challenging and is at the frontier of reinforcement learning research. We present an unsupervised learning algorithm to train agents to achieve perceptually-specified goals using only a stream of observations and actions. Our agent simultaneously learns a goal-conditioned policy and a goal achievement reward function that measures how similar a state is to the goal state. This dual optimization leads to a co-operative game, giving rise to a learned reward function that reflects similarity in controllable aspects of the environment instead of distance in the space of observations. We demonstrate the efficacy of our agent to learn, in an unsupervised manner, to reach a diverse set of goals on three domains -- Atari, the DeepMind Control Suite and DeepMind Lab.
1 INTRODUCTION
The paper frames environment mastery as learning controllable aspects and arbitrary achievable goals without relying on a single extrinsic reward. DISCERN jointly learns a goal-conditioned policy and discriminative reward from observations and actions, and succeeds across three visual domains.
- Environment mastery can enable learning without extrinsic rewards and reaching arbitrary achievable goals, but remains an open reinforcement-learning challenge.
- DISCERN is a model-free architecture that learns from streams of observations and actions while training a policy to reach reachable goal states.
- DISCERN jointly learns a goal-conditioned policy and a goal-achievement reward using a mutual-information objective.
- The learned reward measures similarity in controllable environmental aspects rather than raw observation-space distance.
- DISCERN handles imperfectly reachable goals by matching controllable elements while ignoring distractors outside the agent’s control.
- The method learns diverse visually specified goals without extrinsic rewards across Atari, the DeepMind Control Suite, and DeepMind Lab.
2 PROBLEM FORMULATION
The paper formulates reward-free control as learning a goal-conditioned policy whose terminal reward measures achievement of a sampled goal. This replaces hand-crafted goal rewards in continuous, high-dimensional observation spaces with a learned achievement criterion.
- Standard reinforcement learning selects actions from states and maximizes expected discounted return under an extrinsic reward function.
- The proposed setting uses only observations and actions, samples a goal at episode start, and trains a goal-conditioned policy for T steps.
- The terminal goal-achievement reward r(s; sg) measures how much the final state achieves the sampled goal, while earlier rewards are zero.
- Hand-crafted rewards limit the generality of goal-conditioned policies in continuous high-dimensional observation spaces.
3 LEARNING A REWARD FUNCTION BY MAXIMIZING MUTUAL INFORMATION
DISCERN learns its policy and reward function by maximizing mutual information between sampled goals and states achieved after policy execution. A variational goal discriminator makes this objective tractable for high-dimensional observations.
- The method jointly learns a goal-conditioned policy and reward function by maximizing mutual information between the goal sg and achieved state sT.
- The formulation treats sg as sampled from pgoal and sT as induced by executing the policy for T steps.
- The time budget T is fixed here, although it could instead depend on the goal, initial state, or the agent.
- A variational distribution qφ(sg|sT) lower-bounds the mutual-information expectation, following a simplification based on prior work.
- The entropy term H(sg) is discarded because it depends on neither the policy parameters θ nor variational parameters φ.
- Restricting goals to a finite, evolving set of previously encountered states avoids directly modeling densities over high-dimensional observations.
- Policy optimization uses terminal log qφ reward, while reward-model optimization is equivalent to maximum-likelihood classification.
- The dual optimization can be interpreted as a cooperative communication game between a goal-conditioned imitator and a variational teacher.
4 DISCRIMINATIVE EMBEDDING REWARD NETWORKS
DISCERN implements reward-free goal learning with a drifting buffer of past observations, discriminative embeddings, and jointly optimized policy and reward components. Its reward uses learned cosine similarity, with practical updates based on sampled decoys and relabelled goals.
- DISCERN jointly learns the goal-conditioned policy and reward function as a practical instantiation of the mutual-information approach.
- Goals are sampled from a fixed-size buffer of past observations whose contents are replaced using recent experience, producing a drifting goal space.
- Training goals are sampled uniformly from the goal buffer rather than through an explicit curriculum.
- The reward is the positive part of cosine similarity between states and goals in a learned embedding space.
- The embedding combines policy features with a learned L2-normalized mapping, while fixing policy features during reward learning regularizes the reward model.
- The discriminator compares each trajectory goal against K decoy observations sampled from the same goal distribution.
- The embedding objective increases similarity to the trajectory goal and decreases similarity to decoys through a softmax nearest-neighbor classifier.
- The implementation uses max(0, ℓg) as a [0, 1]-scaled, less noisy reward instead of log ˆq.
5 RELATED WORK
The paper situates DISCERN among goal-conditioned reinforcement learning, reward learning, imitation, skill discovery, unsupervised representation learning, and hindsight relabelling. It also specifies actor-side goal substitution and hindsight replay procedures for training.
- DISCERN procedure: The actor samples goals, executes the behavior policy, optionally replaces the goal with a recent state at rate pHER, and sends relabelled trajectories and rewards to the learner.
- Goal-conditioned reinforcement learning: Goal-conditioned reinforcement learning extends value-function approximation to represent conditional values across states and related goals.
- Reward and imitation learning: Prior work learned goal-achievement rewards through GAN-based methods, imitation learning, pretrained representations, or dynamics models, whereas this work jointly learns the reward and policy.
- Skill discovery: Unsupervised skill-discovery methods use eigendecompositions, mutual-information bounds, proxy rewards, or auxiliary tasks to learn skills and policies.
- Unsupervised representation learning: Other approaches pretrain unsupervised representations or observation models, then sample goals for model-free control in simulated environments.
- Goal relabelling: Hindsight experience replay relabels trajectories using achieved terminal observations, and related work extends relabelling to all-goal updates, skill discovery, auxiliary tasks, and hierarchical control.
6 EXPERIMENTS
DISCERN is evaluated against goal-conditioned-policy baselines across Atari, DeepMind Control Suite, and DeepMind Lab, using qualitative and quantitative goal-achievement measures. It outperforms baselines on several control domains, discovers controllable dimensions, and achieves visually specified goals in diverse environments, although manipulator performance is negligible for all evaluated methods.
- Evaluation setup: DISCERN is compared with conditioned autoencoder, WGAN-discriminator, and pixel-distance reward baselines sharing the same goal-conditioned policy architecture and hindsight experience replay.The comparisons therefore primarily ablate the goal-achievement reward learning mechanism.
- Atari: On Seaquest and Montezuma’s Revenge, DISCERN achieved a large fraction of fixed-set goals, whereas no baseline reliably achieved goals.Success required matching the avatar’s x- and y-coordinates within 10% of the playable area; DISCERN also matched the submarine while ignoring uncontrollable fish positions in Seaquest.
- DeepMind Control Suite: Across DeepMind Control Suite domains, DISCERN outperformed baselines on several but not all tasks and reliably matched major controllable dimensions on difficult cartpole and finger tasks.The evaluation used pixel renderings, fixed sets of 100 goals, and 20 trials per goal; Figure 4 reports dimension-wise matching within 10% of each possible range.
- DeepMind Control Suite: On manipulator, none of the evaluated methods achieved non-negligible aggregate goal-achievement performance, although DISCERN discovered several major controllability dimensions.The aggregate results omit manipulator, while an appendix analysis and video provide evidence of partial controllability discovery.
- DeepMind Lab: In DeepMind Lab’s watermaze, DISCERN approximately matched wall and horizon positions but did not achieve the goal image’s position and viewpoint.The result nevertheless indicates a learned space of goals in a first-person 3D environment.
7 DISCUSSION
DISCERN learns visually specified goals without extrinsic rewards or expert demonstrations, while its discriminative objective recovers controllable structure across visual domains. Fixed episode lengths and naive goal selection remain important scope limitations, with more adaptive strategies suggested as future improvements.
- DISCERN achieves observation-specified goals without extrinsic rewards or expert demonstrations.
- Its discriminative reward objective recovers dominant underlying degrees of controllability across varied visual domains.
- Fixed-length episodes assume sampled goals are approximately achievable within T steps and do not require earlier completion.
- Goal selection ignores starting states and current policy skills, while frontier-based sampling may substantially improve performance.
- DISCERN’s controllability discovery is positioned as useful for robust low-level control and compatible with hierarchical reinforcement learning.
A1 DISTRIBUTED TRAINING
The distributed implementation uses many CPU actors to collect experience and a centralized GPU learner to batch updates. Unlike IMPALA’s stochastic actor-critic policy, DISCERN learns a goal-conditioned Q-function with Q-learning.
- A centralized GPU learner batches updates from experience collected by many parallel CPU-based actors.
- DISCERN replaces IMPALA’s stochastic actor-critic policy with a goal-conditioned state-action value function trained by Q-learning.
A2 ARCHITECTURE DETAILS
DISCERN processes the current and goal observations through a shared convolutional network, adds periodic time information, and produces action values through an LSTM-based dueling head.
- The same convolutional network processes st and sg, whose final outputs are concatenated for goal-conditioned action-value prediction.
- Goal features are computed once per episode, reducing repeated processing of the fixed goal observation.
- Periodic features sin(2πt/T) and cos(2πt/T) encode the current timestep over the goal-achievement period T.
- An LSTM feeds a dueling action-value network whose value and advantage branches are linear mappings.
A3 GOAL BUFFER
DISCERN updates a finite goal buffer using either uniform replacement or diverse sampling, with the latter improving early training coverage of states in pixel space.
- Uniform replacement substitutes the current observation for a uniformly selected goal-buffer entry with probability preplace.
- Diverse goal sampling retains observations that improve coverage relative to randomly selected buffer entries.
- The diverse strategy uses L2 pixel-space distance and greatly increases goal-buffer state coverage, especially early in training.
A4 EXPERIMENTAL DETAILS
Experiments use discretized actions, fixed training settings, and comparisons across Control Suite goal-substitution schemes. Additional analyses examine cartpole behavior and per-dimension controllability.
- Training settings: The experiments maintain a goal buffer of size 1024 and substitute hindsight goals 25% of the time.Training also uses four decoys for the teacher and samples hindsight goals from the last three trajectory frames.
- Action representation: Continuous Control Suite actions are discretized into 3A actions with values {−1, 0, 1} per dimension, while manipulator uses diagonal actuator actions.The manipulator variant sets one actuator to ±1 at a time.
- Baselines: An additional Seaquest and Montezuma’s Revenge baseline using hindsight-only policy training achieved 12% and 11.4% of goals, respectively, comparable to uniform random.This ablation removes the learned goal-achievement reward.
- Control Suite evaluation: Figure 6 reports Control Suite goal-achievement curves under the diverse goal substitution scheme.The diverse scheme complements the uniform-strategy learning-progress results reported elsewhere.
- Control Suite evaluation: On cartpole, DISCERN always matches cart position and remains robust to pole positions unseen during training, unlike the Autoencoder baseline.The baseline matches both dimensions when the pole points down but fails when it points up.
- Control Suite evaluation: Per-dimension manipulator evaluation shows that DISCERN reliably controls more underlying-state dimensions than the baselines.The figure provides a quantitative per-dimension comparison.