PaperlayerView exact revision on arXiv

Source-linked AI summary

DARLA: Improving Zero-Shot Transfer in Reinforcement Learning

Irina Higgins, Arka Pal, Andrei A. Rusu, Loic Matthey, Christopher P Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, Alexander Lerchner

arXiv:1707.08475v2stat.MLcs.AIcs.LG

TL;DR

Deep RL domain adaptation seeks policies that transfer from data-rich source settings to target domains without relying on costly target data. DARLA learns disentangled visual representations before learning a source policy, and achieves significantly stronger zero-shot transfer across environments and RL algorithms. Its effectiveness is supported under a shared underlying structure across domains, while β-VAE can discard scene information as β increases.

  • Problem

    Deep RL policies can be brittle under input-distribution changes, and many adaptation methods rely on target-domain data that may be expensive or difficult to obtain.

  • Method

    DARLA uses an unsupervised three-stage pipeline that learns disentangled visual representations, trains a source policy on them, and transfers that policy without target-domain knowledge.

  • Results

    Median 2.7 times improvement in zero-shot transfer across tasks and algorithms, with significant improvements across DeepMind Lab, MuJoCo, MuJoCo-to-Jaco sim2real, DQN, A3C, and EC.

  • Takeaways & Limitations

    Disentangled representations improve robust zero-shot domain adaptation across the evaluated environments, task setups, and RL algorithms.

  • Takeaways & Limitations

    DARLA assumes domains share an underlying factorised structure, and increasing β can discard crucial scene information from the latent representation.

Abstract

from arXiv · show

Domain adaptation is an important open problem in deep reinforcement learning (RL). In many scenarios of interest data is hard to obtain, so agents may learn a source policy in a setting where data is readily available, with the hope that it generalises well to the target domain. We propose a new multi-stage RL agent, DARLA (DisentAngled Representation Learning Agent), which learns to see before learning to act. DARLA's vision is based on learning a disentangled representation of the observed environment. Once DARLA can see, it is able to acquire source policies that are robust to many domain shifts - even with no access to the target domain. DARLA significantly outperforms conventional baselines in zero-shot domain adaptation scenarios, an effect that holds across a variety of RL environments (Jaco arm, DeepMind Lab) and base RL algorithms (DQN, A3C and EC).

1. Introduction

Deep RL policies can be brittle under input-distribution shifts, while existing adaptation methods often rely on costly target-domain data. DARLA addresses this by learning disentangled, domain-independent visual representations before learning source policies, enabling robust zero-shot transfer.

  • Motivation: Domain adaptation tests whether a source-trained RL policy can generalise to a target domain with changed observations but largely intact rewards.The paper defines robustness as minimal performance drop without extra fine-tuning.
  • Motivation: Existing approaches commonly align source and target representations using observation and reward signals from both domains.Such target-domain dependence can be problematic when data is expensive or difficult to obtain, especially in robotics.
  • Approach: DARLA learns low-dimensional factorised representations intended to capture underlying world structure rather than task- or domain-specific features.Relevant factors include object properties, geometry, and lighting.
  • Approach: DARLA separates learning to see, learning to act, and transfer, using disentangled visual representations to acquire a robust source policy.The visual system learns from raw unlabelled observations and parses basic concepts such as objects, positions, and colours.
  • Contribution: DARLA achieves significantly better out-of-the-box domain-adaptation performance than various baselines without requiring target-domain data to form its representations.The proposed agent is designed for zero-shot transfer after source-domain policy learning.

2. Framework

DARLA assumes related MDPs share an underlying factorised world structure and uses disentangled representations to support zero-shot transfer across domain shifts. Its three-stage pipeline learns visual factors unsupervised, trains a fixed-representation source policy, and evaluates that policy on an unretrained target domain.

  • Domain adaptation: Domain adaptation allows source and target MDPs to differ in state spaces while sharing action spaces and structurally similar transitions and rewards.The framework models domains as MDPs and describes shifts through different sampling processes over shared world factors.
  • Shared world structure: DARLA assumes natural-world MDPs share an underlying space of high-level factors of variation, with each domain corresponding to a subset of that space.This shared structure is the basis for learning representations that can apply across domains.
  • Representation problem: Reward-shaped source representations can entangle task-relevant and domain-specific attributes, causing source policies to generalise poorly to target domains.The paper illustrates this with policies that associate object identities with room colours in the source domain.
  • DARLA pipeline: DARLA first learns disentangled generative factors from unlabelled observations, then trains a source policy with a standard RL algorithm while keeping the visual mapping fixed.The unsupervised stage requires sufficient variability of factors and their conjunctions in the pre-training environment.
  • DARLA pipeline: In zero-shot transfer, DARLA evaluates the source policy on a target domain without retraining and compares it with policies using entangled representations.The intended mechanism is that the policy can ignore irrelevant domain attributes when operating on disentangled latent states.
  • Disentangled representation learning: DARLA uses β-VAE to discover factorised latent representations by balancing reconstruction accuracy against latent channel capacity and independence constraints.Values β > 1 typically encourage more disentangled representations by limiting latent-channel capacity, but increasing β can discard crucial scene information.

3. Tasks

The tasks test whether policies trained in source settings transfer without retraining to target domains that recombine or alter visual factors. Experiments cover DeepMind Lab and Jaco/MuJoCo, including sim2sim and sim2real reaching.

  • DeepMind Lab: The DeepMind Lab setup uses the same observation simulator across domains but different factor-sampling processes, testing domain interpolation.All target factor values were previously seen in the source, but their conjunction differs.
  • DARLA pipeline: DARLA’s first stage learns disentangled visual factors from unlabelled observations spanning independent room, object, arm, and object-position variations.The training environments must cover each factor’s values without adding extraneous correlations.
  • DARLA pipeline: In stage two, pretrained vision supports DQN, A3C, or Episodic Control source-policy learning; stage three evaluates zero-shot transfer to DT.The source policy is tested without retraining in the target domain.
  • DeepMind Lab: DeepMind Lab tests transfer to a held-out pink-room hat/can conjunction, where cans remain rewarding and hats remain penalized.Source training presents hat/can pairs in green rooms and cake/balloon pairs in green or pink rooms.
  • Jaco Arm and MuJoCo: Jaco/MuJoCo evaluates object-reaching from raw visual inputs, requiring inference of arm and object positions and velocities.The goal is to place the end effector as close to the object as possible.
  • Jaco Arm and MuJoCo: Sim2sim changes camera distributions and introduces held-out object colours, while sim2real transfers from MuJoCo simulation to the real world.The real-world setting includes visual details such as shadows and specularity absent from simulation.

4. Results

DARLA learns representations whose latent units separately encode environment and object variables, and it zero-shot-generalises better than entangled and denoising-autoencoder variants. Transfer performance also correlates positively with disentanglement, while fine-tuning can worsen A3C transfer.

  • Representation learning: DARLA’s latent units independently represent room, geometry, colour, object type, size, rotation, arm, object, and camera variables across DeepMind Lab and MuJoCo.Entangled DARLA representations distribute multiple factors across each latent instead.
  • Zero-shot transfer: DARLA zero-shot-generalises significantly better than DARLAENT and DARLADAE across target domains without fine-tuning the source policy.The comparison uses agents with equivalent architectures and observed data distributions apart from representation type.
  • Zero-shot transfer: DARLA’s transfer advantage is not explained simply by greater exposure to training observations, because its representation baselines saw the same data.The reported pattern holds across target domains and usually exceeds the second-best agent significantly.
  • Representation learning: DARLA uses fewer observation-specific latents than its baselines: 8 of 32 in MuJoCo/Jaco and 20 in DeepMind Lab, versus all 32 for DARLAENT and DARLADAE.The disentangled β-VAE preserves less raw-observation information because its higher β constrains the latent channel.
  • Fine-tuning: Fine-tuning vision during source-policy learning significantly worsens A3C zero-shot transfer, whereas the off-policy DQN fine-tuned agent performs very well.The authors suggest subsequent source-domain overfitting may explain the A3C result and leave the effect for future work.
  • Disentanglement and transfer: r = 0.6, p < 0.001: disentanglement positively correlates with EC-based DARLA’s zero-shot DeepMind Lab transfer performance.The correlation is measured using the transfer/disentanglement score.
  • Additional result: Disentangled representations also significantly improve source-policy learning speed, although this data-efficiency result is outside the paper’s main focus.The additional results and discussion are provided in the supplementary materials.

5. Conclusion

DARLA learns disentangled visual representations before acquiring a source policy, enabling zero-shot domain adaptation. Its benefits appear across domains, task setups, and RL algorithms.

  • DARLA first learns an unsupervised disentangled visual system, then uses those representations to learn a robust source policy.
  • DARLA achieves zero-shot domain adaptation by learning a robust source policy from disentangled representations.
  • 2.7 times median improvement in zero-shot transfer is reported across tasks and RL algorithms compared with baseline algorithms.The effect is reported across DeepMind Lab, MuJoCo, and MuJoCo-to-Jaco sim2real setups, using DQN, A3C, and EC.
  • The paper presents this as a comprehensive empirical demonstration of disentangled representations for domain adaptation in deep RL.

A.1. The Reinforcement Learning Paradigm

Reinforcement learning models agent–environment interaction as a Markov Decision Process and learns policies that maximize discounted future rewards.

  • An RL agent receives observations generated from environment states and may receive rewards conditional on its actions.
  • The agent–environment interaction is modeled as a Markov Decision Process defined by states, actions, transitions, rewards, and a discount factor.
  • A policy π(a|s) is a probability distribution over actions conditioned on the current state.
  • The learning objective is to maximize expected return, the discounted sum of future rewards.The value function gives expected return from a state, while the action-value function additionally conditions on taking a particular action.

A.2.1. DEEPMIND LAB

The DeepMind Lab source task presents randomized room and object-type combinations in a seek-avoid setting, with transfer evaluated on a held-out combination.

  • Each source episode uses one of three randomly selected room/object type conjunctions.
  • The seek-avoid task rewards collecting one object type with +1 and penalizes collecting the other with -1.
  • Episodes last 60 seconds, after which a new episode begins.
  • Transfer places the agent in a held-out conjunction of object types and room background, while visual pre-training uses other conjunctions.
  • DeepMind Lab observations have dimensions 84x84x3, corresponding to height, width, and channels.

A.2.2. MUJOCO/JACO ARM EXPERIMENTS

The MuJoCo/Jaco experiments train an arm to reach randomly placed objects under shaped rewards and evaluate robustness across simulated and real-control conditions.

  • The source task trains a simulated arm to reach an object using a shaping reward with maximum value 1 near the end-effector’s pinch and grip sites.Distances along x and y count double relative to distance along z.
  • Objects are randomly placed within a 40x40cm area, while the arm begins from a random start position independent of the object.
  • Each episode lasts 150 steps, or 7.5 seconds, with a 50ms control step.
  • Visual pre-training uses 4 million frames with dimensions 64x64x3 and randomized camera parameters around the real camera setup.
  • In the sim2real setup, nine joints independently choose among 11 discretized velocity actions, with simulation noise and real-system delays modeled.

A.3. Vision model details

The vision models use denoising and variational autoencoder components to learn representations for DARLA, with β controlling disentanglement and a frozen DAE providing perceptual reconstruction features.

  • Vision model construction: A denoising autoencoder provides the feature space for the β-VAE reconstruction loss and is trained using occlusion-style masking noise.The DAE is intended to learn a semantic representation of input frames.
  • Vision model construction: The β-VAEDAE passes sampled β-VAE reconstructions through a pretrained DAE and minimizes the L2 distance between reconstructed and original representations.The DAE weights remain frozen during β-VAE training.
  • Architecture: The β-VAE uses four convolutional layers, a 256-neuron fully connected layer, and a 64-neuron latent layer parametrizing 32 marginally independent Gaussian distributions.The decoder reverses the encoder architecture with deconvolutional layers.
  • Training settings: For DeepMind Lab, DARLA uses β=1 while DARLAENT uses β=0.1, with both models computing perceptual similarity at the DAE’s last deconvolutional layer.Both models use Adam with learning rate 1e-4.
  • Training settings: For MuJoCo/Jaco, DARLA uses a standard β-VAE with β=175, whereas DARLAENT uses β=1 corresponding to a standard VAE.The MuJoCo/Jaco optimizer is Adam with learning rate 1e-4.

A.3.4. DENOISING AUTOENCODER FOR BASELINE

The DARLADAE baseline replaces the stochastic β-VAE components with deterministic neurons while retaining the corresponding architecture and denoising pretraining setup.

  • Baseline construction: DARLADAE is a denoising autoencoder baseline trained with occlusion-style masking noise.Its architecture matches the β-VAE architecture described for the vision model.
  • Baseline construction: All stochastic nodes in the β-VAE-matched architecture are replaced with deterministic neurons for DARLADAE.This distinguishes the baseline from the stochastic β-VAE-based models.
  • Training: The baseline is optimized with Adam using a learning rate of 1e-4.

A.4. Reinforcement Learning Algorithm Details

This section describes how DARLA’s frozen vision representations are integrated with several RL algorithms and how source-task learning, disentanglement, and transfer performance are evaluated.

  • RL algorithm integration: DQN replaces its convolutional vision stack with DARLA’s frozen β-VAEDAE encoder and concatenates representations from four consecutive frames.The resulting policy input has dimension k = 4|s_z|, with |s_z|=64 for the compared models.
  • RL algorithm integration: A3C similarly replaces its shared policy-and-value convolutional stack with the frozen β-VAEDAE encoder.Other A3C hyperparameters follow the cited implementation.
  • RL algorithm integration: For MuJoCo/Jaco, DARLA uses feedforward A3C with a β-VAE encoder, followed by a 512-unit linear layer and nine independent policy outputs.Each joint has 11 discrete actions, giving an action space of 99.
  • Disentanglement evaluation: The transfer metric trains linear classifiers to predict room and object classes from fixed latent representations using only a subset of their Cartesian product, then evaluates accuracy on held-out combinations.This measures disentanglement only up to linearity for the selected factors.
  • Disentanglement evaluation: The transfer metric measures only linear disentanglement for object type and room background, although it correlates highly with visual inspection.
  • Source-task results: DARLA initially boosts learning performance, with the effect depending on representation disentanglement and being especially observable for A3C.
  • Source-task results: DARLA’s source policies are about 20% below fine-tuned models, followed by a median 270% improvement in target-domain performance.Fine-tuned baselines can achieve higher asymptotic source performance, especially for DQN and A3C.
  • Source-task results: Fine-tuning DARLA’s vision module accelerates source learning and matches baseline asymptotic performance, but reduces A3C transfer performance; for DQN it appears beneficial overall.
Loading 1707.08475v2…